Posts Tagged ‘Cloud’

vCloud Director Database Migration

March 20th, 2015

This week I’ve been working on getting some lab infrastructure fitted with much needed updates. One of those components was an aging Microsoft SQL Server 2008 R2 server on Windows Server 2008 R2 which I had been using to host databases for various projects.  Since I had chosen to build the new SQL server in parallel, I’m benefiting with fresh and problem free builds of Microsoft SQL Server 2012 on Windows Server 2012 R2.  The downside is that I’m responsible for dealing with all of the SQL databases and logins and potentially scheduled jobs that must be migrated to the new SQL server.

vCloud Director is one of the last databases left to migrate and fortunately VMware has a KB article published which covers the step required to migrate a back end SQL database for vCloud Director.  The VMware KB article is 2092706 Migrating the VMware vCloud Director SQL database to another server.

Looking at the steps, the migration looks like it will be fairly simple.  VMware even provides the SQL queries to automate many of the tasks.  I’ll migrate my vCloud Director database using these steps in the following video.  I did run into a few issues which mostly boil down to copy/paste problems with the SQL queries as published in the KB article but I’ve provided the necessary corrections and workarounds in the video.

As shown in the video, I ran into a syntax issue with step four.

The SQL query provided by the KB article was:

USE master;
EXEC sp_attach_db @dbname = N’vCD_DB_Name‘,
c:\Program Files\Microsoft SQL Server\MSSQL\Backup\vCD_DB_Name.mdf
c:\Program Files\Microsoft SQL Server\MSSQL\Backup\vCD_DB_Name.ldf

The corrected SQL query syntax according to the Microsoft SQL Server Management Stuido appears to be:

USE [master]
( FILENAME = N'c:\Program Files\Microsoft SQL Server\MSSQL\Backup\vCD_DB_Name.mdf' ),
( FILENAME = N'c:\Program Files\Microsoft SQL Server\MSSQL\Backup\vCD_DB_Name.ldf' )

Another issue I’ll note that wasn’t captured in the video deals with step seven where the vCloud Director cell server is reconfigured to point to the new database.  The first time I ran that step, the process failed because the cell attempted to locate the SQL database in its original location which it actually found. When this occurred, the cell configuration script doesn’t prompt me to point to a new SQL instance.  In order for step seven to work correctly, I had to drop or delete the database on the SQL 2008 R2 server and rerun the vCloud Director configuration script.  What happens then is that the cell doesn’t automatically ‘find’ the old instance and so it correctly prompts for the new back end database details.  VMware’s KB article provides most of the steps required to migrate the database but it does need a step inserted prior to step seven which calls for the deletion of the original database instance.  Step two places the vCloud database in READ_ONLY mode but the vCloud cell configuration was still able to ‘see’ which causes step seven to fail.

Blake Garner (@trodemaster on Twitter) provided a helpful tip which will also work with step seven in lieu of dropping or deleting the database on the original SQL server:

You could also clear DB config from the /opt/vmware/vcloud-director/etc/ and run configure again.

Overall the process was still fairly simple and painless thanks to VMware’s published documentation.

Baremetalcloud Special Promo Through

March 14th, 2013

Snagit CaptureHe’s Laverick by name, Maverick by nature (and if I might add, a very cool chap and my friend) – Mike Laverick, formerly of RTFM Education of which I was a LONG time reader going back to my Windows and Citrix days, now has a blog cleverly and conveniently situated at  Since Mike joined forces with VMware, he’s been focused on vCloud evangelism and recently visited the Sydney/Melbourne VMUG where he was inspired with a new interest in home labs by AutoLab ala Alastair Cooke of Demitasse fame.  AutoLab has garnered some much deserved attention and adoption.  One organization that has taken an interest is baremetalcloud who provide IaaS via AutoLab on top of physical hardware for its customers.

Long story short, baremetalcloud is offering a special promotion to the first 100 subscribers through Mike’s blog.  Visit the Maverick’s blog via the link in the previous sentence where you can grab the promo code and reserve your baremetalcloud IaaS while supplies last.  Mike also walks through an end to end deployment so you can get an idea of what that looks like beforehand or use it as a reference in case you get stuck.

Thank you Mike, Alastair, and baremetalcloud for lending your hand to the community.

Creating vCloud Director Transfer Server Storage on NFS

July 3rd, 2012

Six months ago I wrote an article about Expanding vCloud Director Transfer Storage on a local block storage device.  Today I take a step back and document the process of instantiating vCloud Director Transfer Storage on an NFS export which is where all scalable VCD implementations in production should reside.  The process is not extremely difficult but it can be difficult to remember the fine details if Linux is not your native OS.  Basically run through the following steps on each VCD cell server in the server group before installing vCloud Director.  I’ll be performing these steps on a RHEL 5 Update 7 distribution.

First create the directory structure which the NFS export will be mounted to (the -p argument creates the entire path of directories as necessary):

mkdir -p /opt/vmware/vcloud-director/data/transfer

Next, mount the NFS export manually:

mount nfshost.fqdn.orip:/nfs_export_name /opt/vmware/vcloud-director/data/transfer

Finally, let’s make sure the NFS export auto mounts each time the cell is rebooted.  This is done by editing /etc/fstab

nano -w /etc/fstab

Add the following line to /etc/fstab:

nfshost.fqdn.orip:/nfs_export_name      /opt/vmware/vcloud-director/data/transfer       nfs     rw      0 0

Exit nano using CTRL + X. Save /etc/fstab when prompted.

Proceed with the vCloud Director cell installation.  If using the mount path in the example above, it is safe and convenient to press Enter through the default prompt relating to the Transfer Server Storage installation path.

I’ll close by pointing out that although the Transfer Server Storage is used as a temporary holding tank for vApp and catalog media imports and exports, critical cell data is also stored in this repository.  If the Transfer Server Storage area is unavailable (ie. issues with NFS or the network), the VCD cell will not function properly, yielding a range of symptoms such as not being able to authenticate at the provider or organization portals.


February 25th, 2012

I receive a lot of communication from recruiters, some of which I’m allowed to share, so I’ve decided to try something.  On the Jobs page, I’ll pass along virtualization and cloud centric opportunities – mostly US based but in some cases throughout the globe.  Only recruiter requests will be posted.  I won’t syndicate content easily found on the various job boards.  If you’re currently on the bench or looking for a new challenge, you may find it here.  Don’t tell them Jason sent you.  I receive no financial gain or benefit otherwise but I thought I could do something with these opportunities other than deleting them.  Best of luck in your search.

In case you missed the link, the Jobs page.

vCloud Director and vCenter Proxy Service Failure

December 16th, 2011

In the past couple of weeks I have spent some time with VMware vCloud Director 1.5.  The experience yielded three blog articles Collecting diagnostic information for VMware vCloud Director, Expanding vCloud Director Transfer Server Storage and now this one.

In this round, the vCD cell stopped working properly (single cell server environment).  I could log into the vCD provider and organization portals but the deployment of vApps would run for an abnormally long time and then fail after 20 minutes with one of the resulting failure messages being Failed to receive status for task.

Doing some digging in the environment I found a few problems.

Problem #1:  None of the cells have a vCenter proxy service running on the cell server.

Snagit Capture

Problem #2:  Performing a Reconnect on the vCenter Server object resulted in Error performing operation and Unable to find the cell running this listener.

Snagit Capture

Snagit Capture

I search the Community Forums, talked with Chris Colotti (read his blog) for a bit, and then opened an SR with VMware support.  VMware sent me a procedure along with a script to run on the Microsoft SQL Server:

  1. BACKUP the entire SQL Database.
  2. Stop all cells. (service vmware-vcd stop)
  3. Run the attached reset_qrtz_tables_sql_database.sql
    — shutdown all cells before executing
    delete from qrtz_scheduler_state
    delete from qrtz_fired_triggers
    delete from qrtz_paused_trigger_grps
    delete from qrtz_calendars
    delete from qrtz_trigger_listeners
    delete from qrtz_blob_triggers
    delete from qrtz_cron_triggers
    delete from qrtz_simple_triggers
    delete from qrtz_triggers
    delete from qrtz_job_listeners
    delete from qrtz_job_details
  4. Start one cell and verify if issue is resolved. (service vmware-vcd start)
  5. Start the remaining cells.

Before running the script I knew I had to make a few modifications to select the vCloud database first.

When running the script, it failed due to case sensitivity with respect to the table names.  Upon installation, vCD creates all tables with upper case names.  When the MS SQL Server database was first created by yours truly, case sensitivity, along with accent sensitivity, were enabled with COLLATE Latin1_General_CS_AS which comes straight from page 17 of the vCloud Director Installation and Configuration Guide.

After fixing the script, it looked like this:

USE [vcloud]

– shutdown all cells before executing
delete from QRTZ_CALENDARS
delete from QRTZ_TRIGGERS
delete from QRTZ_JOB_DETAILS

The script ran successfully wiping out all rows in each of the named tables.  A little sidebar discussion here.. I talked with @sqlchicken (Jorge Segarra, read his blog here) about the delete from statements in the script. It is sometimes a best practice to use the truncate table statement instead so that the transaction logs are bypassed instead of using the delete from statement which is more resource intensive due to the row by row deletion method and the rows being recorded in the transaction logs. Thank you for that insight Jorge! More on MS SQL Delete vs Truncate here. Jorge was also kind enough to provide a link on the subject matter but credentials will be required to view the content.

I was now able to restart the vCD cell and my problems were gone. Everything was working again. All errors have vanished.  I thanked the VMware support staff and then tried to gain a little bit more information about how the problem was resolved by deleting table rows and what exactly are the qrtz tables?  I had looked at the table rows myself before they were deleted and the information in there didn’t make a lot of sense to me (but that doesn’t necessarily classify it as transient data).  This is what he had to say:

These [vCenter Proxy Service] issues are usually caused by a disconnect from the database, causing the tables to become stale. vCD constantly needs the ability to write to the database and when it cannot, the cell ends up in a state that is similar to the one that you have seen.

The qrtz tables contain information that controls the coordinator service, and lets it know when the coordinator to be dropped and restarted, for cell to cell fail over to another cell in multi cell enviroment.

When the tables are purged it forces the cell on start up to recheck its status and start the coordinator service. In your situation the cell, due to corrupt records in the table was not allowing this to happen.

So by clearing them forced the cell to recheck and to restart the coordinator.

Good information to know going forward. I’m going to keep this in my back pocket. Or on my blog as it were.  Have a great weekend!

Collecting diagnostic information for VMware vCloud Director

December 12th, 2011

I’ve gone a few rounds with VMware vCloud Director in as many weeks recently.  I’ve got an upcoming blog post on a vCenter Proxy Service issue I’ve been dealing with but until I collect the remaining details on that, I thought I’d point out VMware KB 1026312 Collecting diagnostic information for VMware vCloud Director.  This knowledge base article details the steps required to collect the necessary support logs for both vCD versions 1.0 and 1.5.

The vmware-vcd-support script collects host log information as well as these vCloud Director logs. The script is located in the following folders:

For vCloud Director 1.0, run /opt/vmware/cloud-director/bin/vmware-vcd-support

For vCloud Director 1.5, run /opt/vmware/vcloud-director/bin/vmware-vcd-support

Once executed, the script will bundle the following log files from /opt/vmware/vcloud-director/logs/ into a .tgz tarball saving it in the directory from which the script was run (providing there is enough storage available):

  1. cell.log – Console output from the vCloud Director cell.
  2. diagnostics.log – Cell diagnostics log. This file is empty unless diagnostics logging is enabled in the local logging configuration.
  3. vcloud-container-info.log – Informational log messages from the cell. This log also shows warnings or errors encountered by the cell.
  4. vcloud-container-debug.log – Debug-level log messages from the cell.
  5. vcloud-vmware-watchdog.log – Informational log messages from the cell watchdog. It records when the cell crashes, is restarted, etc.

On the subject of vCD log files, also mentioned in the KB article is VMware KB 1026815 Configuring logging for VMware vCloud Director.  The information in this article is useful for specifying the quantity and size of vCD log files to be maintained on the cell server.

Once the log files have been collected, you may analyze them offline or upload them to VMware’s FTP site in association with an SR by following VMware KB 1008525 Uploading diagnostic information to VMware.

Expanding vCloud Director Transfer Server Storage

December 5th, 2011

Installing vCloud Director 1.5 can be like installing a VCR.  For the most part, you can get through it without reading the instructions.  However, there may be some advanced or obscure features (such as programming the clock or automatically recording a channel) which require knowledge you’ll only pick up by referring to the documentation.  Such is the case with vCD Transfer Server Storage.  Page 13 of the vCloud Director Installation and Configuration Guide discusses Transfer Server Storage as follows:

To provide temporary storage for uploads and downloads, an NFS or other shared storage volume must be accessible to all servers in a vCloud Director cluster. This volume must have write permission for root. Each host must mount this volume at $VCLOUD_HOME/data/transfer, typically /opt/vmware/vcloud-director/data/transfer. Uploads and downloads occupy this storage for a few hours to a day. Transferred images can be large, so allocate at least several hundred gigabytes to this volume.

This is the only VMware documentation I could find covering Transfer Server Storage.  There is a bit of extra information revealed about Transfer Server Storage upon the initial installation of the vCD cell which basically states that at that point in time you should configure Transfer Server Storage to point to shared NFS storage for all vCD cells to use, or if there is just a single cell, local cell storage may be used:

If you will be deploying a vCloud Director cluster you must mount the shared transfer server storage prior to running the configuration script.  If this is a single server deployment no shared storage is necessary.

Transfer Server Storage is used for uploading and downloading (exporting) vApps.  A vApp is one or more virtual machines with associated virtual disks.  Small vApps in .OVF format will consume maybe 1GB (or potentially less depending on its contents).  Larger vApps could be several hundred GBs or beyond.  By default, Transfer Server Storage will draw capacity from /.  Lack of adequate Transfer Server Storage capacity will result in the inability to upload or download vApps (it could also imply you’re out of space on /).  Long story short, if you skipped the brief instructions on Transfer Server Storage during your build of a RHEL 5 vCD cell, at some point you may run short on Transfer Server Storage and even worse you’d run / out of available capacity.

I ran into just such a scenario in the lab and thought I’d just add a new virtual disk with adequate capacity, create a new mount point, and then adjust the contents of /etc/profile.d/ (export VCLOUD_HOME=/opt/vmware/vcloud-director) to point vCD to the added capacity.  I quickly found out this procedure does not work.  The vCD portal dies and won’t start again.  I did some searching and wound up at David Hill’s vCloud Director FAQ which confirms the transfer folder cannot be moved (Chris Colotti has also done some writing on Transfer Server Storage here in addition to related content I found on the vSpecialist blog).  However, we can add capacity to that folder by creating a new mount at that folder’s location.

I was running into difficulties trying to extend / so I collaborated with Bob Plankers (a Linux and Virtualization guru who authors the blog The Lone Sysadmin) to identify the right steps, in order, to get the job done properly for vCloud Director.  Bob spent his weekend time helping me out with great detail and for that I am thankful.  You rule Bob!

Again, consider the scenario: There is not enough Transfer Server Storage capacity or Transfer Server Storage has consumed all available capacity on /.  The following steps will grow an existing vCloud Director Cell virtual disk by 200GB and then extend the Transfer Server Storage by that amount.  The majority of the steps will be run via SSH, local console or terminal:

  1. Verify rsync is installed. To verify, type rsync followed by enter. All vCD supported versions of RHEL 5 (Updates 4, 5, and 6) should already have rsync installed.  If a minimalist version of RHEL 5 was deployed without rsync, execute yum install rsync to install it (RHN registration required).
  2. Gracefully shut down the vCD Cell.
  3. Now would be a good time to capture a backup of the vCD cell as well as the vCD database if there is just a single cell deployed in the environment.
  4. Grow the vCD virtual disk by 200 GB.
  5. Power the vCD cell back on and at boot time go into single user mode by interrupting GRUB (press an arrow key to move the kernel selection).  Use ‘a‘ to append boot parameters. Append the word single to the end (use a space separator) and hit enter.
  6. Use # sudo fdisk /dev/sda to partition the new empty space:
    1. Enter ‘n’ (for new partition)
    2. Enter ‘p’ (for primary)
    3. Enter a partition number.  For a default installation of RHEL 5 Update 6, 1 and 2 will be in use so this new partition will likely be 3.
    4. First cylinder… it’ll offer a number, probably the first free cylinder on the disk. Hit enter, accept the default.
    5. Last cylinder… hit enter. It’ll offer you the last cylinder available. Use it all!
    6. Enter ‘x’ for expert mode.
    7. Enter ‘b’ to adjust the beginning sector of the partition.
    8. Enter the partition number (3 in this case).
    9. In this step align the partition to a multiple of 128.  It’ll ask for “new beginning of data” and have a default number. Take that default number and round it up to the nearest number that is evenly divisible by 128. So if the number is 401660, I take my calculator and divide it by 128 to get the result 3137.968. I round that up to 3138 then multiply by 128 again = 401664. That’s where I want my partition to start for good I/O performance, and I enter that.
    10. Now enter ‘w’ to write the changes to disk. It’ll likely complain that it cannot reread the partition table but this is safe to ignore.
  7. Reboot the vCD cell using shutdown -r now
  8. When the cell comes back up, we need to add that new space to the volume group.
    1. pvcreate /dev/sda3 to initialize it as a LVM volume. (If you used partition #4 then it would be /dev/sda4).
    2. vgextend VolGroup00 /dev/sda3 to grow the volume.
  9. Now create a filesystem:
    1. lvcreate –size 199G –name transfer_lv VolGroup00 to create a logical volume 199 GB in size named transfer_lv. Adjust the numbers as needed. Notice we cannot use the entire space available due to slight overhead.
    2. mke2fs -j -m 0 /dev/VolGroup00/transfer_lv to create an ext3 filesystem on that logical volume.  The -j parameter indicates journaled, which is ext3.  The -m 0 parameter tells the OS to reserve 0% of the space for the superuser for emergencies. Normally it reserves 5%, which is a complete waste of 5% of your virtual disk.
  10. Now we need to mount the filesystem somewhere where we can copy the contents of /opt/vmware/vcloud-director/data/transfer first.  mount /dev/VolGroup00/transfer_lv /mnt will mount it on /mnt which is a good temporary spot.
  11. Stop the vCloud Director cell service to close any open files or transactions in flight with service vmware-vcd stop.
  12. rsync -av /opt/vmware/vcloud-director/data/transfer/ /mnt to make an exact copy of what’s there. Mind the slashes, they’re important.
  13. Examine the contents of /mnt to be sure everything from /opt/vmware/vcloud-director/data/transfer was copied over properly.
  14. rm -rf /opt/vmware/vcloud-director/data/transfer/* to delete the file and directory contents in the old default location. If you mount over it, the data will still be there sucking up disk space but you won’t be able to see it (instead you’ll see lost+found). Make sure you have a good copy in /mnt!
  15. umount /mnt to unmount the temporary location.
  16. mount /dev/VolGroup00/transfer_lv /opt/vmware/vcloud-director/data/transfer (all one line) to mount it in the right spot.
  17. df -h to confirm the mount point is there and vCD data (potentially along with transient transfer storage files) is consuming some portion of it.
  18. To auto mount correctly on reboot:
    1. nano -w /etc/fstab to edit the filesystem mount file.
    2. At the very bottom add a new line (but no blank lines between) that looks like the rest, but with our new mount point. Use tab separation between the fields. It should look like this:
      /dev/VolGroup00/transfer_lv /opt/vmware/vcloud-director/data/transfer/ ext3 defaults 1 2
    3. Ctrl-X to quit, ‘y’ to save modified buffer, enter to accept the filename.
  19. At this time we can either start the vCD cell with service vmware-vcd start or reboot to ensure the new storage automatically mounts and the cell survives reboots. If after a reboot the vCD portal is unavailable, it’s probably due to a typo in fstab.

This procedure, albeit a bit lengthy and detailed, worked well and was the easiest solution for my particular scenario.  There are some other approaches which would work to solve this problem.  One of them would be almost identical to the above but instead of extending the virtual disk of the vCD cell, we could add a new virtual disk with the required capacity and then mount it up.  Another option would be to build a new vCloud Director server with adequate space and then decommission the first vCD server.  This wasn’t an option for me because the certificate key files for the first vCD server no longer existed.