Posts Tagged ‘vCloud Director’

VMworld 2012 Announcements – Part I

August 27th, 2012

VMworld 2012 is underway in San Francisco.  Once again, a record number of attendees is expected to gather at the Moscone Center to see what VMware and their partners are announcing.  From a VMware perspective, there is plenty.

Given the sheer quantity of announcements, I’m actually going to break up them up into a few parts, this post being Part I.  Let’s start with the release of vSphere 5.1 and some of its notable features.

Enhanced vMotion – the ability to now perform a vMotion as well as a Storage vMotion simultaneously. In addition, this becomes an enabler to perform vMotion without the shared storage requirement.  Enhanced vMotion means we are able to migrate a virtual machine stored on local host storage, to shared storage, and then to local storage again.  Or perhaps migrate virtual machines from one host to another with each having their own locally attached storage only.  Updated 9/5/12 The phrase “Enhanced vMotion” should be correctly read as “vMotion that has been enhanced”.  “Enhanced vMotion” is not an actual feature, product, or separate license.  It is an improvement over the previous vMotion technology and included wherever vMotion is bundled.

Snagit Capture

Enhanced vMotion Requirements:

  • Hosts must be managed by same vCenter Server
  • Hosts must be part of same Datacenter
  • Hosts must be on the same layer-2 network (and same switch if VDS is used)

Operational Considerations:

  • Enhanced vMotion is a manual process
  • DRS and SDRS automation do not leverage enhanced vMotion
  • Max of two (2) concurrent Enhanced vMotions per host
  • Enhanced vMotions count against concurrent limitations for both vMotion and Storage vMotion
  • Enhanced vMotion will leverage multi-NIC when available

Next Generation vSphere Client a.k.a. vSphere Web Client – An enhanced version of the vSphere Web Client which has already been available in vSphere 5.0.  As of vSphere 5.1, the vSphere Web Client becomes the defacto standard client for managing the vSphere virtualized datacenter.  Going forward, single sign-on infrastructure management will converge into a unified interface which any administrator can appreciate.  vSphere 5.1 will be the last platform to include the legacy vSphere client. Although you may use this client day to day while gradually easing into the Web Client, understand that all future development from VMware and its partners now go into the Web Client. Plug-ins currently used today will generally still function with the legacy client (with support from their respective vendors) but they’ll need to be completely re-written vCenter Server side for the Web Client.  Aside from the unified interface, the architecture of the Web Client has scaling advantages as well.  As VMware adds bolt-on application functionality to the client, VMware partners will now have the ability to to bring their own custom objects objects into the Web Client thereby extending that single pane of glass management to other integrations in the ecosystem.


Here is a look at that vSphere Web Client architecture:

Snagit Capture


  • Internet Explorer / FireFox / Chrome
  • others (Safari, etc.) are possible, but will lack VM console access

A look at the vSphere Web Client interface and its key management areas:

Snagit Capture

Where the legacy vSphere Client fall short and now the vSphere Web Client solves these issues:

  • Single Platform Support (Windows)
    • vSphere Web Client is Platform Agnostic
  • Scalability Limits
    • Built to handle thousands of objects
  • White Screen of Death
    • Performance
  • Inconsistent look and feel across VMware solutions
    • Extensibility
  • Workflow Lock
    • Pause current task and continue later right where you left off (this one is cool!)
    • Browser Behavior
  • Upgrades
    • Upgrade a Single serverside component

 vCloud Director 5.1

In the recent past, VMware aligned common application and platform releases to ease issues that commonly occurred with compatibility.  vCloud Director, the cornerstone of the vCloud Suite, is obviously the cornerstone in how VMware will deliver infrastructure, applications, and *aaS now and into the future. So what’s new in vCloud Director 5.1?  First an overview of the vCloud Suite:

Snagit Capture

And a detailed list of new features:

  • Elastic Virtual Datacenters – Provider vDCs can span clusters leveraging VXLAN allowing the distribution and mobility of vApps across infrastructure and the growing the vCloud Virtual Datacenter
  • vCloud Networking & Security VXLAN
  • Profile-Driven Storage integration with user and storage provided capabilities
  • Storage DRS (SDRS) integration
    • Exposes storage Pod as first class storage container (just like a datastores) making it visible in all workflows where a datastore is visible
    • Creation, modification, and deletion of spods not possible in vCD
    • Member datastore operations not permissible in VCD
  • Single level Snapshot & Revert support for vApps (create/revert/remove); integration with Chargeback
  • Integrated vShield Edge Gateway
  • Integrated vShield Edge Configuration
  • vCenter Single Sign-On (SSO)
  • New Features in Networking
    • Integrated Organization vDC Creation Workflow
    • Creates compute, storage, and networking objects in a single workflow
    • The Edge Gateway are exposed at Organization vDC level
    • Organization vDC networks replace Organization networks
    • Edge Gateways now support:
      • Multiple Interfaces on a Edge Gateway
      • The ability to sub-allocate IP pools to a Edge Gatewa
      • Load balancing
      • HA (not the same as vSphere HA)
        • Two edge VMs deployed in Active-Passive mode
        • Enabled at time of gateway creation
        • Can also be changed after the gateway has been completed
        • Gets deployed with first Organizational network created that uses this gateway
      • DNS Relay
        • Provides a user selectable checkbox to enable
        • If DNS servers are defined for the selected external network, DNS requests will be sent to the specified server. If not, then DNS requests will be sent to the default gateway of the external network.
      • Rate limiting on external interface
    • Organization networks replaced by Organization vDC Networks
      • Organization vDC Networks are associated with an Organization vDC
      • The network pool associated with Organization vDC is used to create routed and isolated Organization vDC networks
      • Can be shared across Organization vDCs in an Organization
    • Edge Gateways
      • Are associated with an Organization vDC, can not be shared across Organization vDCs
      • Can be connected to multiple external networks
        • Multiple routed Organization vDC networks will be connected to the same Edge Gateway
      • External network connectivity for the Organization vDC Network can be changed after creation by changing the external networks which the edge gateway is connected.
      • Allows IP pool of external networks to be sub-allocated to the Edge Gateway
        • Needs to be specified in case of NAT and Load Balancer
    • New Features in Gateway Services
      • Load balancer service on Edge Gateways
      • Ability to add multiple subnets to VPN tunnels
      • Ability to add multiple DHCP IP pools
      • Ability to add explicit SNAT and DNAT rules providing user with full control over address translation
      • IP range support in Firewall and NAT services
      • Service Configuration Changes
        • Services are configured on Edge Gateway instead of at the network level
        • DHCP can be configured on Isolated Organization vDC networks.
  • Usability Features
    • New default branding style
      • Cannot revert back to the Charcoal color scheme
      • Custom CSS files will require modification
    • Improved “Add vApp from Catalog” wizard workflow
    • Easy access to VM Quota and Lease Expirations
    • New dropdown menu that includes details and search
    • Redesigned catalog navigation and sub-entity hierarchy
    • Enhanced help and documentation links
  • Virtual Hardware Version 9
    • Supports features presented by HW9 (like 64 CPU support)
    • Supports Hardware Virtualization Calls
    • VT-x/EPT or AMD-V/RVI
    • Memory overhead increased, vMotion limited to like hardware
    • Enable/Disable exposed to users who have rights to create a vApp Template
  • Additional Guest OS Support
    • Windows 8
    • Mac OS 10.5, 10.6 and 10.7
  • Storage Independent of VM Feature
    • Added support for Independent Disks
    • Provides REST API support for actions on Independent Disks
      • As these consume disk space, the vCD UI was updated to show user when they are used:
      • Organizations List Page
      • A new Independent Disks count column is added.
      • Organization Properties Page
      • Independent Disks tab is added to show all independent disks belonging to vDC
      • Tab is not shown if no independent disk exists in the vDC.
      • Virtual Machine Properties Page
      • Hardware tab->Hard Disks section, attached independent disks are shown by their names and all fields for the disk are disabled as they are not editable.

That’s all I have time for right now.  As I said, there is more to come later on topics such as vDS enhancements, VXLAN, SRM, vCD Load Balancing, and vSphere Replication.  Stay tuned!

Adding an IP Alias to the vCloud Director Cell Server

July 5th, 2012

Hola! Yo Soy Dora!  I hope you are having a great week and for those in the US, I hope your 4th of July holiday was fun and relaxing.

Here’s another “how to” for those not real familiar with Linux when standing up a vCloud Director infrastructure.  If you’re following the documentation, you’ll notice on page 13 of the vCloud Director Installation and Configuration Guide that two NICs or an IP alias are required to support two separate SSL connections on each vCloud Director cell server.  One IP is used for the vCloud Director HTTP service and the other is used for the console proxy service.  I’ve deployed both methods, multiple NICs and IP aliasing, for the VCD cell server.  Neither method has a distinct advantage over the other in terms of performance or other important metrics.  Where both the http and console proxy addresses are on the same subnet, I prefer to use the IP Alias method to keep things a little cleaner but using two NICs is better at full disclosure in terms of how the VCD Cell Server is built and configured from a network standpoint.

To wrap some visualization around the two options, if you’re not familiar with Linux IP Aliasing, you’d probably deploy each VCD cell server in a multihomed configured with a minimum of two NICs and two IP addresses required for VCD, one IP established for each of the required SSL connections.

Snagit Capture

The IP Alias method involves just a single NIC with two IP addresses on the same subnet sharing a common mask and default gateway for the two required SSL connections.  Don’t forget that with either method, without routed NFS on the network, each VCD cell server would likely have one additional NIC dedicated to an NFS network for vCloud Director Transfer Storage assuming the clustered cell configuration recommended for production and highly available cloud infrastructures.

Snagit Capture

I think everyone knows how to install and configure a multihomed server, so this writing will focus on adding an IP alias to a NIC in RHEL 5 Update 7, or at least it will focus on how I learned to do it via the command line.  I’ll also show a second method to accomplish adding an IP alias through the GUI (X is enabled by default in RHEL 5.7).

Assuming RHEL 5 Update 7 is already installed with a NIC having an IP address, adding an additional IP address via an alias takes just a few steps via CLI.

  1. Use nano -w /etc/sysconfig/network-scripts/ifcfg-eth0 to edit the network configuration for eth0.  If it exists, remove the line GATEWAY= or comment it out by placing a hash (#) character at the beginning of the line like so: # GATEWAY=  Save and exit nano with CTRL+X.
  2. Make a copy of ifcfg-eth0 to use for the IP alias.  Do this with the command cp /etc/sysconfig/network-scripts/ifcfg-eth0/etc/sysconfig/network-scripts/ifcfg-eth0:0
  3. Use nano -w /etc/sysconfig/network-scripts/ifcfg-eth0:0 to edit the network configuration for eth0:0.  Change DEVICE=eth0 to read DEVICE=eth0:0.  Change IPADDR= to read IPADDR=  Change ONBOOT=yes to read ONPARENT=yes  Save and exit nano with CTRL+X.
  4. Use nano -w /etc/sysconfig/network to add a commonly shared default gateway for eth0 and eth0:0.  Add the line GATEWAY=  Save and exit nano with CTRL+X.
  5. Restart networking with service network restart

At this point, the Linux platform has a single NIC with two IP addresses and the installation of vCloud Director on this cell can begin.

A second method to accomplish the above would be through the GUI by running the Networking application in RHEL 5 Update 7.

Seen here, eth0 is already configured.  Click the New button to add an IP alias:

Snagit Capture

Select Ethernet connection, choose the existing NIC for eth0, assign the IP address, Subnet Mask, and Default Gateway for the alias, and then lastly click on the Activate button with eth0:1 highlighted.

Snagit Capture

Once again, at this point, the Linux platform has a single NIC with two IP addresses and the installation of vCloud Director on this cell can begin.  Highlighted in yellow below is the IP alias or second IP address bound to eth0:

Snagit Capture

I’ve found that the GUI approach obsoletes steps 1 and 4 from the CLI approach above.  Basically it strips out the steps where the Default Gateway configuration is moved from the individual ifcfg-eth0 network startup scripts to the centralized /etc/sysconfig/network location.  It further affirms the GATEWAY= entry may remain in each of the individual ifcfg-eth0 network startup scripts.  In the end, both methods work for a vCloud Director cell server however I imagine adding an additional NIC hard wired to an access port not on the subnet will have issues with a GATEWAY= in /etc/sysconfig/network.

Creating vCloud Director Transfer Server Storage on NFS

July 3rd, 2012

Six months ago I wrote an article about Expanding vCloud Director Transfer Storage on a local block storage device.  Today I take a step back and document the process of instantiating vCloud Director Transfer Storage on an NFS export which is where all scalable VCD implementations in production should reside.  The process is not extremely difficult but it can be difficult to remember the fine details if Linux is not your native OS.  Basically run through the following steps on each VCD cell server in the server group before installing vCloud Director.  I’ll be performing these steps on a RHEL 5 Update 7 distribution.

First create the directory structure which the NFS export will be mounted to (the -p argument creates the entire path of directories as necessary):

mkdir -p /opt/vmware/vcloud-director/data/transfer

Update 5/27/18: I happened to notice with RHEL 7.5 (could impact earlier builds as well) that mounting NFS exports now requires nfs-utils. Install this from the local DVD repository for YUM using the command yum install nfs-utils.

As a verification that NFS and networking is configured properly, use the showmount -e command to list mounts from the NFS server:

[root@vcdcell1 transfer]# showmount -e tsfiles.techsol.local
Export list for tsfiles.techsol.local:
/isos (everyone)
/oracle (everyone)
/unix (everyone)
/vcdtransfer (everyone)
/vcdtransfer2 (everyone)
[root@vcdcell1 transfer]#

Next, mount the NFS export manually:

mount nfshost.fqdn.orip:/nfs_export_name /opt/vmware/vcloud-director/data/transfer

Finally, let’s make sure the NFS export auto mounts each time the cell is rebooted.  This is done by editing /etc/fstab

nano -w /etc/fstab

Add the following line to /etc/fstab:

nfshost.fqdn.orip:/nfs_export_name      /opt/vmware/vcloud-director/data/transfer       nfs     rw      0 0

Exit nano using CTRL + X. Save /etc/fstab when prompted.

Proceed with the vCloud Director cell installation.  If using the mount path in the example above, it is safe and convenient to press Enter through the default prompt relating to the Transfer Server Storage installation path.

I’ll close by pointing out that although the Transfer Server Storage is used as a temporary holding tank for vApp and catalog media imports and exports, critical cell data is also stored in this repository.  If the Transfer Server Storage area is unavailable (ie. issues with NFS or the network), the VCD cell will not function properly, yielding a range of symptoms such as not being able to authenticate at the provider or organization portals.

vSphere 5.0 Update 1 and Related Product Launches

March 16th, 2012

VMware has unveiled a point release update to several of their products tied to the vSphere 5 virtual cloud datacenter platform plus a few new product launches.

vCenter 5.0 Update 1 – Added support for new guest operating systems such as Windows 8, Ubuntu, and SLES 11 SP2, the usual resolved issues and bug fixes, plus some updates around vRAM limits licensing.  One other notable – no compatibility at this time with vSphere Data Recovery (vDR) 2.0 according to the compatibility matrix.

ESXi 5.0 Update 1 – Added support for new AMD and Intel processors, Mac OS X Server Lion, updated chipset drivers, resolved issues and bug fixes.  One interesting point to be made here is that according to the compatibility matrix, vCenter 5.0 supports ESXi 5.0 Update 1.  I’m going to stick with the traditional route of always upgrading vCenter before upgrading hosts as a best practices habit until something comes along to challenge that logic.

vCloud Director 1.5.1 – Added support for vSphere 5.0 Update 1 and vShield 5.0.1, plus RHEL 5 Update 7 as a supported server cell platform.  Enhancements were made around firewall rules, AMQP system notifications, log collection, chargeback retention, resolved issues, and added support for AES-256 encryption on Site-to-Site VPN tunnels (unfortunately no vSphere 5.0 Update 1 <-> vCloud Connector 1.5 support).  Oh yes, sometime over the past few months, VMware Marketing has quietly changed the acronym for vCloud Director from vCD to VCD.  We’ll just call that a new feature for 1.5.1 going forward.  I <3 the Marketing team.

Site Recovery Manager 5.0.1 – Added support for vSphere 5.0 Update 1 plus a “Forced Failover” feature which allows VM recovery in cases where storage arrays fail at the protected site which, in the past, lead to unmanageable VMs which cannot be shut down, powered off, or unregistered.  Added IP customization for some Ubuntu platforms.  Many bug fixes, oh yes.  VMware brought back an advanced feature which hasn’t been seen since SRM 4.1 which provided a configurable option, storageProvider.hostRescanCnt, allowing repeated host scans during testing and recovery. This option was removed from SRM 5.0 but has been restored in the Advanced Settings menu in SRM 5.0.1 and can be particularly useful in troubleshooting a failed Recovery Plan. Right-click a site in the Sites view, select Advanced Settings, then select storageProvider. See KB 1008283.  Storage arrays certified on SRM 5.0 (ie. Dell Compellent Storage Center) are automatically certified on SRM 5.0.1.

View 5.0.1 – Added support for vSphere 5.0 Update 1, new Connection Server, Agent, Clients, fixed known issues.  Ahh.. let’s go back to that new clients bit.  New bundled Mac OS X client with support for PCoIP!  I don’t have a Mac so those who would admit to calling me a friend will have to let me know how sharp that v1.4 Mac client is.  As mentioned in earlier release notes, Ubuntu got a plenty of love this week.  Including a new View PCoIP version 1.4 client for Ubuntu Linux.  I might just have to deploy an Ubuntu desktop somewhere to test this client.  But wait, there’s more.  New releases of the View client for Android and iPad tablets.  The Android client adds fixes for Ice Cream Sandwich devices, security stuff, and updates for the Kindle Fire (I need to get this installed on my wife’s Fire).  The updated iPad client improves both connection times as well as external display support but for the most part Apple fans are flipping out simply over something shiny and new.  Lastly, VMware created a one stop shop web portal for all client downloads which can be fetched at

vShield 5.0.1 – Again, added support for vSphere 5.0 Update 1, enhanced reporting and export options, new REST API calls, improved audit logs, simplified troubleshooting, improved vShield App policy management as well as HA enhancements, and enablement of Autodeploy through vShield VIB host modules downloadable from vShield Manager.

So… looking at the compatibility matrix with all of these new code drops, my lab upgrade order will look something like this:

1a. View 5.0 –> View 5.0.1

1b. vCD 1.5 –> VCD 1.5.1

1c. SRM 5.0 –> SRM 5.0.1

1d. vShield App/Edge/Endpoint 5.0 –> 5.0.1

1e. vDR 2.0 –> Go Fish

2. vSphere Client 5.0.1 (it’s really not an upgrade, installs parallel with other versions)

3. vCenter Server 5.0 –> vCenter Server 5.0 Update 1

4. Update Manager 5.0 –> Update Manager 5.0 Update 1

5. ESXi 5.0 –> ESXi 5.0 Update 1

There are a lot of versions in play here which weaves somewhat of a tangled web of compatibility touch points to identify before diving head first into upgrades.  I think VMware has done a great job this time around with releasing products that are, for the most part, compatible with other currently shipping products which provides more flexibility in tactical approach and timelines.  Add to that, some time ago they’ve migrated a two dimensional .PDF based compatibility matrix into an online portal offering interactive input making the set of results customized for the end user.  The only significant things missing in the vSphere 5.0U1 compatibility picture IMO are vCloud Connector, vDR, and based on the results from the compatibility matrix portal – vCenter Operations (output showed no compatibility with vSphere 5.x, didn’t look right to me).  I’ve taken a liberty in creating a component compatibility visual roadmap including most of the popular and currently shipping products vSphere 5.0 and above.  If you’ve got a significant amount of infrastructure to upgrade, this may help you get the upgrade order sorted out quickly.  One last thing – Lab Manager and ESX customers should pay attention to the Island of Misfit Toys.  In early 2013 the Lab Manager ride comes coasting to a stop.  Lab Manager and ESX customers should be formulating solid migration plans with an execution milestone coming soon.

Snagit Capture

VMware vCenter as a vCloud Director vApp

February 27th, 2012

Snagit CaptureThe way things work out, I tend to build a lot of vCenter Servers in the lab.  Or at least it feels like I do.  I need to test this.  A customer I’m meeting with wants to specifically see that.  I need don’t want to taint or impact an existing vCenter Server which may already be dedicated to something else having more importance.  VMware Site Recovery Manager is a good example.  Each time I bring up an environment I need a pair of vCenter Servers which may or not be available.  Whatever the reason, I’ve reached the point where I don’t need to experience the build process repeatedly.

The Idea

A while ago, I had stood up a private cloud for the Technical Solutions/Technical Marketing group at Dell Compellent.  I saved some time by leveraging that cloud environment to quickly provision platforms I could install vCenter Server instances on.  vCenter Servers as vApps – fantastic use case.  However, the vCenter installation process is lengthy enough that I wanted something more in terms of automated cookie cutter deployment which I didn’t have to spend a lot of time on.  What if I took one of the Windows Server 2008 R2 vApps from the vCD Organization Catalog, deployed it as a vApp, bumped up the vCPU and memory count, installed the vSphere Client, vCenter Server, licenses, a local MS SQL Express database, and the Dell Compellent vSphere client plug-in (download|demo video), and then added that vApp back to the vCD Organization Catalog?  Perhaps not such a supported configuration by VMware or Microsoft, but could I then deploy that vApp as future vCenter instances?  Better yet, build a vApp consisting of a pair of vCenter Servers for the SRM use case?  It sounded feasible.  My biggest concerns were things like vCenter and SQL Express surviving the name and IP address change as part of the vCD customization.


Although I ran into some unrelated customization issues which seemed to have something to do with vCD, Windows Server 2008 R2, and VMXNET3 vNICs (error message: “could not find network adapters as specified by guest customization. Log file is at c:\windows\temp\customize-guest.log.” I’ll save that for a future blog post if I’m able to root cause the problem), the Proof of Concept test results thus far have been successful.  After vCD customization, I was able to add vSphere 5 hosts and continue with normal operations from there.

Initially, I did run into one minor issue and that was hosts would fall into a disconnected status approximately two minutes after being connected to the vCenter Server.  This turned out to be a Windows Firewall issue which was introduced during the customization process.  Also, there were some red areas under the vCenter Service Status which pointed to the old instance name (most fixes for that documented well by Rick Vanover here, plus the vCenter Inventory Service cleanup at VMware KB 2009934).

The Conclusion

To The Cloud!  You don’t normally hear that from me on a regular basis but in this case it fits.  A lengthy and increasingly cumbersome task was made more efficient with vCloud Director and vSphere 5.  Using the Linked Clone feature yields both of its native benefits: Fast Provisioning and Space Efficiency.  I’ll continue to leverage vCD for similar and new use cases where I can.  Lastly, this solution can also be implemented with VMware Lab Manager or simply as a vSphere template.  Caveats being Lab Manager retires in a little over a year and a vSphere Template won’t be as space efficient as a Linked Clone.

vCloud Director and vCenter Proxy Service Failure

December 16th, 2011

In the past couple of weeks I have spent some time with VMware vCloud Director 1.5.  The experience yielded three blog articles Collecting diagnostic information for VMware vCloud Director, Expanding vCloud Director Transfer Server Storage and now this one.

In this round, the vCD cell stopped working properly (single cell server environment).  I could log into the vCD provider and organization portals but the deployment of vApps would run for an abnormally long time and then fail after 20 minutes with one of the resulting failure messages being Failed to receive status for task.

Doing some digging in the environment I found a few problems.

Problem #1:  None of the cells have a vCenter proxy service running on the cell server.

Snagit Capture

Problem #2:  Performing a Reconnect on the vCenter Server object resulted in Error performing operation and Unable to find the cell running this listener.

Snagit Capture

Snagit Capture

I search the Community Forums, talked with Chris Colotti (read his blog) for a bit, and then opened an SR with VMware support.  VMware sent me a procedure along with a script to run on the Microsoft SQL Server:

  1. BACKUP the entire SQL Database.
  2. Stop all cells. (service vmware-vcd stop)
  3. Run the attached reset_qrtz_tables_sql_database.sql
    — shutdown all cells before executing
    delete from qrtz_scheduler_state
    delete from qrtz_fired_triggers
    delete from qrtz_paused_trigger_grps
    delete from qrtz_calendars
    delete from qrtz_trigger_listeners
    delete from qrtz_blob_triggers
    delete from qrtz_cron_triggers
    delete from qrtz_simple_triggers
    delete from qrtz_triggers
    delete from qrtz_job_listeners
    delete from qrtz_job_details
  4. Start one cell and verify if issue is resolved. (service vmware-vcd start)
  5. Start the remaining cells.

Before running the script I knew I had to make a few modifications to select the vCloud database first.

When running the script, it failed due to case sensitivity with respect to the table names.  Upon installation, vCD creates all tables with upper case names.  When the MS SQL Server database was first created by yours truly, case sensitivity, along with accent sensitivity, were enabled with COLLATE Latin1_General_CS_AS which comes straight from page 17 of the vCloud Director Installation and Configuration Guide.

After fixing the script, it looked like this:

USE [vcloud]

— shutdown all cells before executing
delete from QRTZ_CALENDARS
delete from QRTZ_TRIGGERS
delete from QRTZ_JOB_DETAILS

The script ran successfully wiping out all rows in each of the named tables.  A little sidebar discussion here.. I talked with @sqlchicken (Jorge Segarra, read his blog here) about the delete from statements in the script. It is sometimes a best practice to use the truncate table statement instead so that the transaction logs are bypassed instead of using the delete from statement which is more resource intensive due to the row by row deletion method and the rows being recorded in the transaction logs. Thank you for that insight Jorge! More on MS SQL Delete vs Truncate here. Jorge was also kind enough to provide a link on the subject matter but credentials will be required to view the content.

I was now able to restart the vCD cell and my problems were gone. Everything was working again. All errors have vanished.  I thanked the VMware support staff and then tried to gain a little bit more information about how the problem was resolved by deleting table rows and what exactly are the qrtz tables?  I had looked at the table rows myself before they were deleted and the information in there didn’t make a lot of sense to me (but that doesn’t necessarily classify it as transient data).  This is what he had to say:

These [vCenter Proxy Service] issues are usually caused by a disconnect from the database, causing the tables to become stale. vCD constantly needs the ability to write to the database and when it cannot, the cell ends up in a state that is similar to the one that you have seen.

The qrtz tables contain information that controls the coordinator service, and lets it know when the coordinator to be dropped and restarted, for cell to cell fail over to another cell in multi cell enviroment.

When the tables are purged it forces the cell on start up to recheck its status and start the coordinator service. In your situation the cell, due to corrupt records in the table was not allowing this to happen.

So by clearing them forced the cell to recheck and to restart the coordinator.

Good information to know going forward. I’m going to keep this in my back pocket. Or on my blog as it were.  Have a great weekend!

Collecting diagnostic information for VMware vCloud Director

December 12th, 2011

I’ve gone a few rounds with VMware vCloud Director in as many weeks recently.  I’ve got an upcoming blog post on a vCenter Proxy Service issue I’ve been dealing with but until I collect the remaining details on that, I thought I’d point out VMware KB 1026312 Collecting diagnostic information for VMware vCloud Director.  This knowledge base article details the steps required to collect the necessary support logs for both vCD versions 1.0 and 1.5.

The vmware-vcd-support script collects host log information as well as these vCloud Director logs. The script is located in the following folders:

For vCloud Director 1.0, run /opt/vmware/cloud-director/bin/vmware-vcd-support

For vCloud Director 1.5, run /opt/vmware/vcloud-director/bin/vmware-vcd-support

Once executed, the script will bundle the following log files from /opt/vmware/vcloud-director/logs/ into a .tgz tarball saving it in the directory from which the script was run (providing there is enough storage available):

  1. cell.log – Console output from the vCloud Director cell.
  2. diagnostics.log – Cell diagnostics log. This file is empty unless diagnostics logging is enabled in the local logging configuration.
  3. vcloud-container-info.log – Informational log messages from the cell. This log also shows warnings or errors encountered by the cell.
  4. vcloud-container-debug.log – Debug-level log messages from the cell.
  5. vcloud-vmware-watchdog.log – Informational log messages from the cell watchdog. It records when the cell crashes, is restarted, etc.

On the subject of vCD log files, also mentioned in the KB article is VMware KB 1026815 Configuring logging for VMware vCloud Director.  The information in this article is useful for specifying the quantity and size of vCD log files to be maintained on the cell server.

Once the log files have been collected, you may analyze them offline or upload them to VMware’s FTP site in association with an SR by following VMware KB 1008525 Uploading diagnostic information to VMware.