Archive for August, 2015

vCloud Director vdnscope-1 could not be found

August 15th, 2015

For whatever reason, I’ve been spending a pretty fair amount of time lately with vCloud Director both at home as well as at the office. It’s a great product. It always has been, beginning with its Lab Manager roots. Like my last blog post, this writing will exhibit another vCloud Director database editing exercise which stemmed from a problem I encountered in the lab.

I was attempting to get away from my VLAN-backed Network Pool by configuring vCloud Director’s Provider vDC-VXLAN-NP Network Pool which is much more dynamic and powerful in nature. The Provider vDC-VXLAN-NP Network Pool is installed by default in vCloud Director but to configure and use it for Organization and vApp networks, one must follow a set of instructions which basically involves configuring upstream physical switch(es) with jumbo frames, a transport VLAN, and multicast settings, preparing the hosts by installing an agent on each of them using vShield Manager, adding VMkernel ports, Network Scopes, Virtual Wires, and so on (Mike Laverick and Rawlinson Rivera both have easy to follow tutorials. The VMware VXLAN Deployment Guide is also a great read). Once it’s all set up and working, VXLAN is pretty effing cool. Anyway, it sounds like a lot of steps and admittedly it requires some reading and attention to detail, but much of it is automated by vCloud Director, with some bumps along the way.

I did run into a few snags which ultimately lead me to going through the configuration process start to finish a few times. In the end I had to configure the Network Scope in vShield Manager manually when normally this step is performed automatically by vCloud Director via the Enable VXLAN Provider VDC right-click menu item.

Once I got beyond the installation hurdles, there was some residual impact left in the vCloud Director database and vShield Manager such that it all looked to be working properly, except that at the very end I could not power on a vApp with an isolated vApp network which relied on the use of the VXLAN-backed Network Pool. The error message was:

Cannot deploy organization VDC network  (uuid for that network)
com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : vdnscope-1 could not be found. Object identifiers are case sensitive.

[ bb505f5e-27f1-419e-9b05-da0d38a7788f ] Unable to deploy network “vApp net1(urn:uuid:7d813867-d3f1-420d-a0a8-a65263369327)”.

com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : vdnscope-1 could not be found. Object identifiers are case sensitive.

– com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : vdnscope-1 could

not be found. Object identifiers are case sensitive.

– VSM response error (202): The requested object : vdnscope-1 could not be found. Object identifiers are case sensitive.

An object named vdnscope-1 seems to be the obvious problem.

I was not able to make use of the Network Pool Repair function as it was unavailable:

Fortunately I was able to locate a related thread in the VMware Communities which more or less explained what might have happened and what I could try to fix the problem (credit to IamTHEvilONE). This is my interpretation.

Each time a Network Scope is created in the vShield Manager, an underlying object reference is tied to the Network Scope with a naming convention of vdnscope-x where x begins at 1 and is incremented at each create iteration. So the first Network Scope created in vShield Manager by vCloud Director is going to be called vdnscope-1. This object is stored in the vCloud Director database and is referenced each time an Org or vApp network is spun up which leans on the VXLAN-backed Network Pool. This is formally handled at vApp power on. The object is also stored somewhere in the vShield Manager although I was never able to locate it. What happened here is that Network Scope object known by both vCloud Director and vShield Manager were not sync and didn’t match. vCloud Direct dials up vShield Manager and says “I need that vdnscope-1 you have” and vShield Manager responds with “I have no idea what that object is”. Obvious problem.

The solution is fairly simple: Edit the vCloud Director database with the correct Network Scope object reference. But a small problem still remains: I was never able to locate the correct object name in vShield Manager. However, going back to the VMware Communities discussion, I’ll eventually be able to find the correct object name by incrementing the vdnscope-x object reference in the vCloud Director database by 1 until the two sides agree and the vApp powers on successfully.

I’ll borrow the same disclaimer from the previous blog post: An obligatory warning on vCloud database editing. Do as I say, not as I do. Editing the vCloud database should be performed only with the guidance of VMware support. Above all, create a point in time backup of the vCloud database with all vCloud Director cell servers stopped (service vmware-vcd stop). There are a variety of methods in which you can perform this database backup. Use the method that is most familiar to and works for you.

So after stopping the vCloud Director services and getting a vcloud database backup…

Step 1: Open Microsoft SQL Server Management Studio and navigate to the [vcloud].[dbo].[network_pool] table. Under the vdn_scope_id column, increment the vdnscope-1 value from 1 to 2.

Step 2: Start the vCloud Director service in all cell servers (service vmware-vcd start) and verify in vShield Manager the Virtual Wire has been created and the vApp can power on successfully. If it fails, stop vCloud services and repeat Step 1 above while incrementing the vdnscope value to 3, then 4, and so on. In my case, vdnscope-5 did the trick.

vCloud Director is awesome. VXLAN with 16 million networks capability kicks it up a notch.

Updated 8/22/15: I received a tip from Jon Hemming in the form of a blog comment. Jon states he has written a VMware KB article titled Creating an isolated network in VMware vCloud Director reports the error: vdnscope-x does not exist (2065485) which documents a process to get the correct VDN Scope ID via the REST API of vShield as well as update the vCloud Director database. Thank you Jon! I did find the syntax for the curl statement to be slightly off. The KB article calls for the following syntax:

curl -k -u admin:default -X GET https://vshield.boche.lab/api/2.0/vdn/scopes/

The result is HTTP Status 404 The requested resource is not available.

What did work was:

curl -k -u admin:default -X GET https://vshield.boche.lab/api/2.0/vdn/scopes

The only change was removing the trailing forward slash on the URL.

vCloud Director Error Cannot delete network pool

August 15th, 2015

I ran into a small problem this week in vCloud Director whereby I was unable to Delete a Network Pool. The error message stated Cannot delete network pool because It is still in use. It went on to list In use items along with a moref identifier. This was not right because I had verified there were no vApps tied to the Network Pool. Furthermore the item listed still in use was a dynamically created dvportgroup which also no longer existed on the vNetwork Distributed Switch in vCenter.

I suspect this situation came about due to running out of available storage space earlier in the week on the Microsoft SQL Server where the vCloud database is hosted. I was performing Network Pool work precisely when that incident occurred and I recall an error message at the time in vCloud Director regarding tempdb.

I tried removing state data from QRTZ tables which I blogged about here a few years ago and has worked for specific instances in the past but unfortunately that was no help here. Searching the VMware Communities turned up sparse conversations about roughly the same problem occurring with Org vDC Networks. In those situations, manually editing the vCloud Director database was required.

An obligatory warning on vCloud database editing. Do as I say, not as I do. Editing the vCloud database should be performed only with the guidance of VMware support. Above all, create a point in time backup of the vCloud database with all vCloud Director cell servers stopped (service vmware-vcd stop). There are a variety of methods in which you can perform this database backup. Use the method that is most familiar to and works for you.

Opening up Microsoft SQL Server Management Studio, there are rows in two different tables which I need to delete to fix this. This has to be done in the correct order or else a REFERENCE constraint conflict occurs in Microsoft SQL Server Management Studio and the statement will be terminated.

So after stopping the vCloud Director services and getting a vcloud database backup…

Step 1: Delete the row referencing the dvportgroup in the [vcloud].[dbo].[network_backing] table:

Step 2: Delete the row referencing the unwanted Network Pool in the [vcloud].[dbo].[network_pool] table:

That should take care of it. Start the vCloud Director service in all cell servers (service vmware-vcd start) and verify the Network Pool has been removed.

RHEL 7, open-vm-tools, and guest customization

August 9th, 2015

I spent some time this weekend working with vCloud Director 5.5.4 build 2831206 (on vSphere 6) and Red Hat Enterprise Linux vApp/guest customization. I’m not a *nix guru but I’m comfortable enough with legacy versions of RHEL 5 and 6 as I’ve worked with them quite a bit, particularly for vSphere applications and solutions such as vCloud Director to provide just one example. Quite honestly internet research or peer networking provides supplemental knowledge for whatever I can’t figure out. However I hadn’t spent much time with RHEL 7. There are some new twists and this blog post is an attempt to document what I’ve uncovered to answer questions and hopefully save myself some time in the future. If you’re in a hurry, skip to the “Tying It All Together” section at the end.

vSphere Templates and vCloud vApp Templates

When it comes to vSphere templates that I use myself, I’ll bake in commonly utilized software packages, patches, as well as tweaks and best practices. However, when it comes to shared vApp Templates in a vCloud Catalog, I employ more of a purist philosophy to minimize issues or questions raised regarding the DNA of the OS build I’m sharing with the organization which serves as their base starting point for their vApp. Aside from installing VMware Tools, my Windows 2012 R2 vApp is about as vanilla as it gets. The same can be said for my RHEL 5 and RHEL 6 vApps. When I applied that same approach to RHEL 7, that’s where some noticeable changes became apparent.

The RHEL 7 Minimal Install

The mere existence of this blog post stems from here. The default installation of RHEL 7 is a Minimal Install. While it’s not encumbered with extra software that may never be used depending on the server’s role, it’s also missing packages commonly installed in the past. Some of which are core dependencies in a virtualized datacenter. However, not knowing this, I gladly accepted the opportunity of a minimalist installation. And that’s exactly what I got.

VMware Tools

After completing a rather uneventful RHEL 7 installation, typically the first and last order of business is to install VMware Tools. Those who attempt it on RHEL 7 (as well as other newer versions of *nix such as CentOS 7) will be greeted with rather stern wording that VMware Tools should be avoided and rather the OS provided open-vm-tools should be used instead. VMware support of open-vm-tools (2073803) provides background information, detail, and outlines the benefits of open-vm-tools. It’s not that you can’t install VMware Tools on RHEL 7, you can, but VMware is not recommending it at this point. In the previously linked KB article:

VMware recommends using open-vm-tools redistributed by operating system vendors.

VMware fully supports…

VMware aids in the development of…

VMware does not recommend removing open-vm-tools redistributed by operating system vendors.

Those who choose to install VMware Tools anyway on a RHEL 7 Minimal Install will soon discover that they cannot do so without installing some additional support RHEL 7 packages. VMware Tools cannot be installed on RHEL 7 due to missing ifconfig (2075519) explains that net-tools is missing and must be installed as follows (you’ll need a yum repository; the next section covers that):

#sudo yum install net-tools

I’d also argue that you’re going to need to install supporting PERL packages to execute the /usr/bin/ script because it’s also missing in a RHEL 7 Minimal Install. More on that a little later but for now, the other packages that are needed can be installed as follows:

# yum install perl gcc make kernel-headers kernel-devel -y

Creating A Local DVD Repository For YUM

Without a Red Hat subscription (I fall into this category), or the networking means to reach your subscription on the internet, you’ll need to rely on your RHEL 7 DVD or .iso to install necessary packages such as net-tools mentioned above. In order to access these packages, you’ll need to mount the DVD and create a local DVD repository.

Mounting the DVD:

mount /dev/cdrom /mnt/

Creating the local DVD repository is slightly more involved but the steps are easy to follow. Create the file /etc/yum.repos.d/dvd.repo. The file should contain the following text:


The local DVD repository is now available and its existence can now be queried (note that it only remains available for as long as the RHEL 7 DVD is mounted):

yum repolist all

An example of installing a yum package is shown above although it does not always require the use of sudo.

Open VM Tools

Red Hat Enterprise Linux 7 Guest Operating System Installation Guide documents the process of installing open-vm-tools. Remember that open-vm-tools is distributed by the OS vendor so everything you need from that respect is available from the RHEL 7 DVD and the local DVD repository created above. That said, installing open-vm-tools is straightforward:

# yum install open-vm-tools

Verify open-vm-tools has been installed in the guest:

# yum search open-vm-tools

With open-vm-tools installed, the guest now has the following vSphere feature functionality:

  • Synchronization of the guest OS clock with the virtualization platform
  • Enables the virtual infrastructure to perform graceful power operations (shut down) and file system quiescing of the virtual machine
  • Provides a heartbeat from guest to the virtualization infrastructure to support vSphere High Availability (HA)
  • Publishes information about the guest OS to the virtualization platform, including resource utilization and networking information
  • Provides a secure and authenticated mechanism to perform various operations within the guest OS from the virtualization infrastructure
  • Accepts additional plug-ins that can extend or customize open-vm-tools functionality

Guest customization and the deployPkg Tools Plug-in

Looking at the bulleted list above, a number of features are provided by open-vm-tools. Unfortunately guest customization isn’t one of them (guest customization is typically used in deploying templates in vSphere as well as deploying available vApps from a vCloud Director catalog). At this point if you attempt to clone a RHEL 7 guest with open-vm-tools, you’ll get the exact same VM over and over again with no unique guest customization. The last bullet speaks to a plug-in architecture for which a guest customization plug-in is available from VMware called the deployPkg Tools Plug-in.

Red Hat Enterprise Linux 7 Guest Operating System Installation Guide talks about the plug-in and while it appears to provide the installation instructions, it’s missing a few required steps for installing the VMware Packaging Public Keys so refer to Installing the deployPkg plug-in in a Linux virtual machine (2075048) for the correct process. In this process, yum will be used to install a package available via the internet from VMware instead of from the local DVD repository described previously.

Download the two VMware Packaging Public Keys from VMware at

Copy them to /tmp/ on the RHEL 7 guest

Import each of the two keys (that’s a double dash in front of import):

# rpm –import /tmp/

# rpm –import /tmp/

Create the yum repository by creating a file called /etc/yum.repos.d/vmware-tools.repo containing the following text:

name = VMware Tools
baseurl =
enabled = 1
gpgcheck = 1

Execute the command

sudo yum install open-vm-tools-deploypkg

Followed by

sudo systemctl restart vmtoolsd

At this point, both open-vm-tools from Red Hat as well as open-vm-tools-deploypkg from VMware have been installed and guest customization should work and you’d be done, except…

RHEL 7 Guest Customization Fails Because The Minimal Install Is Missing PERL

Under the RHEL 7 Minimal Install, guest customization still does not produce unique VMs during a cloning process. Taking a look at the clone in /var/log/vmware-imc/toolsDeployPkg.log, I noticed the following:

Launching deployment /usr/bin/perl -I/tmp/.vmware/linux/deploy/scripts /tmp/.vmware/linux/deploy/scripts/ /tmp/.vmware/linux/deploy/cust.cfg.

Command to exec : /usr/bin/perl

Customization command output:

Deploy error: Deployment failed. The forked off process returned error code.

Package deploy failed in DeployPkg_DeployPackageFromFile

The folder /usr/bin/perl/ does not exist.

So then where is PERL? I already know the answer before I’m told.. it doesn’t exist under a RHEL 7 Minimal Install

[root@localhost ~]# whereis perl
perl:[root@localhost ~]#

Install PERL from the RHEL 7 local DVD repository. This installation should be performed on the template or vApp before it’s placed into the catalog so that the resulting guest customization works (obviously it has little effect on a guest customization which has already failed):

# yum install perl gcc make kernel-headers kernel-devel -y

PERL is now installed and can be called upon for guest customization:

[root@localhost ~]# whereis perl
perl: /usr/bin/perl /usr/share/man/man1/perl.1.gz
[root@localhost ~]#

RHEL 7 Guest Agents

The RHEL 7 Minimal Install turned out to be a bit of learning process. A more streamlined approach, if available, would be to utilize the Infrastructure Server base environment during the RHEL 7 installation instead of the Minimal Install. Infrastructure Server is going to automatically include PERL and net-tools. It’s also going to expose the ability to install the Guest Agents Add-On. It’s talked about in full in the Red Hat Enterprise Linux 7 Guest Operating System Installation Guide. Installing the Guest Agents includes open-vm-tools from the RHEL 7 DVD without the extra steps of manually creating the RHEL 7 local DVD repository.

While this is certainly more efficient, the one remaining caveat is that Guest Agents does not include the deployPkg Tools Plug-in from VMware. The plug-in will still need to be manually installed from the VMware repository if customization of the VM or vApp is required. For templates, this is almost always a necessity.

RHEL 7 Networking

One last note is that networking in RHEL 7 has seen some changes. For openers, legacy device names such as eth0, eth1, etc. are replaced by a profile name such as eno16780032 (the corresponding files reflect these name changes in /etc/sysconfig/network-scripts/). Menu driven network configuration (previously accessed from setup) has been replaced by a Network Manager which is accessible via nmtui (Network Manager Text User Interface), nmcli (Network Manager Command Line Interface), or Network Scripts. Also recall from the top of the article that the old standby ifconfig will not be present under a Minimal Install – it requires the net-tools package. Last but not least, detected Ethernet adapters in a Minimal Installation are not automatically enabled for use. Discovered Ethernet devices can be enabled during the initial RHEL 7 setup (I believe it’s under Hostname and Network), or it can be enabled after installation by running nmtui and check the Automatically Connect box for the appropriate Edit a connection menu. A detected Ethernet adapter listing can be obtained at any time with nmcli d.

Although this article was specific RHEL 7, open-vm-tools is available with the following operating systems as documented by VMware support of open-vm-tools (2073803)

  • Fedora 19 and later releases
  • Debian 7.x and later releases
  • openSUSE 11.x and later releases
  • Recent Ubuntu releases (12.04 LTS, 13.10 and later)
  • Red Hat Enterprise Linux 7.0 and later releases
  • CentOS 7 and later releases
  • Oracle Linux 7 and later releases
  • SUSE Linux Enterprise 12 and later releases

RHEL 7 Templates – Tying It All Together

In the end, there are a few different Base Environment types available with varying steps for building a RHEL 7 image which supports guest customization in vSphere or vCloud Director.

Minimal Install (default)

  1. Choose Minimal Install Base Environment
  2. Enable Ethernet card to automatically connect (at install or later using nmtui)
  3. Add RHEL 7 local DVD repository
  4. Install net-tools
  5. Install PERL
  6. Install open-vm-tools
  7. Add yum repository for VMware
  8. Install the deployPkg Tools Plug-in

Infrastructure Server

  1. Choose Infrastructure Server Base Environment and Guest Agents Add-On (open-vm-tools will automatically be installed)
  2. Enable Ethernet card to automatically connect (at install or later using nmtui)
  3. Add yum repository for VMware
  4. Install the deployPkg Tools Plug-in

Clearly the Minimal Install route has more steps while the Infrastructure Server route has less steps and is quicker. Regardless of Base Environment type, VMware does not recommend the installation of VMware Tools.

I’ve linked several resources throughout this article. Just about all of the information was available, it was merely a matter of finding and reading the relevant documentation which isn’t always in one place. The only dots I had to connect on my own which I didn’t see mentioned anywhere was the lack of PERL for the deployPkg Tools Plug-in from VMware as well as for the installation of VMware Tools on RHEL 7 which isn’t recommended by VMware.

Update 8/22/15: vCloud Director guest customization is also problematic with CentOS 7 but with one additional hang up. I’ve found several references on the internet with the workaround including one I’ll link here from my good friend Bob Plankers/etc/redhat-release must read Red Hat Enterprise Linux Server release 7.0 (Maipo)

Update 9/11/15: Brian Graf authored a nice piece yesterday titled Open-VM-Tools (OVT): The Future of VMware Tools for Linux which anyone who wound up here should find interesting.