Posts Tagged ‘Windows’

VMware Tools causes virtual machine snapshot with quiesce error

July 30th, 2016

Last week I was made aware of an issue a customer in the field was having with a data protection strategy using array-based snapshots which were in turn leveraging VMware vSphere snapshots with VSS quiesce of Windows VMs. The problem began after installing VMware Tools version 10.0.0 build-3000743 (reported as version 10240 in the vSphere Web Client) which I believe is the version shipped in ESXI 6.0 Update 1b (reported as version 6.0.0, build 3380124 in the vSphere Web Client).

The issue is that creating a VMware virtual machine snapshot with VSS integration fails. The virtual machine disk configuration is simply two .vmdks on a VMFS-5 datastore but I doubt the symptoms are limited only to that configuration.

The failure message shown in the vSphere Web Client is “Cannot quiesce this virtual machine because VMware Tools is not currently available.”  The vmware.log file for the virtual machine also shows the following:

2016-07-29T19:26:47.378Z| vmx| I120: SnapshotVMX_TakeSnapshot start: ‘jgb’, deviceState=0, lazy=0, logging=0, quiesced=1, forceNative=0, tryNative=1, saveAllocMaps=0 cb=1DE2F730, cbData=32603710
2016-07-29T19:26:47.407Z| vmx| I120: DISKLIB-LIB_CREATE : DiskLibCreateCreateParam: vmfsSparse grain size is set to 1 for ‘/vmfs/volumes/51af837d-784bc8bc-0f43-e0db550a0c26/rmvm02/rmvm02-000001.
2016-07-29T19:26:47.408Z| vmx| I120: DISKLIB-LIB_CREATE : DiskLibCreateCreateParam: vmfsSparse grain size is set to 1 for ‘/vmfs/volumes/51af837d-784bc8bc-0f43-e0db550a0c26/rmvm02/rmvm02_1-00000
2016-07-29T19:26:47.408Z| vmx| I120: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
2016-07-29T19:26:56.292Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
2016-07-29T19:27:07.790Z| vcpu-0| I120: Tools: Tools heartbeat timeout.
2016-07-29T19:27:11.294Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
2016-07-29T19:27:17.417Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
2016-07-29T19:27:17.417Z| vmx| I120: Msg_Post: Warning
2016-07-29T19:27:17.417Z| vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
2016-07-29T19:27:17.417Z| vmx| I120: —————————————-
2016-07-29T19:27:17.420Z| vmx| I120: Vigor_MessageRevoke: message ‘msg.snapshot.quiesce.rpc_timeout’ (seq 10949920) is revoked
2016-07-29T19:27:17.420Z| vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
2016-07-29T19:27:17.420Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot ‘jgb': 0
2016-07-29T19:27:17.420Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (31).
2016-07-29T19:27:17.420Z| vmx| I120: VigorTransport_ServerSendResponse opID=ffd663ae-5b7b-49f5-9f1c-f2135ced62c0-95-ngc-ea-d6-adfa seq=12848: Completed Snapshot request.
2016-07-29T19:27:26.297Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

After performing some digging, I found VMware had released VMware Tools version 10.0.9 on June 6, 2016. The release notes identify the root cause has been identified and resolved.

Resolved Issues

Attempts to take a quiesced snapshot in a Windows Guest OS fails
Attempts to take a quiesced snapshot after booting a Windows Guest OS fails

After downloading and upgrading VMware Tools version 10.0.9 build-3917699 (reported as version 10249 in the vSphere Web Client), the customer’s problem was resolved. Since the faulty version of VMware Tools was embedded in the customer’s templates used to deploy virtual machines throughout the datacenter, there were a number of VMs needing their VMware Tools upgraded, as well as the templates themselves.

Dell Compellent Storage Center Command Set Shell cmdlets

January 9th, 2015

If you manage Dell Compellent storage, you may or may not be aware that Windows PowerShell cmdlets are available to ease management pain by way of automation and consistency. While I am able to recognize when scripting is the right tool for the job, I do not author PowerShell scripts on a regular basis. For that reason, I’m not as deeply familiar with all of the cmdlets available within the Dell Compellent Storage Center Command Set Shell as I would like to be.

So how do I get started – what are the cmdlets? There are a few different ways to retrieve a list of cmdlets made available by a PowerShell snapin or module.

VMware vSphere PowerCLI simplifies the process by providing a cmdlet called Get-VICommand. When executed, it returns a list of all the cmdlets provided by the VMware.VimAutomation.Core snapin used to manage a vSphere environment via PowerShell. As of this writing in the 5.5.x generation of vSphere, there are a few other vSphere specific snapins installed with PowerCLI but the cmdlets provided by those aren’t returned by Get-VICommand. Those snapins are:

  • VMware.VimAutomation.Vds – This Windows PowerShell snap-in contains cmdlets that let you manage vSphere Distributed Switches.
  • VMware.VimAutomation.License – This Windows Powershell snap-in contains cmdlets for managing License components.
  • VMware.DeployAutomation – Cmdlets for Rule-Based-Deployment
  • VMware.ImageBuilder – This Windows PowerShell snap-in contains VMware ESXi Image Builder cmdlets used to generate custom images.
  • VMware.VimAutomation.Cloud – This Windows Powershell snap-in contains cmdlets used to manage VMware vCloud Director.

However, not all PowerShell snapins ship with a native shortcut to retrieve a list of their respective cmdlets. In these cases, use Get-Command. Now Get-Command by itself returns cmdlets for all snapins. For a snapin specific list, either of the following will work:

Get-Command -Module “snapin_name”
Get-Command | Where-Object{$_.PSSnapin.Name -eq “snapin_name”}

In the case of Dell Compellent Storage Center Command Set Shell, the snapin is named Compellent.StorageCenter.PSSnapin. To retrieve a list of Dell Compellent cmdlets, use one of the following:

Get-Command -Module “Compellent.StorageCenter.PSSnapin”
Get-Command | Where-Object{$_.PSSnapin.Name -eq “Compellent.StorageCenter.PSSnapin”}

At the time of this writing, there are 105 cmdlets:

Get-Command -Module Compellent.StorageCenter.PSSnapin | Measure-Object

Count    : 105
Average  :
Sum      :
Maximum  :
Minimum  :
Property :

Those who don’t use PowerShell on a regular basis may find the above difficult to easily recall from memory. I had a discussion with Justin Braun (author of The Braun Blog – check out his Dell Compellent articles here) and Mike Matthews (a peer in my office who specialize in Microsoft SQL Server, PowerShell, and is an all around good guy). Is there an easier and persistent method to retrieve cmdlets from a given snapin? What resulted was a function that can be added to a PowerShell profile which performs just like VMware’s Get-VICommand (I’ll be original and call this one Get-SCCommand to get the list of Storage Center cmdlets).

Edit the PowerShell profile (%profile). It’s default location is:


If the path and profile doesn’t already exist, it can be created in PowerShell using the following cmdlet:

new-item -itemtype file -path $profile -force

If using PowerShell ISE, the default profile location is:


Add the following to verify the Dell Compellent snapin is loaded. If not, load it.

If ( !( Get-PSSnapin | Where-Object { $_.Name -eq “Compellent.StorageCenter.PSSnapin” } ) )
Add-PSSnapin Compellent.StorageCenter.PSSnapin | Out-Null

Add the Get-SCCommand shortcut function:

Function Get-SCCommand { Get-Command -Module “Compellent.StorageCenter.PSSnapin” }

Save the profile.

Now open any PowerShell environment and use Get-SCCommand which shows a list of 105 Dell Compellent cmdlets (There are 49 additional cmdlets in the compellent.replaymanager.scripting snapin for Replay Manager):

It works with PowerShell ISE as well when the Microsoft.PowerShellISE_profile.ps1 profile is modified:

How about PowerGUI? Yes…

Of course the shortcut function provided in the example above is specific to the Dell Compellent snapin but it should work for for any PowerShell snapin including the list of VMware snapins not included in Get-VICommand discussed at the top of the article.

For more on scripting Storage Center, visit the Dell Storage PowerShell Community. Rick Gouin also has a nice collection of scripts that he has authored.

Have a great weekend!

vCloud Director, RHEL 6.3, and Windows Server 2012 NFS

July 16th, 2013

One of the new features introduced in vCloud Director 5.1.2 is cell server support on the RHEL 6 Update 3 platform (you should also know that cell server support on RHEL 5 Update 7 was silently removed in the recent past – verify the version of RHEL in your environment using cat /etc/issue).  When migrating your cell server(s) to RHEL 6.3, particularly from 5.x, you may run into a few issues.

First is the lack of the libXdmcp package (required for vCD installation) which was once included by default in RHEL 5 versions.  You can verify this at the RHEL 6 CLI with the following command line:

yum search libXdmcp


yum list |grep libXdmcp

Not to worry, the package is easily installable by inserting/mounting the RHEL 6 DVD or .iso, copying the appropriate libXdmcp file to /tmp/ and running either of the following commands:

yum install /tmp/libXdmcp-1.0.3-1.el6.x86_64.rpm


rpm -i /tmp/libXdmcp-1.0.3-1.el6.x86_64.rpm

Next up is the use of Windows Server 2012 (or Windows 8) as NFS for vCloud Transfer Server Storage in conjunction with the newly supported RHEL 6.3.  Creating the path and directory for the Transfer Server Storage is performed during the initial deployment of vCloud Director using the command mkdir -p /opt/vmware/vcloud-director/data/transfer. When mounting the NFS export for Transfer Server Storage (either manually or via /etc/fstab f.q.d.n:/vcdtransfer/opt/vmware/vcloud-director/data/transfer nfs rw 0 0 ), the mount command fails with error message mount.nfs: mount system call failed. I ran across this in one particular environment and my search turned up Red Hat Bugzilla – Bug 796352.  In the bug documentation, the problem is identified as follows:

On Red Hat Enterprise Linux 6, mounting an NFS export from a Windows 2012 server failed due to the fact that the Windows server contains support for the minor version 1 (v4.1) of the NFS version 4 protocol only, along with support for versions 2 and 3. The lack of the minor version 0 (v4.0) support caused Red Hat Enterprise Linux 6 clients to fail instead of rolling back to version 3 as expected. This update fixes this bug and mounting an NFS export works as expected.

Further down in the article, Steve Dickson outlines the workarounds:

mount -o v3 # to use v3


Set the ‘Nfsvers=3′ variable in the “[ Server “Server_Name” ]”
section of the /etc/nfsmount.conf file
An Example will be:
[ Server “nfsserver.lab.local” ]

The first option works well at the command line but doesn’t lend itself to /etc/fstab syntax so I opted for the second option which is to establish a host name and NFS version in the /etc/nfsmount.conf file.  With this method, the mount is attempted as called for in /etc/fstab and by reading /etc/nfsmount.conf, the mount operation succeeds as desired instead of failing at negotiation.

There is a third option which would be to avoid the use of /etc/fstab and /etc/nfsmount altogether and instead establish a mount -o v3 command in /etc/rc.local which is executed at the end of each RHEL boot process.  Although this may work, it feels a little sloppy in my opinion.

Lastly, one could install the kernel update (Red Hat reports as being fixed in kernel-2.6.32-280.el6). The kernel package update is located here.

QuickPrep and Sysprep

May 2nd, 2013

Those who manage VMware View currently or have used it in the past may be familiar with desktop customization which is required to provide a unique identity on the network for each View Composer VDI session in a pool.  If you’ve got a pretty good Microsoft background, you’re probably already familiar with Sysprep – Microsoft’s tool for customizing Windows server and desktop OS deployments.  VMware View Administrators have an alternative tool which can be used for desktop customization called QuickPrep.  For all intents and purposes, QuickPrep was designed to accomplish many of the same tasks Sysprep did, but the obvious advantage QuickPrep has is that the code and development belongs to VMware and as a result can be tightly integrated with products in VMware’s portfolio.

I was on a call this morning with VMware Senior Technical Trainer Linus Bourque (Twitter: @LinusBourque Blog: Cigars: yes) when he pulled up a table slide which was the result of VMware KB Article 2003797 Differences between QuickPrep and Sysprep.  For those who are curious about the similarities and differences between the two (like me), look no further.

From the KB Article:

QuickPrep is a VMware system tool executed by View Composer during a linked-clone desktop deployment. QuickPrep personalizes each desktop created from the Master Image. Microsoft Sysprep is a tool to deploy the configured operating system installation from a base image. The desktop can then be customized based on an answer script. Sysprep can modify a larger number of configurable parameters than QuickPrep.
During the initial startup of each new desktop, QuickPrep:
  • Creates a new computer account in Active Directory for each desktop.
  • Gives the linked-clone desktop a new name.
  • Joins the desktop to the appropriate domain.
  • Optionally, mounts a new volume that contains the user profile information.
This table lists the main differences between QuickPrep and Sysprep:
Function QuickPrep Sysprep
Removing local accounts No Yes
Changing Security Identifiers (SID) No Yes
Removing parent from domain No Yes
Changing computer name Yes Yes
Joining the new instance to the domain Yes Yes
Generating new SID No Yes
Language, regional settings, date, and time customization No Yes
Number of reboots 0 1 (seal & mini-setup)
Requires configuration file and Sysprep No Yes
Note: A Guest Customization script is required in vCenter Server to use Sysprep. Sysprep is bundled in with Windows 7. For Windows XP, an appropriate Sysprep program needs to be installed on the vCenter Server.
For information on installing Sysprep tools, see Sysprep file locations and versions (1005593).
For more information on the use of Sysprep and the Guest Customisation wizard, see the Customizing Guest Operating Systems and Installing the Microsoft Sysprep Tools sections of the vSphere Virtual Machine Administration Guide.

Microsoft Windows Server 2012 Tips

August 16th, 2012

One of the benefits of working for Dell Compellent is having the privilege to collaborate with some very smart people who are subject matter experts in areas of technology I don’t get as much time to spend time on as I’d like to.  I get to share information with team members about vSphere, as well as Exchange, SQL, *nix, Oracle, and you might have guessed it… Microsoft Windows (including Hyper-V).  One of my colleagues has been working with Windows Server 2012 lately and he drew up a quick guide on some of the findings he had made.  Not only was he gracious enough to share it with his teammates, he was more than happy to share with the community when asked.  When I say community, of course I’m referring to readers of this blog.  So without further to do, here are some Windows Server 2012 (and perhaps even Windows 8) tips to get you started.

Navigating the New Server 2012 GUI

The look and feel of the Server 2012 GUI is quite different than Server 2008. While most of the familiar options and features are still available, the process of getting to them is quite different, and in some cases, more difficult.

Snagit Capture

1)      The “Start” button no longer exists in Server 2012.  To expose Start, jiggle your mouse in the lower left corner of the desktop and the Start option will appear as shown above.  This is a bit cumbersome in RDP sessions and takes some getting used to.

Snagit Capture

2)      The Start Menu presents applications and other options as tiles.

3)      To access Lock and Sign out, click on the User in the upper right for a drop-down menu.

Snagit Capture

4)      To access All Applications, right-click on any tile under Start, and then an options bar will appear at the bottom of the screen.  On this options bar, click on All Apps in the lower right.

Snagit Capture

5)      Under All Apps, you can find all the rest of the familiar (but now more difficult to find) options such as Command Prompt and Run.  To make these more easily accessible, pin them to the taskbar.

Snagit Capture

6)      Another hidden menu exits off the right side of the desktop.  To access it, move your mouse to the far right or lower-right corner of the screen and hold it there for a couple seconds.   Again, this is cumbersome in RDP sessions and takes some getting used to.

7)      As you can see above, the Restart and Shut down options are now buried a few layers deep so accessing them is a bit tedious.   Some customization suggestions below will help alleviate this.

Snagit Capture

8)      To stop the Server Manager window from automatically starting every time you log on, edit the Server Manager Properties and check the box Do not start Server Manager automatically at logon.

Customizations to Facilitate Better User Experience with Server 2012

You may find yourself a little frustrated with the changes introduced with the Server 2012 GUI because many apps/options/tools have been relocated and are therefore more difficult (and more time consuming) to find.

Below are some quick and simple customization changes to “restore” some of the of the Server 2008 look/feel/agility to the 2012 GUI.

1)      The first step is to install the Desktop Experience as found under Features.  Once installed, then the (My) Computer icon can be added back to the desktop.

Snagit Capture

a)      Launch Server Manager from the taskbar.

Snagit Capture

b)      Click on Add roles and features to launch the Add Roles and Features Wizard.  Under Features, check the box for Desktop Experience and then complete the wizard (requires a reboot).

Snagit Capture

c)       After rebooting, from the Desktop, right click and choose Personalize, Change Desktop Icons, and add the desired icons such as Computer and Control Panel.

d)      Right click on the Desktop again, and under View, set icon size to Small, and set Auto Arrange and Sort By options according to your preference.

Snagit Capture

2)      Customize the taskbar by pinning shortcuts for I.E., Run, Command Prompt, and other frequently used apps (as found under Start and All Apps) that you want to be quickly accessible.  For directions on how to access the Start and All Apps menus, see Page 2.

3)      Right click on the taskbar, select Properties, and select Use Small taskbar buttons, and under the Toolbars tab, add the Desktop toolbar.

4)      If you desire to add the Background Info (BGI) utility to your Windows 2012 server desktop, then complete the following steps:

Snagit Capture

  • From your network share or software repository containing BGInfo, copy the folder BGInfo to C:\BGInfo.  Edit the BGInfo.bgi config file to customize (if desired) the BGInfo settings.  (this is the latest 64-bit version of BGInfo)

Snagit Capture

  • To automatically refresh BGInfo each time you log on to the server, add a reg key (string value) called BGInfo with value of C:\BGInfo\LaunchBGI.batto:HKLM\Software\Microsoft\Windows\CurrentVersion\Run

Snagit Capture

  • If using mRemote, change the Display Wallpaper setting to Yes under the configuration settings for your server (the default setting is No).  Otherwise the BGInfo screen will not be passed to your display.

5)      To work around the cumbersome process of having to navigate to log-off, shutdown, or reboot commands under the hidden menus, place shortcuts to these operations on the Server 2012 desktop.  To make this process quick and easy, pre-defined shortcuts can be saved on a network share and copied down to each server installation.

 Snagit Capture


  • From the network share, copy the desktop shortcuts to Libraries\Documents\Public Documents on your 2012 server.

Snagit Capture

  • Once copied, open the Desktop_Icons folder, and copy and paste the icons found there to the public desktop (a hidden folder) which can be accessed at C:\Users\public\desktop (manually type this path in Windows Explorer as shown above to get to it).
  • Add or create other shortcuts as desired here so they will show on the public desktop.
  • By placing them on the public desktop, they will be there for all users, and will be preserved even when the server is sysprepped.

Snagit Capture

6)      When finished, your desktop will look similar to the above screen capture:

  • (My) Computer and Control Panel icons added to the desktop
  • Shutdown, Logoff, and Restart icons (which are shortcuts to the shutdown command) added to the desktop.  This is much quicker than having to access these options from the hidden menus on the left or right sides of the desktop, and it skips having to provide a reason for shutting down.
  • Shortcut to launch Disk Manager added to the desktop (add other shortcuts as desired)
  • Shortcuts to I.E., Run, and Command Prompt added to the taskbar
  • Desktop toolbar added to the taskbar
  • Background Info (BGInfo) provides for a blue background with the server name and other essential server specs on the desktop.  This will automatically refresh at each logon due to adding LaunchBGI.bat to Run in the system registry, and it can be refreshed manually at any time by clicking on the LaunchBGI icon on the public desktop.

Sysprep Suggestions

1)      When building a new gold image of a Windows 2012 server, include the above customizations before running Sysprep to allow cloned copies to boot with these modifications in place.  Most of the changes will be preserved in the sysprep image saving configuration time.

2)      Other suggested modifications you may want to consider making to a Windows 2012 image before sysprepping it to use as a gold image it include:

  1. Enable RDP
  2. Install Adobe Reader
  3. Using Roles and Features, install .Net 3.5 (set the path to \sources\sxs when prompted); Failover Clustering, MPIO, and Hyper-V
  4. Disable the firewall
  5. Disable I.E. security
  6. Disable User Account Control security (set to never notify)
  7. Fully patch the server
  8. If a physical server, run the applicable driver and firmware management/update utility to apply the latest drivers and firmware.
  9. Set the time zone to Central
  10. Install JRE (version of your choice, both the 32bit and 64bit versions)
  11. Other apps and features as desired

Update VMware Tools via Windows System Tray

May 31st, 2012

A Windows platform owner may inquire why he or she is unable to update an Out-of-date VMware tools installation using the VMware Tools applet in the system tray.  Clicking on the Update Tools button either produces an error similar to Update Tools failed or nothing at all happens.

Snagit Capture

Although the option to update VMware Tools is generally available via the system tray, the functionality is disabled by default in the VM shell.  The solution to the issue can be found in VMware KB 2007298 Updating VMware Tools fails with the error: Update Tools failed. Edit the virtual machine’s vmx file.

Shut down the virtual machine and add the following line to the virtual machine’s .vmx configuration file via Edit Settings | Options | General | Configuration Parameters: = “FALSE”

Power on the virtual machine.  From this point forward, a VMware Tools update can be successfully performed from within the guest VM.

VMware Workstation & Fusion Christmas In August Sale!

August 2nd, 2011

30% off through August 4th! All boxed and shrink wrapped copies of VMware Workstation (for Windows & Linux) and VMware Fusion (for Mac) must go!  Hurry while supplies last!  Use promo code PREHOLSALE at checkout for your 30% discount.  Mention and it is likely that nothing additional will happen.

8-2-2011 11-12-56 PM