Posts Tagged ‘VMware Tools’

VMware Tools causes virtual machine snapshot with quiesce error

July 30th, 2016

Last week I was made aware of an issue a customer in the field was having with a data protection strategy using array-based snapshots which were in turn leveraging VMware vSphere snapshots with VSS quiesce of Windows VMs. The problem began after installing VMware Tools version 10.0.0 build-3000743 (reported as version 10240 in the vSphere Web Client) which I believe is the version shipped in ESXI 6.0 Update 1b (reported as version 6.0.0, build 3380124 in the vSphere Web Client).

The issue is that creating a VMware virtual machine snapshot with VSS integration fails. The virtual machine disk configuration is simply two .vmdks on a VMFS-5 datastore but I doubt the symptoms are limited only to that configuration.

The failure message shown in the vSphere Web Client is “Cannot quiesce this virtual machine because VMware Tools is not currently available.”  The vmware.log file for the virtual machine also shows the following:

2016-07-29T19:26:47.378Z| vmx| I120: SnapshotVMX_TakeSnapshot start: ‘jgb’, deviceState=0, lazy=0, logging=0, quiesced=1, forceNative=0, tryNative=1, saveAllocMaps=0 cb=1DE2F730, cbData=32603710
2016-07-29T19:26:47.407Z| vmx| I120: DISKLIB-LIB_CREATE : DiskLibCreateCreateParam: vmfsSparse grain size is set to 1 for ‘/vmfs/volumes/51af837d-784bc8bc-0f43-e0db550a0c26/rmvm02/rmvm02-000001.
2016-07-29T19:26:47.408Z| vmx| I120: DISKLIB-LIB_CREATE : DiskLibCreateCreateParam: vmfsSparse grain size is set to 1 for ‘/vmfs/volumes/51af837d-784bc8bc-0f43-e0db550a0c26/rmvm02/rmvm02_1-00000
2016-07-29T19:26:47.408Z| vmx| I120: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
2016-07-29T19:26:56.292Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
2016-07-29T19:27:07.790Z| vcpu-0| I120: Tools: Tools heartbeat timeout.
2016-07-29T19:27:11.294Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
2016-07-29T19:27:17.417Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
2016-07-29T19:27:17.417Z| vmx| I120: Msg_Post: Warning
2016-07-29T19:27:17.417Z| vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
2016-07-29T19:27:17.417Z| vmx| I120: —————————————-
2016-07-29T19:27:17.420Z| vmx| I120: Vigor_MessageRevoke: message ‘msg.snapshot.quiesce.rpc_timeout’ (seq 10949920) is revoked
2016-07-29T19:27:17.420Z| vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
2016-07-29T19:27:17.420Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot ‘jgb’: 0
2016-07-29T19:27:17.420Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (31).
2016-07-29T19:27:17.420Z| vmx| I120: VigorTransport_ServerSendResponse opID=ffd663ae-5b7b-49f5-9f1c-f2135ced62c0-95-ngc-ea-d6-adfa seq=12848: Completed Snapshot request.
2016-07-29T19:27:26.297Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.

After performing some digging, I found VMware had released VMware Tools version 10.0.9 on June 6, 2016. The release notes identify the root cause has been identified and resolved.

Resolved Issues

Attempts to take a quiesced snapshot in a Windows Guest OS fails
Attempts to take a quiesced snapshot after booting a Windows Guest OS fails

After downloading and upgrading VMware Tools version 10.0.9 build-3917699 (reported as version 10249 in the vSphere Web Client), the customer’s problem was resolved. Since the faulty version of VMware Tools was embedded in the customer’s templates used to deploy virtual machines throughout the datacenter, there were a number of VMs needing their VMware Tools upgraded, as well as the templates themselves.

RHEL 7, open-vm-tools, and guest customization

August 9th, 2015

Update 5/26/18: For RHEL 7.2 and newer, be sure to read the 5/26/18 update below as some of the steps below are no longer necessary.

I spent some time this weekend working with vCloud Director 5.5.4 build 2831206 (on vSphere 6) and Red Hat Enterprise Linux vApp/guest customization. I’m not a *nix guru but I’m comfortable enough with legacy versions of RHEL 5 and 6 as I’ve worked with them quite a bit, particularly for vSphere applications and solutions such as vCloud Director to provide just one example. Quite honestly internet research or peer networking provides supplemental knowledge for whatever I can’t figure out. However I hadn’t spent much time with RHEL 7. There are some new twists and this blog post is an attempt to document what I’ve uncovered to answer questions and hopefully save myself some time in the future. If you’re in a hurry, skip to the “Tying It All Together” section at the end.

vSphere Templates and vCloud vApp Templates

When it comes to vSphere templates that I use myself, I’ll bake in commonly utilized software packages, patches, as well as tweaks and best practices. However, when it comes to shared vApp Templates in a vCloud Catalog, I employ more of a purist philosophy to minimize issues or questions raised regarding the DNA of the OS build I’m sharing with the organization which serves as their base starting point for their vApp. Aside from installing VMware Tools, my Windows 2012 R2 vApp is about as vanilla as it gets. The same can be said for my RHEL 5 and RHEL 6 vApps. When I applied that same approach to RHEL 7, that’s where some noticeable changes became apparent.

The RHEL 7 Minimal Install

The mere existence of this blog post stems from here. The default installation of RHEL 7 is a Minimal Install. While it’s not encumbered with extra software that may never be used depending on the server’s role, it’s also missing packages commonly installed in the past. Some of which are core dependencies in a virtualized datacenter. However, not knowing this, I gladly accepted the opportunity of a minimalist installation. And that’s exactly what I got.

VMware Tools

After completing a rather uneventful RHEL 7 installation, typically the first and last order of business is to install VMware Tools. Those who attempt it on RHEL 7 (as well as other newer versions of *nix such as CentOS 7) will be greeted with rather stern wording that VMware Tools should be avoided and rather the OS provided open-vm-tools should be used instead. VMware support of open-vm-tools (2073803) provides background information, detail, and outlines the benefits of open-vm-tools. It’s not that you can’t install VMware Tools on RHEL 7, you can, but VMware is not recommending it at this point. In the previously linked KB article:

VMware recommends using open-vm-tools redistributed by operating system vendors.

VMware fully supports…

VMware aids in the development of…

VMware does not recommend removing open-vm-tools redistributed by operating system vendors.

Those who choose to install VMware Tools anyway on a RHEL 7 Minimal Install will soon discover that they cannot do so without installing some additional support RHEL 7 packages. VMware Tools cannot be installed on RHEL 7 due to missing ifconfig (2075519) explains that net-tools is missing and must be installed as follows (you’ll need a yum repository; the next section covers that):

#sudo yum install net-tools

I’d also argue that you’re going to need to install supporting PERL packages to execute the /usr/bin/ script because it’s also missing in a RHEL 7 Minimal Install. More on that a little later but for now, the other packages that are needed can be installed as follows:

# yum install perl gcc make kernel-headers kernel-devel -y

Creating A Local DVD Repository For YUM (from Red Hat)

Without a Red Hat subscription (I fall into this category), or the networking means to reach your subscription on the internet, you’ll need to rely on your RHEL 7 DVD or .iso to install necessary packages such as net-tools mentioned above. In order to access these packages, you’ll need to mount the DVD and create a local DVD repository.

Mounting the DVD:

mount /dev/cdrom /mnt/

Creating the local DVD repository is slightly more involved but the steps are easy to follow. Create the file /etc/yum.repos.d/dvd.repo. The file should contain the following text:


The local DVD repository is now available and its existence can now be queried (note that it only remains available for as long as the RHEL 7 DVD is mounted):

yum repolist all

An example of installing a yum package is shown above although it does not always require the use of sudo.

Open VM Tools (from Red Hat)

Red Hat Enterprise Linux 7 Guest Operating System Installation Guide documents the process of installing open-vm-tools. Remember that open-vm-tools is distributed by the OS vendor so everything you need from that respect is available from the RHEL 7 DVD and the local DVD repository created above. That said, installing open-vm-tools is straightforward:

# yum install open-vm-tools

Verify open-vm-tools has been installed in the guest:

# yum search open-vm-tools

With open-vm-tools installed, the guest now has the following vSphere feature functionality:

  • Synchronization of the guest OS clock with the virtualization platform
  • Enables the virtual infrastructure to perform graceful power operations (shut down) and file system quiescing of the virtual machine
  • Provides a heartbeat from guest to the virtualization infrastructure to support vSphere High Availability (HA)
  • Publishes information about the guest OS to the virtualization platform, including resource utilization and networking information
  • Provides a secure and authenticated mechanism to perform various operations within the guest OS from the virtualization infrastructure
  • Accepts additional plug-ins that can extend or customize open-vm-tools functionality

Guest customization and the deployPkg Tools Plug-in (from VMware)

Looking at the bulleted list above, a number of features are provided by open-vm-tools. Unfortunately guest customization isn’t one of them (guest customization is typically used in deploying templates in vSphere as well as deploying available vApps from a vCloud Director catalog). At this point if you attempt to clone a RHEL 7 guest with open-vm-tools, you’ll get the exact same VM over and over again with no unique guest customization. The last bullet speaks to a plug-in architecture for which a guest customization plug-in is available from VMware called the deployPkg Tools Plug-in.

Red Hat Enterprise Linux 7 Guest Operating System Installation Guide talks about the plug-in and while it appears to provide the installation instructions, it’s missing a few required steps for installing the VMware Packaging Public Keys so refer to Installing the deployPkg plug-in in a Linux virtual machine (2075048) for the correct process. In this process, yum will be used to install a package available via the internet from VMware instead of from the local DVD repository described previously.

Download the two VMware Packaging Public Keys from VMware at

Copy them to /tmp/ on the RHEL 7 guest

Import each of the two keys (that’s a double dash in front of import):

# rpm –import /tmp/

# rpm –import /tmp/

Create the yum repository by creating a file called /etc/yum.repos.d/vmware-tools.repo containing the following text:

name = VMware Tools
baseurl =
enabled = 1
gpgcheck = 1

Execute the command

sudo yum install open-vm-tools-deploypkg

Followed by

sudo systemctl restart vmtoolsd

At this point, both open-vm-tools from Red Hat as well as open-vm-tools-deploypkg from VMware have been installed and guest customization should work and you’d be done, except…

RHEL 7 Guest Customization Fails Because The Minimal Install Is Missing PERL

Under the RHEL 7 Minimal Install, guest customization still does not produce unique VMs during a cloning process. Taking a look at the clone in /var/log/vmware-imc/toolsDeployPkg.log, I noticed the following:

Launching deployment /usr/bin/perl -I/tmp/.vmware/linux/deploy/scripts /tmp/.vmware/linux/deploy/scripts/ /tmp/.vmware/linux/deploy/cust.cfg.

Command to exec : /usr/bin/perl

Customization command output:

Deploy error: Deployment failed. The forked off process returned error code.

Package deploy failed in DeployPkg_DeployPackageFromFile

The folder /usr/bin/perl/ does not exist.

So then where is PERL? I already know the answer before I’m told.. it doesn’t exist under a RHEL 7 Minimal Install

[root@localhost ~]# whereis perl
perl:[root@localhost ~]#

Install PERL from the RHEL 7 local DVD repository. This installation should be performed on the template or vApp before it’s placed into the catalog so that the resulting guest customization works (obviously it has little effect on a guest customization which has already failed):

# yum install perl gcc make kernel-headers kernel-devel -y

PERL is now installed and can be called upon for guest customization:

[root@localhost ~]# whereis perl
perl: /usr/bin/perl /usr/share/man/man1/perl.1.gz
[root@localhost ~]#

RHEL 7 Guest Agents

The RHEL 7 Minimal Install turned out to be a bit of learning process. A more streamlined approach, if available, would be to utilize the Infrastructure Server base environment during the RHEL 7 installation instead of the Minimal Install. Infrastructure Server is going to automatically include PERL and net-tools. It’s also going to expose the ability to install the Guest Agents Add-On. It’s talked about in full in the Red Hat Enterprise Linux 7 Guest Operating System Installation Guide. Installing the Guest Agents includes open-vm-tools from the RHEL 7 DVD without the extra steps of manually creating the RHEL 7 local DVD repository.

While this is certainly more efficient, the one remaining caveat is that Guest Agents does not include the deployPkg Tools Plug-in from VMware. The plug-in will still need to be manually installed from the VMware repository if customization of the VM or vApp is required. For templates, this is almost always a necessity.

RHEL 7 Networking

One last note is that networking in RHEL 7 has seen some changes. For openers, legacy device names such as eth0, eth1, etc. are replaced by a profile name such as eno16780032 (the corresponding files reflect these name changes in /etc/sysconfig/network-scripts/). Menu driven network configuration (previously accessed from setup) has been replaced by a Network Manager which is accessible via nmtui (Network Manager Text User Interface), nmcli (Network Manager Command Line Interface), or Network Scripts. Also recall from the top of the article that the old standby ifconfig will not be present under a Minimal Install – it requires the net-tools package. Last but not least, detected Ethernet adapters in a Minimal Installation are not automatically enabled for use. Discovered Ethernet devices can be enabled during the initial RHEL 7 setup (I believe it’s under Hostname and Network), or it can be enabled after installation by running nmtui and check the Automatically Connect box for the appropriate Edit a connection menu. A detected Ethernet adapter listing can be obtained at any time with nmcli d.

Although this article was specific RHEL 7, open-vm-tools is available with the following operating systems as documented by VMware support of open-vm-tools (2073803)

  • Fedora 19 and later releases
  • Debian 7.x and later releases
  • openSUSE 11.x and later releases
  • Recent Ubuntu releases (12.04 LTS, 13.10 and later)
  • Red Hat Enterprise Linux 7.0 and later releases
  • CentOS 7 and later releases
  • Oracle Linux 7 and later releases
  • SUSE Linux Enterprise 12 and later releases

RHEL 7 Templates – Tying It All Together

In the end, there are a few different Base Environment types available with varying steps for building a RHEL 7 image which supports guest customization in vSphere or vCloud Director.

Minimal Install (default)

  1. Choose Minimal Install Base Environment
  2. Enable Ethernet card to automatically connect (at install or later using nmtui)
  3. Add RHEL 7 local DVD repository
  4. Install net-tools
  5. Install PERL
  6. Install open-vm-tools
  7. Add yum repository for VMware (for RHEL 7.2 and newer, be sure to read the 5/26/18 update below as this step is no longer necessary)
  8. Install the deployPkg Tools Plug-in (for RHEL 7.2 and newer, be sure to read the 5/26/18 update below as this step is no longer necessary)

Infrastructure Server

  1. Choose Infrastructure Server Base Environment and Guest Agents Add-On (open-vm-tools will automatically be installed)(for RHEL 8.0 and newer, open-vm-tools is automatically installed with the default Base Environment of Server with GUI)
  2. Enable Ethernet card to automatically connect (at install or later using nmtui)
  3. Add yum repository for VMware (for RHEL 7.2 and newer, be sure to read the 5/26/18 update below as this step is no longer necessary)
  4. Install the deployPkg Tools Plug-in (for RHEL 7.2 and newer, be sure to read the 5/26/18 update below as this step is no longer necessary)

Clearly the Minimal Install route has more steps while the Infrastructure Server route has less steps and is quicker. Regardless of Base Environment type, VMware does not recommend the installation of VMware Tools.

I’ve linked several resources throughout this article. Just about all of the information was available, it was merely a matter of finding and reading the relevant documentation which isn’t always in one place. The only dots I had to connect on my own which I didn’t see mentioned anywhere was the lack of PERL for the deployPkg Tools Plug-in from VMware as well as for the installation of VMware Tools on RHEL 7 which isn’t recommended by VMware.

Update 8/22/15: vCloud Director guest customization is also problematic with CentOS 7 but with one additional hang up. I’ve found several references on the internet with the workaround including one I’ll link here from my good friend Bob Plankers/etc/redhat-release must read Red Hat Enterprise Linux Server release 7.0 (Maipo)

Update 9/11/15: Brian Graf authored a nice piece yesterday titled Open-VM-Tools (OVT): The Future of VMware Tools for Linux which anyone who wound up here should find interesting.

Update 5/26/18: As noted by daVikes in the comment section below (thank you daVikes!), relevant updates have been made as of RHEL 7.2 which streamlines templates and guest customization even further. In short, deployPkg is included with open-vm-tools. Whether you choose to start with a minimal installation and install open-vm-tools afterwards, or you choose the Infrastructure Server and install Guest Agents (this installs open-vm-tools), deployPkg will be included with open-vm-tools. Do note that if choosing the Infrastructure Server method, downloading the two VMware Packaging Public Keys and creating the /etc/yum.repos.d/vmware-tools.repo yum repository is no longer necessary. I still make it a habit to follow the steps for Creating A Local DVD Repository For YUM.

Update 6/28/19: RHEL 8.0: open-vm-tools is automatically installed with the default Base Environment of Server with GUI

vMA 5.1 Patch 1 Released

April 5th, 2013

Expendable news item here only worthy of a Friday post.  For those who may have missed it, VMware has released an update to the vSphere Management Assistant (vMA) 5.1 appliance formally referred to as Patch 1.  This release is documented in VMware KB 2044135 and the updated appliance bits can be downloaded here.  Log in, choose the VMware vSphere link, then the Drivers & Tools tab.

Patch 1 bundles with it the following enhancements:

  • The base operating system is updated to SUSE Linux Enterprise Server 11 SP2 (12-Jan-2013).
  • JRE is updated to JRE 1.6.0_41, which includes several critical fixes.
  • VMware Tools is updated to 8.3.17 (build 870839).
  • A resxtop connection failure issue has been fixed.
    In vMA 5.1, resxtop SSL verification checks has been enabled. This might cause resxtop to fail when connecting to hosts and displays an exception message similar the following:
    HTTPS_CA_FILE or HTTPS_CA_DIR not set.
    This issue is fixed through this patch.

Update VMware Tools via Windows System Tray

May 31st, 2012

A Windows platform owner may inquire why he or she is unable to update an Out-of-date VMware tools installation using the VMware Tools applet in the system tray.  Clicking on the Update Tools button either produces an error similar to Update Tools failed or nothing at all happens.

Snagit Capture

Although the option to update VMware Tools is generally available via the system tray, the functionality is disabled by default in the VM shell.  The solution to the issue can be found in VMware KB 2007298 Updating VMware Tools fails with the error: Update Tools failed. Edit the virtual machine’s vmx file.

Shut down the virtual machine and add the following line to the virtual machine’s .vmx configuration file via Edit Settings | Options | General | Configuration Parameters: = “FALSE”

Power on the virtual machine.  From this point forward, a VMware Tools update can be successfully performed from within the guest VM.

VMware Tools install – A general system error occurred: Internal error

June 16th, 2010

When you invoke the VMware Tools installation via the vSphere Client, you may encounter the error “A general system error occurred: Internal error“.

6-16-2010 9-45-42 AM

One thing to check is that the VM shell has the correct operating system selected for the guest operating system type.  For example, a setting of “Other (32-bit)” will cause the error since VMware cannot determine the correct version of the tools to install in the guest operating system because the flavor of guest operating system is unknown (ie. Windows or Linux).

6-16-2010 10-45-49 AM

Other causes for this error can be found at VMware KB Article 1004718:

The virtual machine has CD-ROM configured.
The windows.iso is present under the /vmimages/tools-iso/ folder.
The virtual machine is powered on.
The correct guest operating system selected. For example, if the guest operating system is Windows 200, ensure you have chosen Windows 2000 and not Other.

Windows 2008 R2 and Windows 7 on vSphere

March 28th, 2010

If you run Windows Server 2008 R2 or Windows 7 as a guest VM on vSphere, you may be aware that it was advised in VMware KB Article 1011709 that the SVGA driver should not be installed during VMware Tools installation.  If I recall correctly, this was due to a stability issue which was seen in specific, but not all, scenarios:

If you plan to use Windows 7 or Windows 2008 R2 as a guest operating system on ESX 4.0, do not use the SVGA drivers included with VMware Tools. Use the standard SVGA driver instead.

Since the SVGA driver is installed by default in a typical installation, it was necessary to perform a custom installation (or scripted perhaps) to exclude the SVGA driver for these guest OS types.  Alternatively, perform a typical VMware Tools installation and remove the SVGA driver from the Device Manager afterwards.  What you ended up with, of course, is a VM using the Microsoft Windows supplied SVGA driver and not the VMware Tools version shown in the first screenshot.  The Microsoft Windows supplied SVGA driver worked and provided stability as well, however one side effect was that mouse movement via VMware Remote Console felt a bit sluggish.

Beginning with ESX(i) 4.0 Update 1 (released 11/19/09), VMware changed the behavior and revised the above KB article in February, letting us know that they now package a new version of the SVGA driver in VMware Tools in which the bits are populated during a typical installation but not actually enabled:

The most effective solution is to update to ESX 4.0 Update 1, which provides a new WDDM driver that is installed with VMware Tools and is fully supported. After VMware Tools upgrade you can find it in C:\Program Files\Common Files\VMware\Drivers\wddm_video.

After a typical VMware Tools installation, you’ll still see a standard SVGA driver installed.  Following the KB article, head to Windows Device Manager and update the driver to the bits located in C:\Program Files\Common Files\VMware\Drivers\wddm_video:


The result is the new wddm driver, which ships with the newer version of VMware Tools, is installed: 

After a reboot, the crisp and precise mouse movement I’ve become accustomed to over the years with VMware has returned.  The bummer here is that while the appropriate VMware SVGA drivers get installed in previous versions of Windows guest operating systems, Windows Server 2008 R2 and Windows 7 require manual installation steps, much like VMware Tools installation on Linux guest VMs.  Add to this the fact that the automated installation/upgrade of VMware Tools via VMware Update Manager (VUM) does not enable the wddm driver.  In short, getting the appropriate wddm driver installed for many VMs will require manual intervention or scripting.  One thing you can do is to get the wddm driver installed in your Windows Server 2008 R2 and Windows 7 VM templates.  This will ensure VMs deployed from the templates have the wddm driver installed and enabled.

The wddm driver install method from VMware is helpful for the short term, however, it’s not the scalable and robust long term solution.  We need an automated solution from VMware to get the wddm driver installed.  It needs to be integrated with VUM.  I’m interested in finding out what happens with the next VMware Tools upgrade – will the wddm driver persist, or will the VMware Tools upgrade replace the wddm version with the standard version?  Stay tuned.

Update 11/6/10:  While working in the lab tonight, I noticed that with vSphere 4.1, the correct wddm video driver is installed as part of a standard VMware Tools installation on Windows 7 Ultimate x64 – no need to manually replace the Microsoft video driver with VMware’s wddm version as this is done automatically now.

Update 12/10/10: As a follow up to these tests, I wanted to see what happens when the wddm driver is installed under ESX(i) 4.0 Update 1 and its corresponding VMware Tools, and then the VM is moved to an ESX(i) 4.1 cluster and the VMware Tools are upgraded.  Does the wddm driver remain in tact, or will the 4.1 tools upgrade somehow change the driver?  During this test, I opted to use Windows 7 Ultimate 32-bit as the guest VM guinea pig.  A few discoveries were made, one of which was a surprise:

1.  Performing a standard installation of VMware Tools from ESXi 4.0 Update 1 on Windows 7 32-bit will automatically install the wddm driver, version as shown below.  No manual steps were required to install this driver forcing a 2nd reboot.  I wasn’t counting on this.  I expected the Standard VGA Graphics Adapter driver to be installed as seen previously.  This is good.

SnagIt Capture

After moving the VM to a 4.1 cluster and performing the VMware Tools upgrade, the wddm driver was left in tact, however, its version was upgraded to  This is also good in that the tools ugprade doesn’t negatively impact the desired results of leveraging the wddm driver for best graphics performance.

SnagIt Capture

More conclusive testing should be done with Windows 7 and Windows Server 2008 R2 64-bit to see if the results are the same.  I’ll save this for a future lab maybe.

VMware ESX Guest OS I/O Timeout Settings (for NetApp Storage Systems)

October 29th, 2009

You may already be aware that installing VMware Tools in a Windows VM configures a registry value which controls the I/O timeout for all Windows disk in the event of a short storage outage. This is to help the guest operating system survive high latency or temporary outage conditions such as SAN path failover or maybe a network failure in Ethernet based storage.  VMware Tools changes the Windows default value of 10 seconds for non-cluster nodes, 20 seconds for cluster nodes, to 60 seconds (or x03c hex).

Did you know that disk I/O timeout is a configurable parameter in other guest operating systems as well? And why not, it makes sense that we would want every guest OS to be able to outlast a storage deficiency.

NetApp offers a document titled VMware ESX Guest OS I/O Timeout Settings for NetApp Storage Systems. It’s published as kb41511 and you’ll need a free NetApp NOW account to access the document. This white paper serves a few useful purposes:

  • Defines recommended disk I/O timeout settings for various guest operating systems on NetApp storage systems
  • Defines benchmark disk I/O timeout settings for various guest operating systems which could be used on any storage system, including local SCSI
  • In some cases provides scripts to make the necessary changes
  • Explains the methods to make the disk I/O timeout changes on the following guest operating systems:
    • RHEL4
    • RHEL5
    • SLES9
    • SLES10
    • Solaris 10
    • Windows

Now on the subject disk I/O timeouts, understand the above is to be used as chance for extending the uptime of a VM during adverse storage conditions. As in life, there are no guarantees. A guest OS with high disk I/O activity may not be able to tolerate sustained read and/or write requests for the duration of the timeout value. Windows guests may freeze or BSOD. Linux guests may go read-only on their root volumes which requires a reboot. Which brings me to the next point…

A larger timeout value isn’t necessarily better. In extending disk I/O timeout values, we’re applying virtual duct tape to an underlying storage issue which needs further looking into. Given the complex and wide variety of shared storage systems available to the datacenter today, storage issues can be caused by many variables including but not limited to disks (spindles), target controllers, fabric components such as fibre cables, SFP/GBICs, HBAs, fabric switches, zoning, network components such as copper cabling, NICs, network switches, routers, and firewalls. Also keep in mind that while the OS may survive the disk I/O interruption, application(s) running on the OS platform may not.  Applications themselves implement response timeout values which are likely going to be hard coded and non-configurable by a platform or virtualization administrator in the application itself.

Lastly, try to remember that if you go through the effort of increasing your disk I/O timeout values on Windows guests beyond 60 seconds, future installation of VMware Tools or other applications/updates may reset the disk I/O timeout back to 60 seconds.  What this means is that in medium to large environments, you’re going to need an automated method to deploy custom disk I/O timeout values at least for Windows guests.  For those with NetApp storage, NetApp pushes these standards firmly, along with other VMware best practices which I’ll save for a future blog article.

Update 4/28/10:  VMware Tools for vSphere installation doesn’t change the disk timeout setting if a custom value was previously set (ie. 190 seconds)

Update 9/12/11:  See also VMware KB article 1009465 Increasing the disk timeout values for a Linux 2.6 virtual machine