I ran into an disturbing problem this evening in the lab. While in the Virtual Infrastructure Client (VIC), I attempted to perform a graceful shut down on a VM by right clicking on it and choosing Shut Down Guest. Unfortunately the graceful shutdown and restart options were grayed out which is a good indicator that the VMware Tools are not installed or not running. I logged into the VM and strangely enough, the VMware Tools were installed and the VMware Tools service was running. Even stranger, when I went back to the VIC, the VMware Tools status now showed “Tools OK”.
It was then that I noticed VMware Tools status was showing “Not Running” for a whole slew of other VMs which I knew had tools installed.
A quick search uncovered a recently updated VMware KB article 1008709 “VMware Tools status shows as not running after running VMware Consolidated Backup“. Mind you, I’m not running VCB in the lab (thank God and Veeam), however, the description in the KB article mostly matched my situation.
During the normal VMware Consolidated Backup (VCB) operation, the VMware Tools status changes from OK to Not Running for some time during the initial snapshot operation, but it returns to OK after the VCB operation completes.
However, on hosts installed with the patch bundle ESX350-200901401-SG, the VMware Tools status on the virtual machines may stay as Not Running even after the VCB operation completes.
Although the KB article specifically ties the problem to VCB, the problem is not limited to VCB in my experience. Other applications that perform snapshots can cause the behavior, such as the product I’m using: Veeam Backup 3.0. The root cause stems from a January 2009 VMware patch: ESX350-200901401-SG.
There are a few work arounds, the second of which I discovered on my own:
- Restart the mgmt-vmware service immediately after the backup job is done. This changes the Tools status to OK. You can write a cron job to do it periodically.OR
- Log in and log out, or log out if you are already logged in, from the virtual machine. This changes the Tools status to OK if it was showing as Not running.OR
- Use VCBMounter to look for virtual machine name or UUID rather than virtual machine IP. Virtual machine IP only works when the status of tools is OK, but virtual machine name and UUID works even if the Tools status shows as Not running.
After reading the KB article, I ran a service mgmt-vmware restart and after about a minute, the VMware Tools status for all my VMs changed in status from “Not Running” to “Tools OK”. The host and all of its VMs briefly disconnected as well but don’t worry, they’ll come back on their own shortly.
Until VMware releases a permanent fix, it sounds like I can expect this behavior daily after each Veeam backup completes.
By the way, if you’re running VCB, this condition will cause future VCB backups to fail if the VCBMounter is set to look for the virtual machine IP rather than virtual machine name or UUID. Nobody likes failed backups so please make sure you get this sorted out in your environment if the problem exists.