boche.net – VMware vEvangelist

Archive for March, 2009

VMware Tools “Not Running”

March 31st, 2009

I ran into an disturbing problem this evening in the lab. While in the Virtual Infrastructure Client (VIC), I attempted to perform a graceful shut down on a VM by right clicking on it and choosing Shut Down Guest. Unfortunately the graceful shutdown and restart options were grayed out which is a good indicator that the VMware Tools are not installed or not running. I logged into the VM and strangely enough, the VMware Tools were installed and the VMware Tools service was running. Even stranger, when I went back to the VIC, the VMware Tools status now showed “Tools OK”.

It was then that I noticed VMware Tools status was showing “Not Running” for a whole slew of other VMs which I knew had tools installed.

A quick search uncovered a recently updated VMware KB article 1008709 “VMware Tools status shows as not running after running VMware Consolidated Backup“. Mind you, I’m not running VCB in the lab (thank God and Veeam), however, the description in the KB article mostly matched my situation.

During the normal VMware Consolidated Backup (VCB) operation, the VMware Tools status changes from OK to Not Running for some time during the initial snapshot operation, but it returns to OK after the VCB operation completes.

However, on hosts installed with the patch bundle ESX350-200901401-SG, the VMware Tools status on the virtual machines may stay as Not Running even after the VCB operation completes.

Although the KB article specifically ties the problem to VCB, the problem is not limited to VCB in my experience. Other applications that perform snapshots can cause the behavior, such as the product I’m using: Veeam Backup 3.0. The root cause stems from a January 2009 VMware patch: ESX350-200901401-SG.

There are a few work arounds, the second of which I discovered on my own:

Restart the mgmt-vmware service immediately after the backup job is done. This changes the Tools status to OK. You can write a cron job to do it periodically.OR
Log in and log out, or log out if you are already logged in, from the virtual machine. This changes the Tools status to OK if it was showing as Not running.OR
Use VCBMounter to look for virtual machine name or UUID rather than virtual machine IP. Virtual machine IP only works when the status of tools is OK, but virtual machine name and UUID works even if the Tools status shows as Not running.

After reading the KB article, I ran a service mgmt-vmware restart and after about a minute, the VMware Tools status for all my VMs changed in status from “Not Running” to “Tools OK”. The host and all of its VMs briefly disconnected as well but don’t worry, they’ll come back on their own shortly.

Until VMware releases a permanent fix, it sounds like I can expect this behavior daily after each Veeam backup completes.

By the way, if you’re running VCB, this condition will cause future VCB backups to fail if the VCBMounter is set to look for the virtual machine IP rather than virtual machine name or UUID. Nobody likes failed backups so please make sure you get this sorted out in your environment if the problem exists.

9 comments »

Posted in Virtualization

Tags: 3rd Party Apps ESX Virtual Infrastructure Client VMware

VMware ESX/ESXi 3.5 Update 4 released

March 30th, 2009

Today VMware released Update 4 for it’s flagship bare metal ESX and free ESXi products. The build number has been incremented to build 153875.

Release notes include:

What’s New

Notes:

Not all combinations of VirtualCenter and ESX Server versions are supported and not all of these highlighted features are available unless you are using VirtualCenter 2.5 Update 4 with ESX Server 3.5 Update 4. See the ESX Server, VirtualCenter, and VMware Infrastructure Client Compatibility Matrixes for more information on compatibility. (ESX 3.5u4 is not compatible with versions of VirtualCenter prior to version 2.5u2)
This version of ESX Server requires a VMware Tools upgrade.

The following information provides highlights of some of the enhancements available in this release of VMware ESX Server:

Expanded Support for Enhanced vmxnet Adapter - This version of ESX Server includes an updated version of the VMXNET driver (VMXNET enhanced) for the following guest operating systems:

Microsoft Windows Server 2003, Standard Edition (32-bit)
Microsoft Windows Server 2003, Standard Edition (64-bit)
Microsoft Windows Server 2003, Web Edition
Microsoft Windows Small Business Server 2003
Microsoft Windows XP Professional (32-bit)

The new VMXNET version improves virtual machine networking performance and requires VMware tools upgrade.

Enablement of Intel Xeon Processor 5500 Series - Support for the Xeon processor 5500 series has been added. Support includes Enhanced VMotion capabilities. For additional information on previous processor families supported by Enhanced VMotion, see Enhanced VMotion Compatibility (EVC) processor support (KB 1003212).

QLogic Fibre Channel Adapter Driver Update – The driver and firmware for the QLogic fibre channel adapters have been updated to version 7.08-vm66 and 4.04.06 respectively. This release provides interoperability fixes for QLogic Management Tools for FC Adapters and enhanced NPIV support.

Emulex Fibre Channel Adapter Driver Update - The driver for Emulex Fibre Channel Adapters has been upgraded to version 7.4.0.40. This release provides support for the HBAnyware 4.0 Emulex management suite.

LSI megaraid_sas and mptscsi Storage Controller Driver Update - The drivers for LSI megaraid_sas and mptscsi storage controllers have been updated to version 3.19vmw and 2.6.48.18 vmw respectively. The upgrade improves performance and enhance event handling capabilities for these two drivers.

Newly Supported Guest Operating Systems – Support for the following guest operating systems has been added specifically for this release:

For more complete information about supported guests included in this release, see the Guest Operating System Installation Guide: http://www.vmware.com/pdf/GuestOS_guide.pdf.

SUSE Linux Enterprise Server 11 (32-bit and 64-bit).
SUSE Linux Enterprise Desktop 11 (32-bit and 64-bit).
Ubuntu 8.10 Desktop Edition and Server Edition (32-bit and 64-bit).
Windows Preinstallation Environment 2.0 (32-bit and 64-bit).

Furthermore, pre-built kernel modules (PBMs) were added in this release for the following guests:

Ubuntu 8.10
Ubuntu 8.04.2

Newly Supported Management Agents - Refer to VMware ESX Server Supported Hardware Lifecycle Management Agents for the most up-to-date information on supported management agents.

Newly Supported I/O Devices - in-box support for the following on-board processors, IO devices, and storage subsystems:

SAS Controllers and SATA Controllers:

The following are newly supported SATA Controllers.

PMC 8011 (for SAS and SATA drives)

Intel ICH9

Intel ICH10

CERC 6/I SATA/SAS Integrated RAID Controller (for SAS and SATA drivers)

HP Smart Array P700m ControllerNotes:

Some limitations apply in terms of support for SATA controllers. For more information, see SATA Controller Support in ESX 3.5 (KB 1008673).

Storing VMFS datastores on native SATA drives is not supported.

Network Cards: The following are newly supported network interface cards:

HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter

HP NC362i Integrated Dual port Gigabit Server Adapter

Intel 82598EB 10 Gigabit AT Network Connection

HP NC360m Dual 1 Gigabit/NC364m Quad 1 Gigabit

Intel Gigabit CT Desktop Adapter

Intel 82574L Gigabit Network Connection

Intel 10 Gigabit XF SR Dual Port Server Adapter

Intel 10 Gigabit XF SR Server Adapter

Intel 10 Gigabit XF LR Server Adapter

Intel 10 Gigabit CX4 Dual Port Server Adapter

Intel 10 Gigabit AF DA Dual Port Server Adapter

Intel 10 Gigabit AT Server Adapter

Intel 82598EB 10 Gigabit AT CX4 Network Connection

NetXtreme BCM5722 Gigabit Ethernet

NetXtreme BCM5755 Gigabit Ethernet

NetXtreme BCM5755M Gigabit Ethernet

NetXtreme BCM5756 Gigabit Ethernet

Expanded Support: The E1000 Intel network interface card (NIC) is now available for NetWare 5 and NetWare 6 guest operating systems.

Onboard Management Processors:

IBM system management processor (iBMC)

Storage Arrays:

SUN StorageTek 2530 SAS Array

Sun Storage 6580 Array

Sun Storage 6780 Array

No comments »

Posted in Virtualization

Tags: ESX ESXi VMware

Twitter explained in 267 seconds

March 29th, 2009

I was the guy on the left until last fall when John Troyer showed me how useful and powerful this tool can be during the VMworld 2008 virtualization conference. Properly used, it’s a real time professional networking and knowledge sharing tool, commonly called a microblog.

Thanks to the internet, the delivery of information to the masses can be ranked as follows in order of most timely to least timely:

Twitter
RSS feeds via blog posts and news articles
Email
Traditional mail

Notice I left out instant messaging (IM). IM from a technology perspective is as timely as Twitter except it differs significantly in one facet:

Tweets (messages in Twitter) are multicasted to hundreds, thousands, or millions of people instantly.
Instant Messages are spoken in one on one conversations. It could take days or weeks for spoken information to travel to the volumes of people that Twitter has the ability to reach instantly. Not only that, but think about how broken the message will become after it is repeated by dozens or hundreds of people. Like that old childhood game “Telephone to Norway”. An IM that originally started with “The sky is blue” may eventually end up as “Jesus had a 24 inch LCD”.

Although Twitter can be used with a web browser, getting the most out of it involves a combination of things like following the right people, using 3rd party Twitter clients like TweetDeck, setting up searches to refine incoming tweets only to what you want to see, etc. These are the things that will really help narrow the scope and define its intended use through customization.

But if you use Twitter merely for being a social butterfly, then yeah, it’s pretty much like how the guy on the left describes it. Not that there’s anything wrong with that…

Each person makes Twitter what they want it to be. With that in mind, it’s not so easy to stereotype its use.

Thanks for the link to the video William Lam.

1 comment »

Posted in General

Tags: Blog Social

GuessMyOS plugin released

March 29th, 2009

Andrew the magnificent (vExpert Andrew Kutz of Hyper9) has unleashed a new plugin for the VMware Virtual Infrastructure Client called “GuessMyOS“.

System Requirements:

Microsoft Windows Installer 3.1
Microsoft .NET 3.5 (might as well install the SP1 version while you’re at it)
VMware Virtual Infrastructure Client

Andrew is the plugin Master. Now that he is officially and fully commissioned by Hyper9 to crank out cool stuff (instead of coding on his spare time), expect neato tools at a more consistent pace. I highly advise following his H9Labs RSS feed to stay up to date with his latest works:

http://community.hyper9.com/blogs/h9labs/rss.aspx

Oh. What does it do? Remember VMware GSX Server and the web MUI where VMs were graphically represented by the guest OS thumbnail? That’s what it does, but now for ESX and ESXi. One thing you’ll notice is that in the Hosts and Clusters view, it displays the thumbnail in the left column, but not the main window pane on the right side of the screen. Same behavior in the Virtual Machines and Templates view. Maybe in the next version. Thanks a lot Andrew and keep up the great work! I can absolutely say that we live in a better VMware world with you in it.

No comments »

Posted in Virtualization

Tags: 3rd Party Apps vCenter Server Virtual Infrastructure Client VirtualCenter VMware

Cloud butt kicking

March 29th, 2009

Mike DiPetrillo wrote a nice piece yesterday entitled The Cloud is Kicking My Butt. This has helped put my mind at ease. Those whom I talk to might recall that I’ve been somewhat behind the learning curve on “the cloud”. To say my mind was blown at VMworld 2008 would have been an understatement with all of the cloud buzz and announcements.

It’s comforting to know that I’m in good company. In Mike’s article, he describes his new endeavor in 2009 as VMware’s Global Cloud Architect. Mike has talked to literally hundreds of people with mixed backgrounds and agendas about the cloud and he has come up with a few findings thus far:

Cloud means 1,000 different things to 1,000 different people
Everyone wants cloud today
No one trusts external clouds and yet everyone wants to use them
Absolutely everyone is ignorant on cloud
There are only about 20 people in the world right now that truly “get it”

I can relate to bullets 1 and 4 above. And of the 20 people in the world who are cloud gurus… I’m not one of ‘em. I’ve made some progress since fall of last year though. Simplified, I understand cloud is really no more than a collection of service offerings with special characteristics like portability, scalability, flexibility, high availability, compatibility, security, etc. It’s this very list of what the cloud is that people like me struggle with – because not everyone is going by the same list. The cloud means different things to different people. It’s going to be interesting to watch the when, where, what, and hows of inter cloud compatibility unfold. Without established standards I doubt we’d ever get there. We’re going to need a lot of cooperation between all vendors also, not just the key ones.

I visited with a VMware SE last week and heard one of the best, yet simple definitions of cloud yet: A Cluster Boundary. If you think about that from a VMware infrastructure perspective, it’s easy. VMs are servers, desktops, and applications that move about in the cluster. They are comprised of shared storage, networks, memory, and cpu. They are going to have isolation in most cases where needed and will be secure. Hardware independence makes them compatible with other hosts in the infrastructure that may not be the same exact make and model. It also affords them the ability to float to these other hosts in the cluster using technologies like VMotion, DRS, sVMotion, HA, and FT. But the cluster defines the boundaries of their mobility and thus it defines “a cloud”. The technologies and collaborative initiatives of tomorrow will be what make these clouds compatible and extensible so that are boundaries of the cloud are much much larger. In fact, in a perfect cloud world where all cloud components are compatible and seamless, there are no boundaries. It will be like space, where you have the ability to float wherever you’d like. For a fee or subscription of course.

The cloud is a magnificent undertaking. There are efforts involved which I don’t think we’ve even come close to seeing since the ratification of the TCP/IP protocol. I applaud people like Mike that have the stamina and drive to tackle this great initiative.

Mike concludes with:

Cloud is in flux. Standards don’t really exist. Everyone is labeling everything from toaster ovens to BMWs as “cloud enabled”. The information I share can and will change. Some of it will become obsolete. Some of it may seem very scary like no one has their act together. That’s just the nature of cloud at the moment. Everyone in the industry is stumbling through this together. In the end it’s all going to be worth it. For now though sit back, keep reading the posts, and prepare to get your butt kicked by the cloud.

I don’t think he could be more right if he tried. BTW, congrats on your new gig Mike! Go get ‘em!

2 comments »

Posted in Virtualization

Tags: Cloud VMware

Anti-affinity rules are not honored in cluster with more than 2 virtual machines

March 27th, 2009

We can put a man on the moon and we can hot migrate virtual machines with SMP and gigs of RAM, but we can’t create anti-affinity rules with three or more VMs. This has been a thorn in my side since 2006, long before I requested it fixed in February 2007 on the VMTN Product and Feature Suggestions forum.

VMware updated KB article 1006473 on 3/26 outlining anti-affinity rule behavior when using three or more VMs:

“This is expected behavior, as anti affinity rules can be set only for 2 virtual machines.

When a third virtual machine is added any rule becomes disabled (with 2.0.2 or earlier).

There has been a slight change in behavior with VirtualCenter 2.5, wherein input validation occurs, where a third virtual machine added produces a warning message indicating a maximum of two virtual machines only can be added to this rule.

To workaround this, create more rules to cover all of the combinations of virtual machines.

For example, create rules for (VM1 & VM2), then (VM2 & VM3), and (VM1 & VM3).”

That last sentence is what has been burning my cookies for the longest time. In my last environment, I had several NLB VMs which could not be on the same host for load balancing and redundancy purposes. Rather than create a minimum amount of rules to intelligently handle all of the VMs, I was left with no choice but to create several rules for each potentially deadly combination.

Work harder, not smarter. Come on VMware.

3 comments »

Posted in Virtualization

Tags: DRS Rant vCenter Server VirtualCenter VMware

VLC media player – free

March 25th, 2009

I came across some video training to preview for which I did not have an application that will play them back. Stu over at Vinternals suggested VLC media player which is a free piece of software that plays most audio and video formats.

Stu was right as rain. I did not recognize the file extension of the video files, and the file extension was not a registered type that VLC would natively support, however, VLC had no problem detecting the video format and then opening and playing back the file.

VLC provides the following charts showing their support for the various formats and platforms: