Archive for January, 2009

Uptime lost during VMotion

January 11th, 2009

That’s right. We lose uptime during every VMotion. Relax just a bit – I’m not talking about actual uptime/downtime availability of the guest VM in the datacenter. I’m speaking to the uptime performance metric tracked in the VirtualCenter database. It’s a bug that was introduced in VirtualCenter 2.0 and has remained in the code to this day in VirtualCenter 2.5.0 Update 3. Here’s how it works:

We’ve got a VM named Exchange1. VirtualCenter displays its uptime as 28 days is indicated by the screenshots below:

1-11-2009 12-04-06 AM

1-11-2009 12-10-14 AM

Now the VM has been VMotioned from host SOLO to host LANDO. Notice what has happened to uptime. It has disappeared from the console:

1-11-2009 12-07-37 AM

The real proof in what has ultimately happened is we see from the performance chart the latest uptime metric has been reset from 27.99 days as shown above to 0.0026620 days:

1-11-2009 12-10-54 AM

Sometimes the VIC console will show the uptime counter start over at 0 days, then on to 1 days, etc. Other times the uptime counter will remain blank for days or weeks as you can see from my three other VMs in the first screenshot which show no uptime.

This brings us to an interesting discussion. What would you like uptime in the VIC to mean exactly? Following are my observations and thoughts on VMware’s implementation of the uptime metric in VirtualCenter.

In previous versions of VirtualCenter, a soft reboot of the VM inside of the OS would reset the uptime statistic in VirtualCenter. I believe this was a function of VMware Tools that triggered this.

Today in VirtualCenter 2.5.0 Update 3, a soft reboot inside the guest VM does not reset the uptime statistic back to zero.

A VM which has no VMware Tools installed that is soft rebooted inside of the OS (ie. we’re not talking about any VMware console power operation here) does not reset the uptime statistic.

I could see the community take a few different sides on this as there are two variations of the definition of uptime we’re dealing with here. Uptime of the guest VM OS and uptime of the VM’s virtual hardware.

  1. Should uptime translate into how long the VM virtual hardware has been powered on from a virtual infrastructure standpoint?
  2. Or should uptime translate into how long the OS inside the VM has been up, tracked by VMware Tools?

The VMware administrator cares about the length of time a VM has been powered on. It is the powered on VM that consumes resources from the four resource food groups and impacts capacity.

The guest VM OS administrator, whether it be Windows or Linux, cares about uptime of the guest OS. The owner of the OS is held to SLAs by the business lines.

My personal opinion is that the intended use of the Virtual Infrastructure Client is for the VMware administrator and thus should reflect virtual infrastructure information. My preference is that the uptime statistic in VirtualCenter tracks power operations of the VM irregardless of any reboot sequences of the OS inside the VM. In other words, uptime is not impacted by VMware Tools heartbeats or reboots inside the guest VM. The uptime statistic should only be reset when the VM is powered off or power reset (including instances where HA has recovered a VM).

At any rate, due to the bug that uptime has in VirtualCenter 2.0 and above, it’s a fairly unreliable performance metric for any virtual infrastructure using VMotion and DRS. Furthermore, the term itself can be misleading depending on the administrators interpretation of uptime versus what’s written in the VirtualCenter code.

I submitted a post in VMware’s Product and Feature Suggestions forum in January of 2007 recording the uptime reset on VMotion issue. As this problem periodically bugs me, I followed up a few times. Once in a follow up post in the thread above, and at least one time externally requesting someone from VMware take a look at it. Admittedly I do not have an SR open.

VMware, can we get this bug fixed? After all, if the hypervisor has become an every day commodity item leaving the management tools as the real meat and potatoes, you should make sure our management tools work properly.

Thank you,


How VMware virtualized Exchange 2007

January 8th, 2009

I often hear questions or concerns about virtualizing Exchange.  E-Oasis found a new VMware white paper and provides a nice lead in explaining how VMware corporate took their physical servers and migrated to virtual, reducing aggregrate hardware usage.

One might ask why VMware’s Exchange servers were not virtualized before this, particularly when VMware was a smaller company with less mailboxes?  Perhaps they decided earlier versions of Exchange were not virtualization candidates?  Maybe limitations in earlier version of ESX made it less than attractive?  I don’t know why but it would have been cooler to see VMware put their money where their mouth is earlier on.  Perhaps someone from VMware can chime in on a comment here.

At any rate, it’s an absolutely beautiful white paper and I’m actually surprised at the level of detail some of the diagrams get into providing network host names and IP addresses for the infrastructure.  I suppose they could be ficticious, but the names look rather authentic and not made up to me.  Kudos.

Take a look at VMware’s whitepaper here.

Cloud computing explanation that anyone can understand

January 7th, 2009

2009’s datacenter trend (which really impacts virtualization and beyond) is cloud computing. Right now there are approximately 1,001 interpretations and explanations of what people think cloud computing is, which for me has made things excessively confusing. It’s still a lot of fluff until I see the SKUs and installable components, then I’ll be able to see where the rubber really meets the road.  For companies like Amazon, the cloud is already reality and it’s catching on quickly with other big names like VMware, Google and IBM.

One of my many goals in 2009 is to get a firm grasp of cloud computing.  It starts today.  Following is the best explanation I’ve come across yet (thank you for the link John Troyer) where cloud computing is explained to me like I’m a five year old so I could understand it better.  I know there are others that “don’t get it”.  I hope this helps.

VMware appoints Tod Nielsen as Chief Operating Officer

January 6th, 2009

He’s a former Borland, BEA, Oracle, and Microsoft executive. This appears to be a direct appointment by VMware CEO Paul Maritz whom Nielsen will report to directly and has worked with in the past.

I hope Maritz, Nielsen, and everyone on down in VMware comes out hitting home runs in 2009. My faith and trust is in them and their innovation and leadership.

Read the official VMware press announcement here.

VMware User Groups (VMUGs)

January 6th, 2009

Lafe Low wrote a nice article for Redmond magazine about the history, fundamentals, and benefits of IT user groups. You may not be aware of this but there are many active VMware User Groups (VMUGs) around the world that you can get involved in.

The largest VMUG event I’ve heard of is in the Netherlands where they had 600 or more attendees at their event just recently. The Dutch are coo-coo for VMware virtualization and it shows by their VCP numbers I’m told.

I lead the Minneapolis area VMUG and we meet quarterly, as do many of the other VMUGs. We had one of our largest meetings last December with 150+ people. Our numbers have been steadily growing.

Back to the user group article, many of the benefits Lafe talks about apply to VMUGs:

“Most user group members feel that face-to-face sharing is essential.”

“The best peers make the best professors.”

“The feverish pace at which new technologies are introduced and integrated into the technological landscape helps make professional-level user groups an essential element. For the new generation of IT pros coming on, the technology has gone off the scale. You have every kind of protocol thrown at you, computers are faster and growing exponentially. Tech professionals have a lot more on their plate to deal with. This incessant technological upheaval naturally leads those who have chosen IT as a profession to seek out the counsel of their peers. We’re seeing them come to user groups in swarms. Their future lies in tying to existing IT pros. There’s a population desperately searching for connectivity and knowledge and advancement for their own sake and for the sake of the industry.”

Read Lafe’s entire user group article here.

There are still many individuals and companies that do not have the understanding or adaptation of virtualization. Meeting after meeting, I see new faces attend, unsure of what this virtualization stuff is, and after the meeting they walk away with a much better understanding, and even better, the confidence they needed to reach out to others and ultimately begin their virtualization adventure.

Jean Williams heads the user group division at VMware and does a great job with the help of Kristyn Ha. I would like to point out, however, that VMUGs are run by the users, for the users. This isn’t corporate propaganda jammed down our throats with a potato masher. These are peer level meetings with a wide range of expertise, knowledge, and experience. It’s straight talk about VMware virtualization benefits, strategies and related 3rd party products, and we don’t sugar coat, overlook, or ignore VMware issues.

I strongly encourage the joining of a VMware User Group. Here are some useful links to help you get started:

VMware User Groups official home page

Become a VMware User Group member today!

No VMUG in your area? Start one!

VMware Communities (forums) – VMware User Groups

Email the VMware User Group team at VMware

Virtualization, Dilbert style

January 6th, 2009

Along with The Far Side, Dilbert is one of my favorite comic strips.  I was unaware Scott Adams produced a short series on virtualization back in February 2008; I’m ashamed to say I hadn’t seen them.  So this may be old news but there is absolutely NO WAY this isn’t getting posted on my blog.  I consider it essential. 


Guest blog entry: VMotion performance

January 5th, 2009

Good afternoon VMware virtualization enthusiasts and Hyper-V users whom Microsoft has condoned on your behalf that you don’t have a need for hot migration if you have an intern and $50,000 cash.

Simon Long has shared with us this fantastic article he wrote regarding VMotion performance.  More specifically, fine tuning concurrent VMotions allowed by vCenter.  This one is going in my document repository and tweaks ‘n’ tricks collection.  Thank you Simon and everyone please remember that virtualization is not best enjoyed in moderation!

Simon can be reached via email at contact (at) as well as @SimonLong_ on Twitter.

I’ll set the scene a little….

I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.

I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.

“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.

3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, Chris Dye, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.

After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!

But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once.

Here are the steps to increase the amount of Simultaneous VMotion Migrations per Host.

1. RDP to your vCenter Server.
2. Locate the vpdx.cfg (Default location “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”)
3. Make a Backup of the vpxd.cfg before making any changes
4. Edit the file in using WordPad and insert the following lines between the <vpdx></vpdx> tags;


5. Now you need to decide what value to give “maxCostPerHost”.

A Cold Migration has a cost of 1 and a Hot Migration aka VMotion has a cost of 4. I first set mine to 12 as I wanted to see if it would now allow 3 VMotions at once, I now permanently have mine set to 24 which gives me 6 simultaneous VMotions per Host (6×4 = 24).

I am unsure on the maximum value that you can use here, the largest I tested was 24.

6. Save your changes and exit WordPad.
7. Restart “VMware VirtualCenter Server” Service to apply the changes.

Now I know how to change the amount of simultaneous VMotions per Host, I decided to run some tests to see if it actually made any difference to the overall VMotion Performance.

I had 2 Host’s with 16 almost identical VM’s. I created a job to Migrate my 16 VM’s from Host 1 to Host 2.

Both Hosts VMotion vmnic was a single 1Gbit nic connected to a CISCO Switch which also has other network traffic on it.

The Network Performance graph above was recorded during my testing and is displaying the “Network Data Transmit” measurement on the VMotion vmnic. The 3 sections highlighted represent the following;

Section 1 – 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 6 simultaneous VMotions.
Time taken = 3.30

Section 2 – This was not a test, I was simply just migrating the VM’s back onto the Host for the 2nd test (Section 3).

Section 3 – 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 2 simultaneous VMotions.
Time taken = 6.36

Time Different = 3.06
3 Mins!! I wasn’t expecting it to be that much. Imagine if you had a 50 Host cluster…how much time would it save you?
I tried the same test again but only migrating 6 VM’s instead of 16.

Migrating off 6 VM’s with only 2 simultaneous VMotions allowed.
Time taken = 2.24

Migrating off 6 VM’s with 6 simultaneous VMotions allowed.
Time taken = 1.54

Time Different = 30secs

It’s still an improvement all be it not so big.

Now don’t get me wrong, these tests are hardly scientific and would never have been deemed as completely fair test but I think you get the general idea of what I was trying to get at.

I’m hoping to explore VMotion Performance further by looking at maybe using multiple physical nics for VMotion and Teaming them using EtherChannel or maybe even using 10Gbit Ethernet. Right now I don’t have the spare Hardware to do that but this is definitely something I will try when the opportunity arises.

Update 4/5/11Limit Concurrent vMotions in vSphere 4.1 by Elias Khnaser.

Update 10/3/12:  Changes to vMotion in vSphere 4.1 per VMware KB 1022851:

In vSphere 4.1:
  • Migration with vMotion and DRS for virtual machines configured with USB device passthrough from an ESX/ESXi host is supported
  • Fault Tolerant (FT) protected virtual machines can now vMotion via DRS. However, Storage vMotion is unsupported at this time.
    Note: Ensure that the ESX hosts are at the same version and build.
In addition to the above, vSphere 4.1 has improved vMotion performance and allows:
  • 4 concurrent vMotion operations per host on a 1Gb/s network
  • 8 concurrent vMotion operations per host on a 10Gb/s network
  • 128 concurrent vMotion operations per VMFS datastore

Note: Concurrent vMotion operation is currently supported only when source and destination hosts are in the same cluster. For further information, see the Configuration Maximums for VMware Sphere 4.1 document.

The vSphere 4.1 configuration maximums above remain true for vSphere 5.x.  Enhanced vMotion operations introduced in vSphere 5.1 also count against the vMotion maximums above as well as the Storage vMotion configuration maximums (8 concurrent Storage vMotions per datastore and 2 concurrent Storage vMotions per host as well as 8 concurrent non-vMotion provisioning operations per host).  Eric Sloof does a good of explaining that here.