Archive for March, 2009

VI Toolkit Quick Reference Guide

March 14th, 2009

Virtu-Al (Alan Renouf) has posted a great two-page cheat sheet for the VMware VI Toolkit version 1.5.

This gem of a document is similar to VI3 card created by Forbes Guthrie over at Excellent job gentlemen!

While you’re at Virtu-Al’s site, check out all the sample code and scripts.  Chances are you could implement one or more of these puppies in your environment to configure ESX or ESXi.  Scripting is definitely one of the ways to become more efficient and agile and it’s a great way to ensure consistency across your environment.  PowerShell and VI Toolkit is where’s it at.  I think they are going to be here for a long time.

Top 10 referring pages year to date

March 14th, 2009

Top 10 referring pages year to date:


I’ve been receiving a steady increase in traffic thanks to the referrals above. Rich Brambley of swiped two of the top 10 spots. Thanks a lot!

3-14-2009 1-48-41 PM

Here is how March 2009 is stacking up so far for referrals:


Referrals tell me not only the number of people browsing my site, but they also reveal what what other web pages are being browsed from the standpoint of what site they just traveled in from. In this case of the number 10 spot, Ken Cline’s blog. Ken’s blog is brand new and he’s already making waves. Nice job Ken!

Interested in seeing more virtualization blog stats? Check out Duncan Epping’s Top 10 referrers over the last 30 days.

New sponsor coming Monday which I’m excited to announce 🙂

Horray! We’re mobile device friendly

March 9th, 2009

I received the feedback that the blog was not easily readable on mobile/handheld devices (Blackberry, iPhone, etc.)  Taking the suggestion from a few friends to install the MobilePress plugin for WordPress, the blog was mobile friendly within five minutes. While I had always “dealt” with the blog’s rendering on my own Blackberry, I hadn’t realized it could be improved (with so much ease).  I like it much better now as a lot of the “noise” has been removed. Thank you for the suggestions and feedback! Update:  The MobilePress plugin has been disabled for the time being due to a bug.

Rapid Virtualization Indexing (RVI)

March 8th, 2009

I’m mildly excited for the upcoming week. If all goes well, I’ll be upgrading to AMD Opteron processors which support a virtualization assist technology called Rapid Virtualization Indexing (or RVI for short).

There is overhead introduced in VMware virtualization via the virtual machine monitor (VMM) and comes in three forms:

  1. Virtualization of the CPU (using software based binary translation or BT for short)
  2. Virtualization of the MMU (using software based shadow paging)
  3. Virtualization of the I/O devices (using software based device emulation)

RVI is found in AMD’s second generation of virtualization hardware support and it incorporates MMU (Memory Management Unit) virtualization. This new technology is designed to eliminate traditional software based shadow paging methods for MMU virtualization thereby reducing the overhead in bullet #2 above. VMware lab tests show that RVI provides performance gains of up to 42% for MMU-intensive benchmarks and up to 500% for MMU-intensive microbenchmarks.

How it works:

Software based shadow page tables store information about the guest VM’s physical memory location on the host. The VMM had to intercept guest VM page table updates to keep guest page tables and shadow page tables in sync. By now you can probably see where this is going: applications and VMs which had frequent guest page table updates were not as efficient as those with less frequent guest page table updates.

The above is similar to guest VM kernel mode calls/context switching to access CPU ring 0. Previously, the architecture wouldn’t allow it directly via the hardware so the VMKernel had to intercept these calls and hand-hold each and every ring 0 transaction. Throw 10,000+ ring 0 system calls at the VMKernel per second and the experience starts to become noticeably slower. Both Intel and AMD resolved this issue specifically for virtualized platforms by introducing a ring -1 (a pseudo ring 0) which guest VMs will be able to access directly.

VMware introduced support for RVI in ESX 3.5.0. RVI eliminates MMU related overhead in the VMM by relying on the technology built into the newer RVI capable processors to determine the physical location of guest memory by walking an extra level of page tables maintained by the VMM. RVI is AMD’s nested page table technology. The Intel version of the technology is called Extended Page Tables (EPT) and is expected sometime this year.

One of the applications of RVI that interests me directly is Citrix XenApp (Presentation Server). XenApp receives a direct performance benefit from RVI because it is an MMU-intensive workload. VMware’s conclusion in lab testing was that XenApp performance increased by approximately 29% using RVI. By way of the performance increase, we can increase the number of concurrent users on each virtualized XenApp box. There are two wins here: We increase our consolidation ratios on XenApp and we reduce the aggregate number of XenApp boxes we have to manage due to more densely populated XenApp servers. This is great stuff!

There is a caveat. VMware observed some memory access latency increases for a few workloads, however, they tell us there is a workaround. Use large pages in the guest and the hypervisor to reduce the stress on the Translation Lookaside Buffer (TLB). VMware recommends that TLB-intensive workloads make extensive use of large pages to mitigate the higher cost of a TLB miss. For optimal performance, the ESX VMM and VMKernel aggressively try to use large pages for their own memory when RVI is used.

For more information and deeper technical jibber jabber, please see VMware’s white paper Performance of Rapid Virtualization Indexing (RVI). Something to note is that all testing was performed on ESX 3.5.0 Update 2 with 64 bit guest VMs. I give credit to this document for the information provided in this blog post, including two directly quoted sentences.

For some more good reading, take a look at Duncan Epping’s experience with a customer last week involving MMU, RVI, and memory over commit.

Ken Cline joins the virtualization blogger continuum

March 8th, 2009

Ken Cline, a legend among the VMware VMTN community who needs no introduction, has started up a blog called Ken’s Virtual Reality. The recently crowned vExpert’s blog is described as a place where Ken will jot down his thoughts on all things virtual. In addition, I’m happy to report that his primary focus will be VMware technologies, but I know he will be objective when discussing the other virtualization platforms as well.

Ken’s first blog post gives us a nice primer on who he is, where he has been, and what technologies he has worked with. Ken has been around the block. You’ll be hard pressed to find a person with as much extensive experience that Ken has.

In Ken’s next blog post, he digs right in to the technical stuff. Ken responds to a Hyper-V vs. XenServer vs. ESX shootout and helps us make more sense of the data. I hope what Ken has written up to this point is a sign of things to come in the future.

Ken is from the Virgina/DC region and is interested in employment opportunities in that area including those which require government security clearance.  If you know of any, I’m sure he’d appreciate it if you dropped him a line.

Welcome Ken!

Say it isn’t so: Hyper-V and XenServer outperform ESX

March 7th, 2009

This isn’t exactly the type of news I like to report but at the same time it can’t be ignored. The February/March 2009 issue of Virtualization Review magazine has an article starting on page 12 where Rick Vanover puts Hyper-V, XenServer, and ESX head to head to head. Some of the conclusions drawn are startling:

“For the first two tests of heavy workloads, VMware underperformed both XenServer and Hyper-V. For the lighter workloads on the third test, the results were almost indistinguishable across the platforms, but ESX had the best results in three of the four categories.”

“After doing these comparisons of ESX to Hyper-V and XenServer, it’s clear that at the hypervisor level, ESX is optimized for a large number of less-intensive workload VMs. For intensive workloads that may not be optimized for memory overcommit apps, Hyper-V and XenServer should definitely be considered-even if that means adding another hypervisor into the data center.”

Rick is saying that both Hyper-V and XenServer deliver better performance for the heavy workloads. ESX is better suited for lighter workloads and actually will handle more of them than Microsoft and Citrix making it the better “scale up” solution. Rick also points out that ESX offers the clear advantage of memory over commit which could not be benchmarked against Hyper-V and XenServer due to memory over commit not being available in the latter two products. For a moment, let’s assume that Rick’s findings are 100% accurate. From an options standpoint, how do you feel about scaling up versus scaling out for the lighter workloads having equal performance across all three platforms? Personally, I’d lean towards higher consolidation ratios, less capital expenditures, less datacenter and utility bill consumption. That’s the ESX option.

I’m concerned that I’m hearing ESX is underperforming against the underdogs. I’m not at all saying Rick’s tests are invalid but I am looking for a response from VMware that is either published, or in the form of ESX4 taking an obvious performance lead once again in benchmark tests. Charging a premium for a lesser performing hypervisor doesn’t sound like the right formula for success.

Update: Slight goof on the title of this blog post. Originally it stated “XenApp” where I meant “XenServer”

Update: Microsoft and VMware have responded to the original article’s performance analysis.

Straighten out licensing in preparation for vSphere

March 6th, 2009

There is a lot of buzz accumulating about the anticipated release of VMware vSphere.  Are you ready for it?  Is your license portal ready for vSphere?  Does anyone remember the licesing upgrades from VI2 to VI3?  Did they go smooth for you?

Double check your answers and be sure.  Inaccurate license counts in your license portal are going to lead to frustrating problems when you attempt to upgrade to vSphere.  When you get to vSphere, your new license key(s) may be missing quantities or SKUs you’ve purchased in the past.  Pay extra special attention if you purchase through a reseller to be sure your license counts and SKUs in the portal are 100% accurate.

DO NOT wait until the release of vSphere to sort out your licensing issues.  I would anticipate a long line of people in the support queues who were not proactive in sorting out their licensing issues prior to the release of vSphere.  Taking care of this ahead of time will help guarantee a smooth vSphere upgrade and it will also help balance the call load on VMware’s support staff.

To verify your licensing, head to the VMware licensing portal:

3-6-2009 4-52-30 PM

Click “Find Serial Number”

3-6-2009 4-52-53 PM

Change the filter parameters as follows:

Change “License Category” to Purchased/Registered.  Doing so will show you more licensing than not doing so in some cases.

Change “Sort Results By” to Product then License Type.  Doing so will make the licenses easier to reconcile.

3-6-2009 5-05-47 PM

Now reconcile all of your serial numbers.  Be aware that there may be more than one page of licenses in your portal.  If you’re missing licenses, check for a page 2, page 3, etc.

For more help on licensing, including help in contacting VMware on licensing issues, see the following blog entry I wrote in January.