This isn’t exactly the type of news I like to report but at the same time it can’t be ignored. The February/March 2009 issue of Virtualization Review magazine has an article starting on page 12 where Rick Vanover puts Hyper-V, XenServer, and ESX head to head to head. Some of the conclusions drawn are startling:
“For the first two tests of heavy workloads, VMware underperformed both XenServer and Hyper-V. For the lighter workloads on the third test, the results were almost indistinguishable across the platforms, but ESX had the best results in three of the four categories.”
“After doing these comparisons of ESX to Hyper-V and XenServer, it’s clear that at the hypervisor level, ESX is optimized for a large number of less-intensive workload VMs. For intensive workloads that may not be optimized for memory overcommit apps, Hyper-V and XenServer should definitely be considered-even if that means adding another hypervisor into the data center.”
Rick is saying that both Hyper-V and XenServer deliver better performance for the heavy workloads. ESX is better suited for lighter workloads and actually will handle more of them than Microsoft and Citrix making it the better “scale up” solution. Rick also points out that ESX offers the clear advantage of memory over commit which could not be benchmarked against Hyper-V and XenServer due to memory over commit not being available in the latter two products. For a moment, let’s assume that Rick’s findings are 100% accurate. From an options standpoint, how do you feel about scaling up versus scaling out for the lighter workloads having equal performance across all three platforms? Personally, I’d lean towards higher consolidation ratios, less capital expenditures, less datacenter and utility bill consumption. That’s the ESX option.
I’m concerned that I’m hearing ESX is underperforming against the underdogs. I’m not at all saying Rick’s tests are invalid but I am looking for a response from VMware that is either published, or in the form of ESX4 taking an obvious performance lead once again in benchmark tests. Charging a premium for a lesser performing hypervisor doesn’t sound like the right formula for success.
Update: Slight goof on the title of this blog post. Originally it stated “XenApp” where I meant “XenServer”
Update: Microsoft and VMware have responded to the original article’s performance analysis.
I read the article. I have an issue with the claim that ‘heavy’ loads were tested when the hardware config used only local RAID 5 storage. How heavy of a disk IO workload will you be able to simulate with that? I would like to see them test a config with more variables – an FC SAN backend with lots of spindles or iSCSI. Really push the host server with workloads that have more disk IO than local disk can provide and I think you will see some different results. A test scenario that doesn’t include SAN does not reflect an enterprise environment.
Thank you for the comment Shawn. While I could agree that the setup didn’t accurately reflect a true enterprise environment, the fact remains that it was a level playing field and ESX did not deliver the performance that Hyper-V and XenServer did using the same configuration.
Small businesses won’t have enterprise level infrastructure but they may still have enterprise level applications and/or heavy workloads. With this information, it sounds like the choice for SMBs is clear: Microsoft or Citrix. You have no idea how painful it is for me to say that.
I really don’t know what to think of the outcome.
As XenServer has a paravirtualized scsi driver (while ESX is emulating a scsi card (lsilogic/buslogic)), it’s storage performance is actually slightly outperforming ESX in most situations (but not much).
This is also what I found here: http://vmfaq.com/entry/33/
..and also what Ron Oglesby found in his testing here: http://www.brianmadden.com/blogs/gabeknuth/archive/2008/04/03/xen-vs-esx-head-to-head-video.aspx
Lars
Jason,
I know it had to really hurt to say that, but look at it this way; no matter what “religion” you are if customers are “getting religion” everyone wins right? Like my grandfather used to say, “a rising tide floats all boats”.
Cheers
Michael
I would like to see some more thorough testing across many different hardware platforms before I would conclude that any of the solutions outperforms another. Even though the config was equal, it was only one. If you don’t have ‘enterprise’ infrastructure (ie no SAN) you don’t get HA, live migration, etc, either. You can’t have those features without shared storage so any of the ‘free’ options will meet your needs. Who is making revenue on that?
Couldn’t agree more with Keen here.
Tarry
Jason,
Great article…see my response here
Thanks!
KLC
Must resist urge to comment on vReligion…..Focus, Sean. Focus! Okay, I’m back from the ledge.
All the hypervisor performance bake-offs have been with local storage. Ron’s bake-off: http://www.brianmadden.com/blogs/gabeknuth/archive/2008/04/03/xen-vs-esx-head-to-head-video.aspx and the VRC bake-offs http://www.virtualrealitycheck.net/ have all been with local storage. Have there been any bake-offs with SAN storage for the back-end? If this same performance benefit doesn’t exist in a shared storage environment, then why would the results matter? The environments small enough to not afford a SAN, wouldn’t have a workload that could benefit from the added performance.
Good post to get people talking, Jason.
Interesting. I’m with Larstr on this one. I do think the paravirtulized architecture has an influence on this.
Could it have something to do with ESX’s 32-bit hypervisor as well, where both XenServer and Hyper-V has 64-bit on their hypervisor. Any could see this as a reason?
And forgot this point. Normally you would also mix and match your heavy workloads with your smaller workloads in a ESX environment.
Hyper-V and Xen use the hardware virtualization assist from the cpu’s. VMware is using software binary translation for 32bit guests. In the early days of Intel VT en AMD-v vmware’s software binary translation was faster then the hardware assist. These days the hardware assist seems to outperform software virtualization. This is why xen and hyper-v are outperforming vmware on 32bit guests.
Just wait for ESX4 🙂 VMware will also start using the hardware assist more and more.
just wondering: did they test 64bit guests? since esx uses hardware assist for these guest one can assume this wil perform the same as xen and hyper-v.
>Rick is saying that both Hyper-V and XenServer
>deliver better performance for the heavy workloads.
>ESX is better suited for lighter workloads and
>actually will handle more of them than Microsoft and
>Citrix making it the better “scale up” solution
This reminds me of what my “high-end” colleagues say about the difference between the mainframe (a truck) and an x86 server (a car). Sure the car is faster but you can bring more stuff with a truck. It’s interesting how all these concepts are now scaling down into my x86 realm as well and into the software space.
Massimo.
P.S. my comment assumes the test is realistic. I am sure there are many people at VMware that would like to challenge the results.
A couple of comments here. I made very clear in the review not to declare a winner – this is because the ‘winning’ criteria was not established. The one comment I made that flirted with that is where Hyper-V exceeded my expectations. Stuart Yarost informed me – well you’ve done a good test if your preconceptions are broken. As I would ‘write the plan’ to meet the expected results.
Back to topic – everyone is assuming I’m offering this as information for the enterprise. Not so. I really am targeting this to the customer who is going to select the free hypervisors for small, unmanaged installations. This is very relevant to them, and as for local storage vs. SAN – yeah, I’ll test SAN – but then get flamed for this configuration or lack of that configuration. And I’d be happy to run 3 weeks of test on your SAN!
Also – note that I plan on more of these type of tests… So, give me ideas!