Posts Tagged ‘VirtualCenter’

Anti-affinity rules are not honored in cluster with more than 2 virtual machines

March 27th, 2009

We can put a man on the moon and we can hot migrate virtual machines with SMP and gigs of RAM, but we can’t create anti-affinity rules with three or more VMs. This has been a thorn in my side since 2006, long before I requested it fixed in February 2007 on the VMTN Product and Feature Suggestions forum.

VMware updated KB article 1006473 on 3/26 outlining anti-affinity rule behavior when using three or more VMs:

“This is expected behavior, as anti affinity rules can be set only for 2 virtual machines.

When a third virtual machine is added any rule becomes disabled (with 2.0.2 or earlier).

There has been a slight change in behavior with VirtualCenter 2.5, wherein input validation occurs, where a third virtual machine added produces a warning message indicating a maximum of two virtual machines only can be added to this rule.

To workaround this, create more rules to cover all of the combinations of virtual machines.

For example, create rules for (VM1 & VM2), then (VM2 & VM3), and (VM1 & VM3).”

That last sentence is what has been burning my cookies for the longest time. In my last environment, I had several NLB VMs which could not be on the same host for load balancing and redundancy purposes. Rather than create a minimum amount of rules to intelligently handle all of the VMs, I was left with no choice but to create several rules for each potentially deadly combination.

Work harder, not smarter. Come on VMware.

DPM best practices. Look before you leap.

March 16th, 2009

It has previously been announced that VMware’s Distributed Power Management (DPM) technology will be fully supported in vSphere. Although today DPM is for experimental purposes only, virtual infrastructure users with VI Enterprise licensing can nonetheless leverage its usefulness of powering down ESX infrastructure during non-peak periods where they see fit.

Before enabling DPM, there are a few precautionary steps I would go through first to test each ESX host in the cluster for DPM compatibility which will help mitigate risk and ensure success. Assuming most, if not all, hosts in the cluster will be identical in hardware make and model, you may choose to perform these tests on only one of the hosts in the cluster. More on testing scope a little further down.

This first step is optional but personally I’d go through the motions anyway. Remove the hosts to be tested individually from the cluster. If the hosts have running VMs, place the host in maintenance mode first to displace the running VMs onto other hosts in the cluster:

3-16-2009 10-31-19 PM

If the step above was skipped or if the host wasn’t in a cluster to begin with, then the first step is to place the clustered host into maintenance mode. The following step would be to manually place the host in Standby Mode. This is going to validate whether or not vCenter can successfully place a host into Standby Mode automatically when DPM is enabled. One problem I’ve run into is the inability to place a host into Standby Mode because the NIC doesn’t support Wake On LAN (WOL) or WOL isn’t enabled on the NIC:

3-16-2009 10-25-53 PM

Assuming the host has successfully been place into Standby Mode, use the host command menu (similar in look to the menu above) to take the host out of Standby Mode. I don’t have the screen shot for that because the particular hosts I’m working with right now aren’t supporting the WOL type that VMware needs.

Once the host has successfully entered and left Standby Mode, the it can be removed from maintenance mode and added back into the cluster. Now would not be a bad time to take a look around some of the key areas such as networking and storage to make sure those subsystems are functioning properly and they are able to “see” their respective switches, VLANs, LUNs, etc. Add some VMs to the host and power them on. Again, perform some cursory validation to ensure the VMs have network connectivity, storage, and the correct consumption of CPU and memory.

My point in all of this is that ESX has been brought back from a deep slumber. A twelve point health inspection is the least amount of effort we can put forth on the front side to assure ourselves that, once automated, DPM will not bite us down the road. The steps I’m recommending have more to do with DPM compatibility with the different types of server and NIC hardware, than they have to do with VMware’s DPM technology in and of itself. That said, at a minimum I’d recommend these preliminary checks on each of the different hardware types in the datacenter. On the other end of the spectrum if you are very cautious, you may choose to run through these steps for each and every host that will participate in a DPM enabled cluster.

After all the ESX hosts have been “Standby Mode verified”, the cluster settings can be configured to enable DPM. Similar to DRS, DPM can be enabled in a manual mode where it will make suggestions but it won’t act on them without your approval, or it can be set for fully automatic, dynamically making and acting on its own decisions:

3-16-2009 10-24-33 PM

DPM is an interesting technology but I’ve always felt in the back of my mind it conflicts with capacity planning (including the accounting for N+1 or N+2, etc.) and the ubiquitous virtualization goal of maximizing the use of server infrastructure. In a perfect world, we’ll always be teetering on our own perfect threshold of “just enough infrastructure” and “not too much infrastructure”. Having infrastructure in excess of what what would violate availability constraints and admission control is where DPM fits in. That said, if you have a use for DPM, in theory, you have excess infrastructure. Why? I can think of several compelling reasons why this might happen, but again in that perfect world, none could excuse the capital virtualization sin of excess hardware not being utilized to its fullest potential (let alone, powered off and doing nothing). In a perfect world, we always have just enough hardware to meet cyclical workload peaks but not too much during the valleys. In a perfect world, virtual server requests come planned so well in advance that any new infrastructure needed is added the day the VM is spun up to maintain that perfect balance. In a perfect world, we don’t purchase larger blocks or cells of infrastructure than what we actually need because there are no such things as lead times for channel delivery, change management, and installation that we need to account for.

If you don’t live in a perfect world (like me), DPM offers those of us with an excess of infrastructure and excuses an environment friendly and responsible alternative to at least cut the consumption of electricity and cooling while maintaining capacity on demand if and when needed. Options and flexibility through innovation is good. That is why I choose VMware.

Andrew Kutz joins Hyper9

February 28th, 2009

This news is a little over a week old but I just found out two nights ago while reading vExpert profiles and it’s definitely worth repeating.

Andrew Kutz is a recently named vExpert by VMware, Inc. and a well known developer in the VMware community. Andrew has authored a number of VirtualCenter plugins, of which the most famous might be his free Storage VMotion (sVMotion) plugin which provides VMware administrators a GUI interface to hot migrate VM storage from one LUN to another. Andrew has received well deserved praise for his work because he makes the lives of VI administrators easier.

Hyper9 is a startup company in Austin, TX that works in the virtualization infrastructure management space, developing tools that automate the management of virtualization in the datacenter. Hyper9 recently secured an additional round of investment funding and it would seem they are totally serious about delivering quality products to the virtualization community in the hiring of Andrew Kutz. What can we expect out of this? Given what I’ve seen from Andrew in the past, I’ll guess the future will be plugin based architecture which I think makes a lot of sense and is probably what the majority of the community wants.

Congratulations to both Andrew Kutz and Hyper9. I look forward to your accomplishments with great anticipation!

Read the official announcement from Hyper9 here.

VMworld Europe 2009 Wednesday

February 25th, 2009

I need to make this quick because it’s 3:25am and I risk not waking up for my sessions tomorrow in four hours.

It has been a whirlwind of a day. I arrived at the conference and found out by word of mouth VMware had announced their list of vExpert recipients. I was one of 300 people on the planet chosen as a vExpert based on various contributions I’ve made to the VMware virtualization community including forum activity over the years, evangelism through blogging, podcasting, VMUG leader, etc. I can proudly display the silver vExpert logo on my blog. This is a nice gesture from VMware to recognize people in the community that have given much of themselves to promote a product that they believe in and help shape the future of our planet.

I attended some good sessions. Yesterday I learned about VMware vCenter Chargeback. It’s features seem fairly consistent with other chargeback solutions I’ve tested. Still not much automated help for estimating VM infrastructure and operational costs prior to VM deployment for new servers/applications/workloads but when I asked about this during Q&A, the speaker assured me this would be coming in future versions. vCenter Chargeback is also going to add an additional database to vCenter. For those with vCenter and Update Manager, we’re now up to three separate databases. The chargeback database has to be pretty simple – I don’t understand why additional tables can’t be created in the vCenter database for chargeback eliminating the need for an additional database. Where I get nervous about databases is during vCenter upgrades and the additional time and effort required to repair or back out from a failed database upgrade.

I attended a few more good sessions today. Most notably TA15 Protecting your vCenter Server using vCenter Heartbeat and LAB11 VMware VI Toolkit for Windows (PowerShell) where I was assisted by none other than Carter Shanklin whom many might recognize from Twitter. Carter also delivered a knockout session which I hear is currently ranked #1 among all sessions. In the past, it wasn’t a show stopper for the virtual infrastructure if VirtualCenter was down for a brief to moderately extended period of time. With all of the components announced recently that tie into vCenter Server, the importance of vCenter Server uptime (and vSphere as a whole) has increased exponentially. vCenter Server is evolving into an enterprise application requiring 99.9999% uptime. The additional moving parts will introduce increased complexity and potentially new operational and support standards for vSphere. Our models will need to be adapted to fit the uptime requirements of vSphere.

DSC00677The second VMTN: Ask the Experts session was held today. We had more people in the community lounge than yesterday but still not many visitors who were looking for assistance with VMware virutalization. I was pulled away by Jessica, a Systems Engineer with VMware, along with a camera crew to give an interview on vExpert along with some general chit chat about the show. That interview will be posted on

DSC00711Moving along into the evening, I attended the VMworld party which started at 20:00. It was a great time. To the left, that’s Mike Laverick walking through the entrance with his video camera in tow. There was live music including two women who kicked things off with some techno violin. I thought the food was pretty good and there was quite a variety. The presentation of the food was also interesting as you will see from the photos below. The man at the bar in the brown jacket with his back turned to me is none other than Jonathan Reeve of Hyper9.

DSC00727 DSC00716 DSC00734

DSC00715 DSC00728 DSC00721

DSC00738I was the lucky recipient of a Flip Video mino HD from Tripwire.

This is a slick little video recording device which records up to 1 hour of HD video and sound on internal memory.

I hung out with a lot friends and talked with some interesting people like Brian Madden who always has interesting stories to tell.

DSC00732 DSC00743

DSC00745The story behind this picture is that while waiting in line to get into the party, I buried five Euros worth of coins in this hot candle wax 1/2 inch deep along with a few US coins. The experiment was to see if anyone would dig them out after the candle wax had dried. When we left the party, all the Euro coins were gone. Someone later took them out of the hot wax and peeled the wax shavings off which were found on the ground. They left the US coins and my card.

The VMworld party ended at midnight and some of us walked down the strip to a small techno bar that was jam packed. There was a live DJ, dancing, drinking, and making out. Like the Veeam boat party the other night, I ran into Tarry Singh, Strategic Business Consultant: Data Center (Cloud Computing, Virtualization). Tarry is funny as hell and that guy can definitely cut a rug. I’ve got a lot of video footage from tonight but cannot post any due to very poor upload speeds from the hotel.

It’s late and the Hyper9 alien and I are tired. Goodnight.


Putting some money where my VMware mouth is

February 15th, 2009

I came home this afternoon from a Valentines Day wedding in North Dakota to find that my one and only workstation in the house (other than the work laptop) had a belated Valentines Day present for me:  It would no longer boot up.  No Windows.  No POST.  No video signal.  No beep codes.


I was feeling adventurous and I needed a relatively quick and inexpensive fix.  I decided to take one of the thin clients I received from Chip PC via VMworld 2008 plus a freshly deployed Windows XP template on the Virtual Infrastructure and promote this VDI solution to main household workstation status for the next few weeks.  The timing on this could not have been better.  The upcoming Minnesota VMUG on Wednesday March 11th is going to be VDI focused.  I guess I’ll have more to contribute at that meeting than I had originally planned on.  With any luck, Chip PC will be in attendance and we can discuss some things.

The thin client:  Chip PC Xtreme PC NG-6600 (model: EX6600N, part number: CPN04209).


  • RMI – Alchemy Au 1550, 500MHz RISC processor (equivalent to 1.2GHz x86 TC processors)
  • 128MB DDR RAM
  • 64MB Disk-On-Chip with TFS
  • 128-bit 3D graphics acceleration engine with separate 2x8MB display memory SDRAM
  • Dual DVI ports each supporting 1920×1200 16-bit color.  Supports quad displays up to 1024×768
  • Audio I/O
  • 4 USB 2.0 ports
  • 10/100 Ethernet NIC
  • Power draw:  3.5W work mode, .35W sleep mode
  • OS:  Enhanced Microsoft Windows CE (6.00 R2 Professional)
  • Integrated applications (Plugins – note plugins are downloaded at no charge from the Chip PC website and are not, by default, embedded or included with the thin client – just enough OS concept)
    • Citrix ICA
    • RDP 5.2 and 6
    • Internet Explorer 6.0
    • VDM Client
    • VDI Client
    • Media Player
    • VPN Client
    • Ultra VNC
    • Pericom (Team Talk) Terminal Emulation
    • LPD Printer
    • ELO Touch Screen
  • Compatibility
    • Citrix WinFrame, MetaFrame, and Presentation Server 4.5
    • MS Windows Server 2000/2003
    • MS Windows NT 4.0 – TS Edition
    • VMware Virtual Desktop Interface using RDP
  • Full support of both local and network printers:  LPD, LPR, SMB, LPT, USB, COM
  • Support for USB mass storage (thumb drives – deal breaker for me)
  • Support for wireless USB NIC (not included)
  • etc. etc. etc.


Truth be told, this isn’t really a promotion in the sense that I had already performed extensive testing on it.  I hadn’t even taken the thing out of the box yet other than to register it for the extended warranty.  I’ve had only a little experience on these devices as I have an identical unit in the lab at work which I’ve spent a total of 30 minutes on.  To the best of my knowledge, this is the Cadillac unit from Chip PC.

I don’t have any fancy VDI brokering solutions here in the home lab and I’m not up to speed on VMware View so the plan is to leverage Thin Client -> RDP -> Windows XP desktop on VMware Virtual Infrastructure 3.5.

I think this is going to be a good test.  A trial by fire of VDI (granted, a fairly simple variation).  I spout a lot about the goodness that is VMware and now I’ll be eating some of my own dog food from the desktop workspace.  I’m a power user.  I’ve got my standard set of applications that I use on a regular basis and I’ve got a few hardware devices such as a flatbed scanner, iPod Shuffle, USB thumb drives, digital cameras, etc.  I should know within a short period of time whether or not this will be a viable solution for the short term.  Also add to the mix my wife’s career.  She uses our home computer to access her servers at work on a fairly regular basis.  Lastly, my wife sometimes works from home while I’m away at the office or traveling.  It’s going to be critical that this solution stays up and running and continues to be viable for my wife while I’m remote and not able to provide computer support.

So where am I at now?  I’ve got the VDI session patched along with my most critical applications installed to get me by in the short term:  Quicken, SnagIt, network printer, and Citrix clients.  I’ll install MS Office later but for now I can use the published application version of Office on my virtualized Citrix servers.  I’ve been listening some Electro House on on the VDI and music quality is as good as it was on my PC before it died, although it doesn’t completely drive my 5.1 surround in the den.  Pretty sure I’m getting 2.1 right now.  Oh well, at least the sub is thumpin.  Shhhh… the thin client is sleeping:


So what else?  As long as I’m throwing caution to the wind, I think it’s time to take the training wheels off VMware DPM (Distributed Power Management) and see what happens in a two node cluster.

2-15-2009 10-53-10 PM

Based on the environment below, what do you think will happen?  CPU load is very low, however, memory utilization is close to being over committed in a one host scenario. Will DPM kick in?

2-15-2009 10-53-59 PM

Most of my infrastructure at home is virtual including all components involving internet access both incoming and outgoing.  If the blog becomes unavailable for a while in the near future, I’ll give you one guess as to what happened.  🙂

No matter what the outcome, aka Roman Haug – you are no longer welcomed to republish my blog articles.  Albeit flattering, the fact that you have not even so much as asked in the first place has officially pissed me off.  You publish my content as if it were your own, written by you as indicated by the “by Roman” header preceeding each duplicated post.  Please remove my content from your site and refrain from syndicating my content going forward.  Thank you in advance.

Update: Roman Haug has offered an apology and I believe we have reached an understanding.  Thank you Roman!

VMGURU to release 4 chapters of VI3 book today

February 10th, 2009

Scott Herold of and co-author of the book VMware Infrastructure 3: Advanced Technical Design Guide and Advanced Operations Guide has announced today the release of four of the book’s chapters in PDF format today.

I’ve read the previous version of this book a few years ago and I’m in the middle of reading the current version.  I HIGHLY recommend this book.  It is worth it’s weight in gold and the fact that the authors are going to begin giving it away for free to the virtualization community is baffling to me but yet at the same time it is a symbol of their generosity and commitment to providing the community with top notch technical and operations detail on VMware virtual infrastructure.

Generally speaking, many technical authors don’t make a pile of money writing books.  Be sure to thank the authors Ron Oglesby, Scott Herold, and Mike Laverick for their hard work and generosity.

More information about this book can be found here and here.  Stay tuned to for the official release of these chapters which should happen sometime today.

Train Signal training discount through the month of February

January 31st, 2009

Train Signal is offering an astounding 25% off any virtualization product they sell through the month of February 2009.

Here is a short sample of their VMware ESX training video where instructor David Davis talks about templates and cloning virtual machines:

To take advantage of the 25% off, use the code BOCHENET at checkout.

I know first hand that the economy is tough.  Take advantage of this offer and get top shelf training for your dollar.  Train Signal offers a 90 day money back guarantee if you are not completely satisfied.