Make VirtualCenter highly available with VMware Virtual Infrastructure

November 17th, 2008 by jason 6 comments »

A few days ago I posted some information on how to make VirtualCenter highly available with Microsoft Cluster Services.

Monday Night Football kickoff is coming up but I wanted follow up quickly with another option (as suggested by Lane Leverett): Deploy the VirtualCenter Management Server (VCMS) on a Windows VM hosted on a VMware Virtual Infrastructure cluster. Why is this a good option? Here are a few reasons:

  1. It’s fully supported by VMware.
  2. You probably already have a VI cluster in your environment you can leverage. Hit the ground running without spending the time to set up MSCS.
  3. Removing MSCS removes a 3rd party infrastructure complexity and dependency which requires an advanced skill set to support.
  4. Removing MSCS removes at least one Windows Server license cost and also removes the need for the more expensive Windows Enterprise Server licensing and the special hardware needs required by MSCS.
  5. Green factor: Let VCMS leverage the use of VMware Distributed Power Management (DPM).

How does it work? It’s pretty simple. A virtualized VCMS shares the same advantages any other VM inherently has when running on a VMware cluster:

  1. Resource balancing of the four food groups (vProcessor, vRAM, vDisk, and vNIC) through VMware Distributed Resource Scheduler (DRS) technology
  2. Maximum uptime and quick recovery via VMware High Availability (HA) in the event of a VI host failure or isolation condition (yes, HA will still work if the VCMS is down. HA is a VI host agent)
  3. Maximum uptime and quick recovery via VMware High Availability (HA) in the event of a VMware Tools heartbeat failure (ie. the guest OS croaks)
  4. Ability to perform host maintenance without downtime of the VCMS

A few things to watch out for (I’ve been there and done that, more than once):

  1. If you’re going to virtualize the VCMS, be sure you do so on a cluster with the necessary licensed options to support the benefits I outlined above (DRS, HA, etc.) This means VI Enterprise licensing is required (see the licensing/pricing chart on page 4 of this document). I don’t want to hide the fact that a premium is paid for VI Enterprise licensing, but as I pointed out above, if you’ve already paid for it, the bolt ons are unlimited use so get more use out of them.
  2. If your VCMS (and Update manager) database is located on the VCMS, be sure to size your virtual hardware appropriately. Don’t go overboard though. From a guest OS perspective, it’s easier to grant additional virtual resources from the four food groups than it is to retract them.
  3. If you have a power outage and your entire cluster goes down (and your VCMS along with it), it can be difficult to get things back on their feet while you don’t have the the use of the VCMS. Particularly if you’ve lost the use of other virtualized infrastructure components such as Microsoft Active Directory. Initially it’s going to be command line city so brush up on your CLI. It really all depends on how badly the situation is once you get the VI hosts back up. One example I ran into is host A wouldn’t come back up. Host B wasn’t the registered owner of the VM I needed to bring up. This requires running the vmware-cmd command to register the VM and bring it up on host B.

Well, I missed the first few minutes of Monday Night Football, but everyone who reads (tolerates) my ramblings is totally worth it.

Go forth and virtualize!

Migrating from ESX to ESXi? Watch the HCL differences

November 16th, 2008 by jason 2 comments »

If you are an ESX shop contemplating a migration to ESXi and all that it has to offer, keep in mind the two products do not share an identical list of compatible hardware. Generally speaking, the ESXi Hardware Compatibility List (HCL) is more limited in terms of hardware than ESX. There is a chance the hardware you have running ESX in the datacenter right now may not be supported by VMware for running ESXi.

Here are some examples of hardware models that are supported for ESX but not for ESXi:

Dell

  • 1855
  • 2600
  • 2650
  • 6650
  • M805
  • R300
  • T300

HP

  • DL360G3/G4
  • DL380G3/G4
  • DL385
  • DL580G2/G3/G4
  • DL585
  • DL760G2
  • All BL p-class blades

IBM

  • HS20-8843
  • LS20-8850
  • 346-8840
  • 366
  • 445
  • 460-8872
  • x3400
  • x3500-7977
  • x3550
  • x3650
  • x3655
  • x3755
  • x3800
  • x3950

Keep in mind VMware updates the various HCLs on average about once per week so what is or is not on the HCL today, may change at any time. Typically, hardware is added to the HCL on a regular basis as it is certified by VMware.

The ESX and ESXi 3.5u3 HCL can be found here.

What’s up with the broken Windows Search?!

November 15th, 2008 by jason 4 comments »

Can someone please explain to me why the search function in Windows hasn’t worked reliably since about Windows 2000 or Windows 98? It got exponentially worse with the release of Microsoft’s “Desktop Search”.

See below. I’m looking for the location of a file called wuauclt.exe. The “Desktop Search” feature, which when installed, replaces the default built in Windows Search, can’t find the file:

search1

I have to resort to the old faithful command prompt search to find what I’m looking for:

search2

This is beyond stupid.

VMware Server 2.0 installation error

November 15th, 2008 by jason 6 comments »

I ran into an installation error this afternoon with VMware Server 2.0 on Windows Server 2003 R2.  It was a dialog box that popped up and said “The System Administrator has set policies to prevent this installation“.  The error prevents the installation from proceeding and offers only one course of action which is to click “Ok” and the installation terminates.

The workaround is as follows:

  1. Click Start
  2. Click Run
  3. gpedit.msc
  4. Drill down to Computer configuration
    1. Windows Settings
      1. Security Settings
        1. Software Restriction Policies
  5. Right click Software Restriction Policies
  6. Choose Create New Policies
  7. On the right hand side, right click Enforcement
  8. Choose Properties
  9. Down below, choose All users except local administrators
  10. Click OK
  11. Close Group Policy
  12. Open a Command Prompt
  13. Run the command gpupdate /force
  14. When local group policy is finished updating, re-run the VMware Server 2.0 setup

Microsoft focus on features and marketing talk; delivery a secondary objective

November 15th, 2008 by jason No comments »

Looking for the SCVMM 2008 management pack? It doesnt exist much to the contrary of the documentation. Microsoft marketing is five steps ahead of the development team. That’s old news.

I’m so glad I’m not a Microsoft virtualization customer. What a let down it must be at times. Want to put Hyper-V and VMware pricing head to head? This is what you get. These are the soft details price comparisons don’t reveal. This is what Microsoft isn’t going to openly admit at the booth. For my dollar, today’s innovative features are what count.

Anyway, that’s quite enough press for Microsoft on my blog. Read the rest at VCritical.

He sed / she sed

November 14th, 2008 by jason 4 comments »

Sed (Stream EDitor) is a powerful Unix and Linux utility which has the ability to parse text files and make changes.  It’s practically a gold mine for scripting.  Fortunately, this utility has been part of the modified Red Hat Enterprise Linux service console (COS) in VMware ESX current and previous versions.  Used in my deployment scripts, it serves me well for the purposes of rapidly standing up new or rebuilding existing ESX hosts.

Admittedly, I don’t have a real strong Linux background.  My foundation has been primarily Microsoft Windows, and before Windows, Microsoft DOS and Commodore 64/VIC-20 BASIC programming which actually did provide me a nice basis for command line going forward.  When I began building ESX hosts, I soon discovered scripting was where it’s at for efficiency and more imporantly consistency.  I learned the use of sed by borrowing snippets of example code available on the internet and VMware Community forums.  By the way, the sed rabbit hole goes very deep.  I’ve only learned the find/replace function of it to get me by for ESX scripting which is just the tip of the iceberg.

I learned something about sed recently that probably would have helped me during my learning process of sed a while back.  The majority of sed examples found on the internet (and probably in the man pages) use a forward slash / as the delimiter.  This is because the / is natively used in ed, more, and vi.  What I discovered is that any character can actually be used as a delimiter.  This would have been helpful for me to know earlier because some of the scripting I do involves Linux paths which of course make use of the forward slash.  Adding an additional forward slash for sed scripting made for an awful lot of forward slashes in one line and at times made my eyes hurt.  Similar to building a complex Microsoft Excel formula whith a lot of parentheses and trying to keep track of the number of open parens versus the number of close parens. 

Let me show a few examples and maybe you can see what I’m talking about.

Here’s one line of an actual script I use to add the full path display in the COS (there are three / delimiters in this example, see if you can find them):

sed -e “s/\\h \\\W/\\h \\\w/g” /etc/bashrc.old > /etc/bashrc

Forward and backwards slashes that butt up against each other are commonly reffered to as “the picket fence effect”.  In the example above, for my sanity I could have chosen a different delimiter, such as an underscore, so that the script looked like this instead:

sed -e “s_\\h \\\W_\\h \\\w_g” /etc/bashrc.old > /etc/bashrc

Here’s another example which prevents the VMFS2 module from loading at startup on ESX3 hosts:

sed -e “s/echo \”vmfs2 vmfs2\”/\#echo \”vmfs2 vmfs2\”/g” /etc/init.d/vmware.old > /etc/init.d/vmware

More picket fence, not quite as bad though.  Maybe this time I use use the asterisk as a delimiter so that it looks like this instead:

sed -e “s*echo \”vmfs2 vmfs2\”*\#echo \”vmfs2 vmfs2\”*g” /etc/init.d/vmware.old > /etc/init.d/vmware

So it boils down to choosing the right sed delimiter for the line of code you’re working with.  Kind of like choosing the right bottle of wine for your meal, and choosing the right cigar for afterwards.

I’ve always said that much of the fun of this career I have chosen is the opportunity to learn something new every day and put it to practical use.  Today has been no exception.  Some days are more fun to learn than others.  It depends on the conditions and circumstances…

Make VirtualCenter highly available with Microsoft Cluster Services

November 12th, 2008 by jason 6 comments »

When VirtualCenter was first introduced, many could make the argument that VC was simply a utility class service that provided centralized management for a virtual infrastructure. If the VirtualCenter Management Server (VCMS) was rebooted in the middle of the day or if the VC services were stopped for some reason, it wasn’t too big of a deal providing the outage didn’t interrupt a key task such as a VMotion migration or a cloning process.

Times are changing. VirtualCenter is becoming a fairly critical component in the VI and high availability of VC and the VCMS is becoming increasingly important. Several factors have contributed to this evolution. To identify just a few:

  • Virtual infrastructures are growing rapidly in the datacenter. The need for a functioning centralized management platform increases exponentially.
  • Increased and more granular VC alerting capabilities are relied upon to keep administrators updated with timely information about the load and health of the VI.
  • The introduction of more granular role base security extended Virtual Infrastructure Client or Web Access deployment to more users and groups in the organization increasing dependability on VC and visibility of downtime.
  • The exposure of the VC API/SDK encouraged many new applications and tools to be written against VC. I’m talking about tools that provide important functions such as backup, reporting, automation, replication, capacity analysis, sVMotion, etc. Without VC running, these tools won’t work.
  • The introduction of plugins. Plugins are going to be the preferred bolt on for most administrators because they snap in to a unified management interface. Obvious dependency on VC.
  • The introduction of new features native to VC functionality. DRS, HA, DPM, VCB, Update Manager, Consolidation, snapshot manager, FT, SRM, etc. Like the bullet above, all of these features require a healthy functioning VCMS.
  • The Virtual Datacenter OS was announced at VMworld 2008 and is comprised of the following essential components: Application vServices, Infrastructure vServices, Cloud vServices, and Management vServices. I don’t know about you, but to me those all sound like services that would need to be highly available. While it is not yet known exactly how existing VI components transform into the VDC-OS, we know the components are going to be integral to VMware’s vision and commitment to cloud computing which needs to be highly available, if not continuously available.

VirtualCenter has evolved from a cornerstone of ESX host management into the the entire foundation on which the VI will be built on. Try to imagine what the impacts will be in your environment if and when VirtualCenter is down now and in the future. Dependencies may have waltzed in that you didn’t realize.

A single VCMS design may be what you’re used to, but fortunately there exists a method by which VC may be made highly available on a multi-node Microsoft Cluster. This document, written by none other than my VI classroom training instructor Chris Skinner, explains how to cluster VirtualCenter 2.5.

If you’re on VirtualCenter 2.0.x, follow this version of the document instead.

Update:  Follow up post here.