Archive for the ‘Virtualization’ category

NFS and Name Resolution

June 11th, 2010

Sometimes I take things for granted.  For instance, the health and integrity of the lab environment.  Although it is “lab”, I do run some workloads which are key to keep online on a regular basis. Primarily the web server which this blog is served from, the email server which is where I do a lot of collaboration, and the Active Directory Domain Controllers/DNS Servers which provide the authentication mechanisms, mailbox access, external host name resolution to fetch resources on the internet, and internal host name resolution.

The workloads and infrastructure in my lab are 100% virtualized.  The only “physical” items I have are type 1 hypervisor hosts, storage, and network.  By this point I’ll assume most are familiar with the benefits of consolidation.  The downside is that when the wheels come off in a highly consolidated environment, the impacts can be severe as they fan out and tip over down stream dependencies like dominos.

A few weeks ago I had decided to recarve the EMC Celerra fibre channel SAN storage.  The VMs which were running on the EMC fibre channel block storage were all moved to NFS on the NetApp filer.  Then last week, the Gb switch which supports all the infrastructure died.  Yes it was a single point of failure – it’s a lab.  The timing for that to happen couldn’t have been worse since all lab workloads were running on NFS storage.  All VMs had lost their virtual storage and the NFS connections on the ESX(i) hosts eventually timed out.

The network switch was replaced later that day and since all VMs were down and NFS storage had disconnected, I took the opportunity to gracefully reboot the ESX(i) hosts; good time for a fresh start.  Not surprised, I had to use the vSphere Client to connect to each host by IP address since at that point I had no functional DNS name resolution in the lab whatsoever. When the hosts came back online, I was about to begin powering up VMs, but instead I encountered a situation which I hadn’t planned for – all the VMs were grayed out, esentially disconnected.  I discovered the cause of this was that after the host reboot, the NFS storage hadn’t come back online – both NetApp and EMC Celerra – on both hosts.  There’s no way both storage cabinets and/or both hosts were having a problem at the same time so I assumed it was a network or cabling problem. With the NFS mounts in the vSphere client staring back at me in their disconnected state, it dawned on me – lack of DNS name resolution was preventing the hosts from connecting to the storage.  The hosts could not resolve the FQDN name of the EMC Celerra or the NetApp filer storage.  I modified /etc/hosts on each ESX(i) host, adding the TCP/IP address and FQDN for the NetApp filer and Celerra Data Movers.  Shortly after I was back in business.

What did I learn?  Not much.  It was more a reiteration of important design considerations which I was already aware of:

  1. 100% virtualization/consolidation is great – when it works.  The web of upstream/downstream dependencies makes it a pain when something breaks.  Consolidated dependencies which you might consider leaving physical or placing in a separate failure domain:
    • vCenter Management
    • Update Manager
    • SQL/Oracle back ends
    • Name Resolution (DNS/WINS)
    • DHCP
    • Routing
    • Active Directory/FSMO Roles/LDAP/Authentication/Certification Authorities
    • Mail
    • Internet connectivity
  2. Hardware redundancy is always key but expensive.  Perform a risk assessment and make a decision based on the cost effectiveness.
  3. When available, diversify virtualized workload locations to reduce failure domain, particularly to split workloads which provide redundant infrastructure support such as Active Directory Domain Controllers, DNS servers.  This can mean placing workloads on separate hosts, separate clusters, separate datastores, separate storage units, maybe even separate networks depending on the environment.
  4. Static entires in /etc/hosts isn’t a bad idea as a fallback if you plan on using NFS in an environment with unreliable DNS but I think the better point to discuss is the risk and pain which will be realized in deploying virtual infrastructure in an unreliable environment. Garbage In – Garbage Out.  I’m not a big fan of using IP addresses to mount NFS storage unless the environment is small enough.

New Microsoft .NET Framework Update Breaks vSphere Client

June 10th, 2010

Just a quick heads up to bring attention to an issue which I caught on Twitter.  VMware published KB 1022611 today which describes a new issue that is introduced by a recent Microsoft .NET Framework 2.0 SP2 & 3.5 SP1 update.  Upon installing the update, the vSphere Client stops working.  According to the article, the issue impacts ESX(i)3.5, 4.0, and vCenter 4.0.  Contrary to the topic of this blog post, I am not placing blame on Microsoft.  It remains unclear to me which company’s development staff is responsible for the software incompatibility.  Microsoft obviously issued the udpate which revealed the problem, but VMware has some skin in this as well in that they need to make sure they are following Microsoft .NET Framework development standards and best practices for their enterprise hypervisor management.

Key details from the VMware KB article:

The vSphere Clients, prior to the Update 1 release, cannot be used to access the vCenter Server or ESX hosts. A Microsoft update that targets the .NET Framework, released on June 9th 2010 is causing this issue. The update causes the vSphere Client to stop working.    To correct the issue there are two options that can be performed:

  • Remove the MS update from your Windows operating system. The vSphere Client works after the update is removed.

Note: This affects Windows XP, Windows 2003, Windows 2008, Windows Vista, and Windows 7.

Win A Free VMworld Pass From

June 6th, 2010

6-6-2010 12-31-26 PMThe economy has been rough.  Individuals and businesses have felt the impacts in various ways.  Reduction of income or revenue.  Increased operational expenses.  Reduction in valuation of homes or assets.  Downsizing of staff.  The slashing of budgets, including training, conferences, and travel.  Those who are in verticals which tail economic trends by a year or two will begin feeling the impacts soon.

As a reader of this blog, you already know VMworld 2010 in San Francisco is just a few months away.  If you’re like me, you’re wondering “How am I going to get there this year?”  Due to the reasons I’ve outlined above, details are sketchy on whether or not you’ll get to go.  Management says “Ask again in August, we’ll let you know.”  It doesn’t sound promising.  Or maybe you’ve already been told “It’s just too expensive given the economy, sorry.” would like to help.  If you can get yourself to the door of the Moscone Center, will get you in.  This is a $1,895 value if you were to purchase a conference pass at the door.  There is no purchase necessary for this contest other than your own T&E (transportation, hotel, van down by the river, etc.)  On Friday June 18th, 2010, one random and lucky winner who has followed the contest rules completely (detailed below) will be revealed.

The intent here is not to save a company money.  Rather, to make the difference between someone going to VMworld versus not going.  Therefore, I would appreciate it if entries would be limited to those who do not already have budget approval for the VMworld conference pass.  At the same time, should you win, you owe it to yourself and the other contestants to follow through and attend the conference.  It would be a shame for the pass to go to waste.  Perhaps another blogger or vendor would like to co-sponsor airfare or hotel for the winner.  Consider this an open invitation for co-sponsorship.

Be sure to read the VMworld 2010 FAQ so that you thoroughly understand the conference logistics, ensuring you are an eligible candidate to attend.

Update 6/6/10: I’m happy to announce that Gestalt IT has graciously offered to pay for the airfare.  In addition to the VMworld conference pass, Gestalt IT will provide the winner with round trip airfare, up to $500.  All we ask in return is that the recipient provide a post-VMworld write-up of what they learned from attending the conference.  This could be a written document, a blog post, a video, you choose.  Thank you Gestalt IT for your donation!

Contest Rules:

  1. Post one comment/reply and only one comment/reply to this blog article below.
    • Include your first and last name.
    • Provide a valid email address when completing the comment form.
    • Include a short bio about yourself and how you use VMware currently or how you would like to leverage VMware products.
    • Include three (3) things you are looking to gain from attending VMworld 2010 (ie. Why do you want to go?)
    • Contest entry must be recieved by noon CST Thursday June 17th, 2010.
  2. One (1) random winner will be chosen Thursday evening June 17th, 2010.
    • Winner will be contacted via email address provided above.
    • Winner will recieve a VMworld 2010 San Francisco conference pass.
    • Winner will receive airfare up to $500 from Gestalt IT.
    • Winner will provide a post-VMworld write-up of what they learned from attending the conference to Gestalt IT.
  3. Contest results will posted Friday June 18th, 2010.
  4. The conference pass is non-transferrable and non-refundable.
  5. Hotel, meals, and other expenses are not covered by
  6. No purchase necessary.

Good Luck!

Update 6/17/10: 


A name has been randomly drawn and we have a winner! Congratulations to contest winner Greg Stuart who will be receiving a VMworld 2010 conference pass and round trip airfare (up to $500).  Greg’s winning entry and BIO is listed below:

I currently work for an organization that has begun to leverage VMware more and more. I’m new to virtualization and would like to gain a better knowledge of the VMware products, attend some hands on sessions and come back with solutions that I can employ in our environment. The ability to discuss scenarios and solutions with vendors in person would be awesome.

I’m pleased with the outcome of the contest.  Greg is new to virtualization and I think there is a lot of valuable information he will be able to pick up at VMworld.  Better yet, VMworld is a 4 day event this year – Monday thru Thursday instead of 3 days as it was prior years.  This affords Greg the opportunity to take in a whole extra day of content.

Thank you to all who participated in the contest including Gestalt IT for contributed the round trip airfare.  Although there could ultimately be only one grand prize contest winner, my hope is that you all will make the show this year somehow.  There are nearly 90 comments/replies to this post explaining the values which VMworld can provide. Much of this content could be borrowed to write or improve your own compelling justification, hopefully earning you a trip to VMworld.

Everyone have a great weekend!

SRM Survey – Free SRM Book

June 1st, 2010

The VMware SRM team is conducting a formal survey on the SRM product and they’d like to hear your feedback.  VMware values your time and suggestions – in return for completing the survey, VMware will donate $10 to UNICEF (for the first 1,000 respondants) and you’ll be eligible to download an electronic copy of Mike Laverick’s Administring VMware Site Recoveyr Manager 4.0 book.

You can read more about this event here.

Complete the survey here.

vSphere Upgrade Prerequisites Checklist

May 27th, 2010

Upgrading your virtual infrastructure to vSphere?  Be sure to check out this handy reference from VMware:  vSphere Upgrade Prerequisites Checklist.  There are several areas which need to be considered and this document covers them all, including both requirements and recommendations.  If you’re a consultant who visits new customer sites on a regular basis, it wouldn’t be a bad idea to bring this with to each engagement, or at least a condensed version of it.

VMware Workstation Upgrade to 7.1

May 26th, 2010

Microsoft Windows 7 Professional 64-bit
VMware Workstation 7.0.1 build-227600

I had heard VMware Workstation 7.1 was released.  Unfortunately, the VMware Workstation “check for updates” feature doesn’t seem to be serving its intended purpose as it told me no updates were available.

I downloaded the installation package manually and performed the upgrade.  Two reboots were required:

  1. After the uninstall of my previous version of Workstation
  2. After the install of Workstation 7.1

I hope the usability experience is better than my upgrade experience.  I realize some of the reboot business is on the Microsoft Windows 7 operating system but come on, would someone please figure this out?  Is there no way to perform an in place upgrade of Workstation to minimize the reboots to one?

What’s New in VMware Workstation 7.1

•Support for 8 virtual processors (or 8 virtual cores) and 2 TB virtual disks.

•Support for OpenGL 2.1 for Windows Vista and Windows 7 guests.

•Greatly improved DirectX 9.0 graphics performance for Windows Vista and Windows 7 guests. Up to 2x faster than Workstation 7.

•Launch virtualized applications directly from the Windows 7 taskbar to create a seamless experience between applications in your virtual machines and the desktop.

•Optimized performance for Intel’s Core i3, i5, i7 processor family for faster virtual machine encryption and decryption.

•Support for more Host and Guest Operating Systems, including: Hosts: Windows 2008 R2, Ubuntu 10.04, RHEL 5.4, and more Guests: Fedora 12, Ubuntu 10.04, RHEL 5.4, SEL 11 SP1, and more.

•Now includes built in Automatic Updates feature to check, download, and install VMware Workstation updates.

•Ability to import and export Open Virtualization Format (OVF 1.0) packaged virtual machines and upload directly to VMware vSphere, the industry’s best platform for building cloud infrastructures.

ESX 3.x Host 64GB Addressable Memory Limit

May 24th, 2010

Some time ago, I became aware of an ESX 3.5.0 Update 2 (build 110268) host which had 128GB of RAM physically installed, but only 64GB RAM usable.  The host was showing 128GB of RAM, however, it was consuming 64GB of memory with no running VMs, leaving the other 64GB of RAM addressable for virtual machines.

After further research, it was determined that this host build did not contain the VMkernel change required to properly acknowledge the amount of physical memory installed on the IBM host hardware. 

VMware’s response was:

Prior to ESX 3.5 Update 3, the ability to address more than 64GB of memory on ESX Server 3.5 is suppressed by default. In a standard installation, a 36bit MTRR mask is forced, even though the machine may support 40bit mask values.  This means that the ESX Server may see any memory above 64GB as memory that is in use. For example, if an ESX server has 256GB of RAM, the Memory Usage counter displays 192GB in use and only 64GB free. If you attempt to create a virtual machine using memory exceeding the available 64GB of memory, you see an Insufficient Memory error.  This condition is documented in the following location:

The boot option force36BitMTRRMask is no longer required because of BIOS MTRR issues on certain platforms, ESX Server hosts previously failed to boot unless the VMkernel force36BitMTRRMask boot option was set to false.  ESX 3.5 Update 3 enables full support for memory up to 64GB with no need to specify a boot option.  As a result of this change, the force36BitMTRRMask VMkernel boot option is no longer supported. If the option is set, the result is no operation (NOP) and boot succeeds.

In conclusion, the resolutions are as follows:

1 ) Upgrade to ESX 3.5 Update 3 build 123630 or newer 

2 ) To utilize more than 64GB of RAM, use a larger MTRR mask by disabling VMkernel.Boot.force36BitMTRRMask from the advanced settings. 

To modify the MTRR mask configuration:

Log in to VirtualCenter as an administrator using the Virtual Infrastructure Client. (If not using VirtualCenter, log in to the ESX Server directly as root.)

From the Inventory click the ESX Server:

Click the Configuration tab.

Click Advanced Settings link.

Navigate to VMKernel>Boot.

Deselect the option for VMkernel.Boot.force36BitMTRRMask .

Reboot the ESX Server host for the change to take effect.

This information is perhaps a bit dated, but I know there are some older 3.x environments still in existence.  If those environments are running on host hardware with more than 64GB of RAM installed, this could prove to be insightful.