After enabling FT on a VM – subtleties to expect

September 16th, 2009 by jason No comments »

While using VMware vSphere, you may encounter a situation where you cannot edit the memory resource settings (shares, reservations, and limits) for a particular VM on the resources tab. The memory resource settings section will be completely grayed out. In addition, a label will clearly state “Memory resources-cannot edit” as shown below:

In this particular instance, the underlying cause for this condition is VMware Fault Tolerance (FT) has been enabled on the FT “primary” VM. The fact that the memory resource settings cannot be modified is by design and is used as a means to help guarantee the FT “secondary” VM stays in vLockstep with the primary. What has actually happened is that when FT was enabled on the VM, a memory reservation was set equal to the amount of memory configured for the VM. This eliminates VMkernel swap file for the VM managed by the host in all cases, not just for FT enabled VMs.

What other subtle changes can you expect when you enable VMware Fault Tolerance (FT) on a VM?

DRS will be disabled for the FT enabled primary VM, although it may be VMotioned manually in cases where maintenance needs to be performed on the ESX(i) host. FT secondaries may also be migrated by right clicking on the FT primary VM and choosing the Fault Tolerance menu item to “Migrate Secondary”:

Thin provisioned disks will be converted to a Thick type:

A FT “secondary” VM will be created on another host in the cluster which will consume CPU and memory on the secondary host. It will share VM storage with the FT “primary”. VM networking is disabled on the FT “secondary” to eliminate the obvious problem of a duplicate machine on the network, however, other considerable host based network traffic will be consumed for two purposes:

  1. Initial creation of the FT “secondary” – dedicated VMotion network is used
  2. Continuous FT logging traffic – dedicated FT logging network is used

If hardware MMU feature exists in the host CPUs (AMD RVI/Intel EPT), the feature is disabled in the VM. This will force a power off of the VM before FT can be enabled.

Storage vMotion will be disabled for the FT enabled VM.

The hypervisor may slow down execution of the FT “primary” VM if the FT “secondary” is not able to keep pace with the FT “primary” using vLockstep technology.

Snapshotting functionality will be disabled. Furthermore and maybe more importantly, backups requiring snapshot technology won’t work.

Virtual hardware that is not compatible with FT will be disabled (ie. USB, Sound, etc.)

vSMP (multiple CPUs) in the VM is not supported. FT can’t be enabled.

Physical RDMs in the VM are not supported. FT can’t be enabled.

For more information on VMware Fault Tolerance, see VMware vSphere™ 4 Fault Tolerance: Architecture and Performance, VMware vSphere Availability Guide, and Xtravirt’s Disaster Recovery and VMware vSphere 4.0 Fault Tolerance whitepaper,

Thank you Gabe and Brenda

September 14th, 2009 by jason No comments »

I’d like to take a moment to thank two people, Gabe and Brenda, for their new and continuous friendship. They hail from the Netherlands and the pair are two of the nicest, funniest, and fun loving people I’ve met. I was first introduced to them in person earlier this year in Cannes, France during the VMworld Europe 2009 virtualization conference. Gabe was attending as a VMware user and Brenda joined him to study conference attendees in their preferred habitat, as well as for some sight seeing. Being from the U.S., I was quite out of my environment while traveling for the first time in France but they made me feel welcomed, teaching me some of the local customs as well as bits and pieces of the French language: “Merci beaucoup” – “Thank you very much” – a valuable phrase for a clueless tourist to individually thank each person for their assistance.

I met up with them again at VMworld 2009 in San Francisco, CA. This time they treated myself, my wife, and my kids to a nice Italian dinner Thursday evening after the conclusion of the conference. In addition, they showered my children with authentic Dutch gifts. Gabe and Brenda, if you are reading this, we very much appreciated this – Thank You! I hope one day we will meet again so that I can reciprocate. Chances are good as I’ve mentally committed to attend at least one VMworld annually, expending whatever efforts it takes to get there.

Where can you find this dynamic duo?

Brenda maintains a very interesting blog called Virtual Gipsy which offers an Anthropologist’s perspective of a tight knit virtualization community. Follow her on Twitter: @b_renda

Gabe runs an excellent virtualization blog called Gabe’s Virtual World and is particularly good with video editing. Follow him on Twitter: @gabvirtualworld

Saturday Grab Bag

September 12th, 2009 by jason No comments »

Here’s a collection of quick hits I’ve been meaning to get to. Individually, their content is a bit on the short side for the length I normally like to write so I thought I’d throw them together in a single post and see how it comes out.

Tasks and Events List Lengths

First up is the listing of Tasks and Events in the vSphere Client. Have you ever started troubleshooting an issue in the vSphere client by looking at the Tasks or Events and the chronological listing of events doesn’t go back far enough to the date or time you’re looking for? Not finding the logs you’re looking for in the vSphere Client usually means you need to open a PuTTY session and start sifting through logs in /var/log/ or /var/log/vmware/ in the Service Console. The reason for this is that the vSphere Client, by default, is configured to tail the last 100 entries in the Tasks or Events list. You can find this setting in your vSphere Client by choosing “Edit|Client Settings” then choose the “Lists” tab:

Simply increase the value from 100 to whatever you’d like, with 1,000 being the highest allowable value. Notice that when this number is increased, you will immediately see more history. In other words, you don’t have to necessarily wait for time to pass and more historical events to accumulate to see the additional rows of information. Also note that this is a vSphere Client setting which is retained client side and applies to both vCenter Server and ESX(i) host connections.

Collecting diagnostic information for VMware products

Like any offering from a software or hardware vendor, VMware products aren’t perfect. During your VMware experience, you may run into a problem which requires the intervention of VMware support. More often than not, VMware is going to ask you to generate a support bundle which consists of a collection of diagnostic and configuration files and logs. Following this paragraph is a link to VMware KB1008524 which contains links to creating support bundles for various VMware products. Note that in some cases there are different methods for different versions of the same product. If you choose to create a VMware SR online, it is helpful to have created these log bundles in advance so you can attach them to the SR. If you’ve done VMware support long enough, you already know how to FTP log bundles to VMware after an SR number has been generated.

Collecting diagnostic information for VMware products

New VMware Update Manager won’t download ESX(i) patches

Scenario: You’ve built a new VMware vCenter Server in addition to a new VMware Update Manager Server (VUM). After properly configuring Update Manager as well as the necessary internet, proxy, baseline, and scheduled task settings, VUM proceeds to download Windows, Linux, and application patches, but it won’t download ESX(i) host patches. As I found out by trench experience, the cause is because no ESX(i) hosts have been added to the vCenter Server and thus no hosts are being managed by VUM. You need to add at least one ESX(i) host to vCenter Server before VUM will be triggered to suck down all the host updates. One might then ask why guest patches are being downloaded. The only answer I have for the inconsistent behavior is due the fact that ESX(i) host patches are downloaded from VMware, while guest OS and application patches are downloaded from a completely different source, Shavlik. The mechanics behind the download processes obviously differ between the two.

What vCenter Server is this ESX(i) host managed by?

Scenario: You administer a large VMware virtual infrastructure with many vCenter Servers. You need to manage or configure a host or cluster but haven’t the slightest idea what vCenter Server to connect to. You can easily find out by attempting a Virtual Infrastructure Client connection to the host in question. Shortly after providing the necessary host credentials, the IP address of the vCenter Server managing this host will be revealed:

Now in theory, you could establish a Virtual Infrastructure Client connection to the IP address, however, I don’t like this because it dirties up the cached connection list with IP addresses which are meaningless short of having them all memorized. I prefer to take it a step further by opening a Command Prompt and using the command ping -a <IP_address> to reveal the name of the vCenter Server managing the host:

The command above reveals jarjar.boche.mcse as the vCenter Server which is managing the ESX(i) host I was wanting to manage via the vCenter Server.

I’m sure a PowerShell expert will follow up with a script which makes this process easier but this a good example to follow if you don’t have PowerShell or the VI Toolkit (Power CLI) installed.

Top 3 New York Style Cheesecake Offerings

September 5th, 2009 by jason No comments »
  1. Timberlodge Steakhouse – Easily and consistently the best cheesecake I’ve ever had. Excellence from the whipped topping to the graham cracker crust. Ginormous portions also.
  2. Rainforest Cafe – Had this last night in San Francisco. It doesn’t come real close to Timberlodge cheesecake but it’s pretty good and will definitely do in a pinch.
  3. Fogo De Chao – Had this cheesecake Wednesday night after the Q3 Minneapolis VMUG. I don’t like the hard outer texture as much, however, once I dug in, I found it to be very delicious. It has a tasty graham cracker crust similar to Timberlodge cheesecake and the strawberries and whipped topping were great as well.
  4. Cheesecake Factory – One would think that by virtue of their name, they’d have the best. Not so. I keep going back expecting it will be better and it never is. It’s not a far fetched idea that one day I’ll find a cheesecake that will push Cheesecake Factory into 4th place. At that point they should really feel ashamed. It’s official, Cheesecake Factory has now fallen to 4th place.

VCP4 Exam

September 5th, 2009 by jason No comments »

After a moderate beta phase, the VCP4 exam officially went public at VMworld 2009. Late Wednesday evening after the VMworld party, I decided that since VMware was offering the VCP4 exam at the Moscone Center at a heavily discounted rate, I would give it a shot first thing Thursday morning in lieu of attending the morning sessions. Passing this exam was on my development objectives for 2009 (along with VCDX certification) and time is starting to run out. OK, to be completely honest, the number of tweets I had seen recently of those who passed the VCP4 exam at VMworld as well as the beta made me a bit jealous and filled me with both encouragement and confidence.

My normal approach to certification is reading books and lab time. Although I hadn’t studied for the exam or even looked at the blueprint (a swell recipe for failure, I personally wouldn’t recommend it), I have been using vSphere 4 quite a bit in my home lab over the past several months. I also attended the two day “What’s New” vSphere course via WebEx but I don’t believe it provided a lot value towards the VCP4 exam.

In addition, I studied for, sat, and passed the Enterprise exam a little over a month ago which in my opinion was quite a bit more difficult than the VCP3 exam. Technically speaking, the Enterprise exam covers VI3 and not vSphere, but conceptually there is still plenty of overlap between VI3 and vSphere 4

Lastly, I had been toying with and troubleshooting the vSphere virtual infrastructure that VCDX #7 Duncan Epping provided attendees at the vExpert booth in the VMworld 2009 Solutions Exchange. As luck would have it, some of the things I was working on applied to the VCP4 exam and were fresh in my mind.

I passed the exam with a score of 350 out of a possible 500. A score of 300 or better is required to pass the exam and a score of 350 is required to be eligible for VMware Certified Instructor (VCI) status. This appears to be the same new grading curve used in the Enterprise exam and I imagine the Design exam for VCDX candidates (I have not sat the Design exam yet so I’m not 100% certain on that).

In a word, my experience was that I found the exam to be fair. 85 questions. 90 minutes. All multiple choice/multiple select, a few with exhibits. No interactive/hands-on/live lab scenarios although I would have preferred them. In comparison it was a degree tougher than the VCP3 exam. I attribute that to the fact that the vSphere content is new and I hadn’t properly prepared. Special thanks to the candidate who blurted on his way out of the exam room as I was walking in “This exam is REALLY HARD – definitely NO JOKE”. You made me feel as if I had just kissed $105 goodbye. 🙄

As with the Enterprise exam, I found time to be a threat as I was left with only three minutes to review about 20 questions I had marked. Poor exam time management seems to be a recurring theme with me lately where it wasn’t in my earlier years. I’m not sure if the exams are getting harder or I’m just getting slower in my old age. Probably a combination of both. Lately I tend to go into deep thought for a number of minutes on some questions. Instead, I should recognize that if the answer doesn’t come to me within 10 seconds, I should quickly choose the best answer, mark the question, and move on. I think the dilemma becomes that sometimes there is more than one best answer and that’s where I end back up in the deep thought.

For passing the exam, VMware gave me an additional VMworld pin, a “VCP4 certified” baseball cap, and flashy VCP battery powered glasses which my 3 year old daughter absolutely loves. Thank you VMware.

Update 10/5/09:  VCI pass mark is 350, not 400. This has been corrected in the paragraph above.

VMworld 2009 Wall of Datacenter Video

September 4th, 2009 by jason No comments »

I’m hanging out in my hotel room on Friday night with my family at Fisherman’s Wharf in San Francisco. I’ve fired up my laptop and I’m just starting to sift through the initial pile of great information published thus far on VMworld 2009.

Without a doubt, there will be many pictures, videos, blogs, and tweets to come out of the show. Here’s one of my favorites so far. Richard Garsthagen, Senior Evangelist at VMware EMEA, interviews “Dan”, whom I believe is an Architect on the VMware Lab Staff team. Dan talks about the $35 million wall of datacenter on display at the bottom of the Moscone Center escalator. As many of the attendees learned, this datacenter was used to power the nearly 40,000 VMs at VMworld 2009. Great video gentlemen!

Closely related, here’s a time lapse photography video of the datacenter build:

VMware Announces vCloud Express

September 1st, 2009 by jason No comments »

Today at VMworld 2009, VMware announced vCloud Express! For those still struggling with cloud as a concept, technology, or offering, this is where the tangible rubber meets the road with VMware virtual infrastructure.

The VMware vCloud™ Express service delivers the ability to provision infrastructure on-demand, via credit card, and pay for use by the hour. As a VMware Virtualized ™ service, it ensures compatibility with other VMware environments both internally and with external services.

  • Quick, easy access to VMware Virtualized™ infrastructure for prototyping and development from external third party providers.
  • Increase flexibility and time to market, and reduce capex and resource challenges associated with trying to address the fluctuating infrastructure needs of development, staging, and production teams.
  • By leveraging the VMware platform, the VMware vCloud™ Express service retains the robustness, interoperability and reliability that VMware is known for while delivering the easy access and cost-effectiveness of a transactional service model.
  • Five partners currently offering VMware vCloud™ Express service (as beta): Terremark, Hosting.com, and BlueLock in the Americas; Logica in EMEA; and Melbourne IT in the APAC.

The announcement was coupled with an on-stage web-based demonstration with Terremark, whom if I remember correctly, was involved with the interactive cloud provisioning presentation during a keynote speech at VMworld Europe 2009. As most keynote demonstrations usually go, the process was very slick, streamlined, and painless.

It will be interesting to watch VMware compete with other big cloud providers in existence. How much market share will VMware gain in the first year? What impact, if any, will “beta” status have on VMware vCloud Express adoption rates?