Saturday Grab Bag

September 12th, 2009 by jason No comments »

Here’s a collection of quick hits I’ve been meaning to get to. Individually, their content is a bit on the short side for the length I normally like to write so I thought I’d throw them together in a single post and see how it comes out.

Tasks and Events List Lengths

First up is the listing of Tasks and Events in the vSphere Client. Have you ever started troubleshooting an issue in the vSphere client by looking at the Tasks or Events and the chronological listing of events doesn’t go back far enough to the date or time you’re looking for? Not finding the logs you’re looking for in the vSphere Client usually means you need to open a PuTTY session and start sifting through logs in /var/log/ or /var/log/vmware/ in the Service Console. The reason for this is that the vSphere Client, by default, is configured to tail the last 100 entries in the Tasks or Events list. You can find this setting in your vSphere Client by choosing “Edit|Client Settings” then choose the “Lists” tab:

Simply increase the value from 100 to whatever you’d like, with 1,000 being the highest allowable value. Notice that when this number is increased, you will immediately see more history. In other words, you don’t have to necessarily wait for time to pass and more historical events to accumulate to see the additional rows of information. Also note that this is a vSphere Client setting which is retained client side and applies to both vCenter Server and ESX(i) host connections.

Collecting diagnostic information for VMware products

Like any offering from a software or hardware vendor, VMware products aren’t perfect. During your VMware experience, you may run into a problem which requires the intervention of VMware support. More often than not, VMware is going to ask you to generate a support bundle which consists of a collection of diagnostic and configuration files and logs. Following this paragraph is a link to VMware KB1008524 which contains links to creating support bundles for various VMware products. Note that in some cases there are different methods for different versions of the same product. If you choose to create a VMware SR online, it is helpful to have created these log bundles in advance so you can attach them to the SR. If you’ve done VMware support long enough, you already know how to FTP log bundles to VMware after an SR number has been generated.

Collecting diagnostic information for VMware products

New VMware Update Manager won’t download ESX(i) patches

Scenario: You’ve built a new VMware vCenter Server in addition to a new VMware Update Manager Server (VUM). After properly configuring Update Manager as well as the necessary internet, proxy, baseline, and scheduled task settings, VUM proceeds to download Windows, Linux, and application patches, but it won’t download ESX(i) host patches. As I found out by trench experience, the cause is because no ESX(i) hosts have been added to the vCenter Server and thus no hosts are being managed by VUM. You need to add at least one ESX(i) host to vCenter Server before VUM will be triggered to suck down all the host updates. One might then ask why guest patches are being downloaded. The only answer I have for the inconsistent behavior is due the fact that ESX(i) host patches are downloaded from VMware, while guest OS and application patches are downloaded from a completely different source, Shavlik. The mechanics behind the download processes obviously differ between the two.

What vCenter Server is this ESX(i) host managed by?

Scenario: You administer a large VMware virtual infrastructure with many vCenter Servers. You need to manage or configure a host or cluster but haven’t the slightest idea what vCenter Server to connect to. You can easily find out by attempting a Virtual Infrastructure Client connection to the host in question. Shortly after providing the necessary host credentials, the IP address of the vCenter Server managing this host will be revealed:

Now in theory, you could establish a Virtual Infrastructure Client connection to the IP address, however, I don’t like this because it dirties up the cached connection list with IP addresses which are meaningless short of having them all memorized. I prefer to take it a step further by opening a Command Prompt and using the command ping -a <IP_address> to reveal the name of the vCenter Server managing the host:

The command above reveals jarjar.boche.mcse as the vCenter Server which is managing the ESX(i) host I was wanting to manage via the vCenter Server.

I’m sure a PowerShell expert will follow up with a script which makes this process easier but this a good example to follow if you don’t have PowerShell or the VI Toolkit (Power CLI) installed.

Top 3 New York Style Cheesecake Offerings

September 5th, 2009 by jason No comments »
  1. Timberlodge Steakhouse – Easily and consistently the best cheesecake I’ve ever had. Excellence from the whipped topping to the graham cracker crust. Ginormous portions also.
  2. Rainforest Cafe – Had this last night in San Francisco. It doesn’t come real close to Timberlodge cheesecake but it’s pretty good and will definitely do in a pinch.
  3. Fogo De Chao – Had this cheesecake Wednesday night after the Q3 Minneapolis VMUG. I don’t like the hard outer texture as much, however, once I dug in, I found it to be very delicious. It has a tasty graham cracker crust similar to Timberlodge cheesecake and the strawberries and whipped topping were great as well.
  4. Cheesecake Factory – One would think that by virtue of their name, they’d have the best. Not so. I keep going back expecting it will be better and it never is. It’s not a far fetched idea that one day I’ll find a cheesecake that will push Cheesecake Factory into 4th place. At that point they should really feel ashamed. It’s official, Cheesecake Factory has now fallen to 4th place.

VCP4 Exam

September 5th, 2009 by jason No comments »

After a moderate beta phase, the VCP4 exam officially went public at VMworld 2009. Late Wednesday evening after the VMworld party, I decided that since VMware was offering the VCP4 exam at the Moscone Center at a heavily discounted rate, I would give it a shot first thing Thursday morning in lieu of attending the morning sessions. Passing this exam was on my development objectives for 2009 (along with VCDX certification) and time is starting to run out. OK, to be completely honest, the number of tweets I had seen recently of those who passed the VCP4 exam at VMworld as well as the beta made me a bit jealous and filled me with both encouragement and confidence.

My normal approach to certification is reading books and lab time. Although I hadn’t studied for the exam or even looked at the blueprint (a swell recipe for failure, I personally wouldn’t recommend it), I have been using vSphere 4 quite a bit in my home lab over the past several months. I also attended the two day “What’s New” vSphere course via WebEx but I don’t believe it provided a lot value towards the VCP4 exam.

In addition, I studied for, sat, and passed the Enterprise exam a little over a month ago which in my opinion was quite a bit more difficult than the VCP3 exam. Technically speaking, the Enterprise exam covers VI3 and not vSphere, but conceptually there is still plenty of overlap between VI3 and vSphere 4

Lastly, I had been toying with and troubleshooting the vSphere virtual infrastructure that VCDX #7 Duncan Epping provided attendees at the vExpert booth in the VMworld 2009 Solutions Exchange. As luck would have it, some of the things I was working on applied to the VCP4 exam and were fresh in my mind.

I passed the exam with a score of 350 out of a possible 500. A score of 300 or better is required to pass the exam and a score of 350 is required to be eligible for VMware Certified Instructor (VCI) status. This appears to be the same new grading curve used in the Enterprise exam and I imagine the Design exam for VCDX candidates (I have not sat the Design exam yet so I’m not 100% certain on that).

In a word, my experience was that I found the exam to be fair. 85 questions. 90 minutes. All multiple choice/multiple select, a few with exhibits. No interactive/hands-on/live lab scenarios although I would have preferred them. In comparison it was a degree tougher than the VCP3 exam. I attribute that to the fact that the vSphere content is new and I hadn’t properly prepared. Special thanks to the candidate who blurted on his way out of the exam room as I was walking in “This exam is REALLY HARD – definitely NO JOKE”. You made me feel as if I had just kissed $105 goodbye. 🙄

As with the Enterprise exam, I found time to be a threat as I was left with only three minutes to review about 20 questions I had marked. Poor exam time management seems to be a recurring theme with me lately where it wasn’t in my earlier years. I’m not sure if the exams are getting harder or I’m just getting slower in my old age. Probably a combination of both. Lately I tend to go into deep thought for a number of minutes on some questions. Instead, I should recognize that if the answer doesn’t come to me within 10 seconds, I should quickly choose the best answer, mark the question, and move on. I think the dilemma becomes that sometimes there is more than one best answer and that’s where I end back up in the deep thought.

For passing the exam, VMware gave me an additional VMworld pin, a “VCP4 certified” baseball cap, and flashy VCP battery powered glasses which my 3 year old daughter absolutely loves. Thank you VMware.

Update 10/5/09:  VCI pass mark is 350, not 400. This has been corrected in the paragraph above.

VMworld 2009 Wall of Datacenter Video

September 4th, 2009 by jason No comments »

I’m hanging out in my hotel room on Friday night with my family at Fisherman’s Wharf in San Francisco. I’ve fired up my laptop and I’m just starting to sift through the initial pile of great information published thus far on VMworld 2009.

Without a doubt, there will be many pictures, videos, blogs, and tweets to come out of the show. Here’s one of my favorites so far. Richard Garsthagen, Senior Evangelist at VMware EMEA, interviews “Dan”, whom I believe is an Architect on the VMware Lab Staff team. Dan talks about the $35 million wall of datacenter on display at the bottom of the Moscone Center escalator. As many of the attendees learned, this datacenter was used to power the nearly 40,000 VMs at VMworld 2009. Great video gentlemen!

Closely related, here’s a time lapse photography video of the datacenter build:

VMware Announces vCloud Express

September 1st, 2009 by jason No comments »

Today at VMworld 2009, VMware announced vCloud Express! For those still struggling with cloud as a concept, technology, or offering, this is where the tangible rubber meets the road with VMware virtual infrastructure.

The VMware vCloud™ Express service delivers the ability to provision infrastructure on-demand, via credit card, and pay for use by the hour. As a VMware Virtualized ™ service, it ensures compatibility with other VMware environments both internally and with external services.

  • Quick, easy access to VMware Virtualized™ infrastructure for prototyping and development from external third party providers.
  • Increase flexibility and time to market, and reduce capex and resource challenges associated with trying to address the fluctuating infrastructure needs of development, staging, and production teams.
  • By leveraging the VMware platform, the VMware vCloud™ Express service retains the robustness, interoperability and reliability that VMware is known for while delivering the easy access and cost-effectiveness of a transactional service model.
  • Five partners currently offering VMware vCloud™ Express service (as beta): Terremark,, and BlueLock in the Americas; Logica in EMEA; and Melbourne IT in the APAC.

The announcement was coupled with an on-stage web-based demonstration with Terremark, whom if I remember correctly, was involved with the interactive cloud provisioning presentation during a keynote speech at VMworld Europe 2009. As most keynote demonstrations usually go, the process was very slick, streamlined, and painless.

It will be interesting to watch VMware compete with other big cloud providers in existence. How much market share will VMware gain in the first year? What impact, if any, will “beta” status have on VMware vCloud Express adoption rates?

8 New ESX 3.5 Patches Released; 4 Critical

September 1st, 2009 by jason No comments »

Four new patches have been released for ESX 3.5.0. It appears ESXi, ESX4, and other versions of ESX are not impacted.

4 of the 8 patches are rated critical and should be evaluated quickly for application in your virtual infrastructure.

ID: ESX350-200908401-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates forcedeth driver
The forcedeth driver installed on the ESX hosts causes the NVIDIA nForce Network Controller NICs to lose network connectivity until the forcedeth driver is reloaded. This patch addresses the issue.

The affected NICS are:

  • NVIDIA nForce Professional 2200 MCP 1Gbe NIC
  • NVIDIA nForce Professional 2050 I/O companion chip 1Gbe NIC
  • NVIDIA nForce Professional 3600 1Gbe NIC

ID: ESX350-200908402-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates VMware Tools
After performing VMotion between ESX 3.0.x and ESX 3.5 hosts, virtual machines running on ESX 3.5 hosts are restarted in order to upgrade to the latest version of VMware Tools. After applying this fix, VMware Tools function as expected.

ID: ESX350-200908403-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates megaraid and mptscsi drivers
This patch fixes the following issues:

  • When the ESX host boots, the megaraid_sas driver heap gets depleted when claiming 4 LSI SAS RAID controllers on IBM System x3950 M2 Athena servers. This issue might cause the ESX host to stop booting. The fix increases the heap size for the megaraid_sas driver from 8 MB to 16 MB.
  • The mptscsi_2xx driver limits the discovery of targets to 63 SAS devices per LSI Serial Attached SCSI (SAS) host bus adapter (HBA). This fix increases the number of targets to the value returned by the HBA firmware.

ID: ESX350-200908404-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates vmkctl
When N-Port ID Virtualization (NPIV) enabled virtual machines are powered on on ESX hosts, a rescan issued from the VI Client results in an error message stating that the rescan failed, even if the rescan is successful.

ID: ESX350-200908405-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates vmkernel
Running the esxtop command on the service console of the ESX hosts lists high values for the max limited (%MLMTD) parameter for virtual machines when no max limited parameter is set. When the high values are listed, the performance of the virtual machines might be affected. In the VI Client, the max limited parameter is set in the Resources tab for CPU in Virtual Machine properties.

ID: ESX350-200908406-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates vmx
This patch provides the following:

  • Adds support for new SCSI-3 status values in the SCSI emulation for virtual machines.
  • Fixes an issue where powering on customized versions of Ubuntu virtual machines from the ESX hosts might cause the ESX hosts to stop responding.

ID: ESX350-200908407-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates kernel source and vmnix
This patch updates the service console kernel for the following fixes:

  • The forcedeth driver installed on the ESX hosts causes the NVIDIA nForce Network Controller NICs to lose network connectivity under certain circumstances. The affected NICS are:
    • NVIDIA nForce Professional 2200 MCP 1Gbe NIC
    • NVIDIA nForce Professional 2050 I/O companion chip 1Gbe NIC
    • NVIDIA nForce Professional 3600 1Gbe NIC
  • A bnx2x firmware dump issue.
  • The mptscsi_2xx driver limits the discovery of targets to 63 SAS devices per LSI Serial Attached SCSI (SAS) host bus adapter (HBA). This fix increases the number of targets to the value returned by the HBA firmware.

ID: ESX350-200908408-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates bnx2x driver
This patch fixes a bnx2x firmware dump issue.

vCalendar Launch

August 19th, 2009 by jason No comments »

Welcome! vCalendar is little something I have been working on in my spare time during the spring and summer months of 2009. I’ve had “Far Side” and “Dilbert” Page-A-Day calendars on my desktop at work for the past 15 years and towards the end of 2008 I thought a virtualization calendar would be nice to have, however, I could find none in existence. So I decided to make my own – and make it available to the virtualization community. vCalendar is a collection of VMware virtualization facts, tips, best practices, configuration maximums, and historical events delivered to your desktop daily. vCalendar was authored by virtualization virtuoso and Senior Systems Engineer Jason Boche.

“This year’s must-have item at VMworld”

-John Troyer, VMware

“Anyone who is in to virtualization HAS to get one of these calendars! They are limited, so they are the ‘must have’ item from VMworld this year! I’ll be going to the Veeam booth to get one for sure!”

-Susan Gudenkauf, VMware, VCP#1

I would like to thank all of the contributors in the virtualization community who share their knowledge to assist others. I’d like to thank VMware & their Partners for their products which keep me entertained & employed. I would like to thank my Technical Editors Duncan Epping, Eric Siebert, & Tom Howarth. I would like to thank my family Amy, Allison, Joshua, Snickers, & Duncan for their patience.

The vCalendar is being made available a few ways:

  1. vCalendar is available for purchase online at The Printed Owl. Select your starting month and year – vCalendars are custom printed to your specification and shipped to your doorstep. Sorry, international shipping is not yet available but I’m working on options.
  2. vCalendar has already arrived in San Francisco and will be available in limited quantities at the VMworld 2009 store.
  3. A special limited edition vCalendar will be available from the Veeam booth at VMworld 2009.
  4. Options are being discussed to run a batch of vCalendars to the upcoming Dutch VMUG.
  5. A vCalendar widget will be voluntarily carried on select virtualization web sites with rotating daily tips. If you are a blogger or if you have a virtualization related web site and you’d like to carry the widget on your blog, contact me.
  6. PowerCLI God, Alan Renouf, has mastered a way to bring vCalendar tips into vCenter as a MOTD.  This is really cool! Check out vTip – A VMware Expert updating your VI