8 New ESX 3.5 Patches Released; 4 Critical

September 1st, 2009 by jason No comments »

Four new patches have been released for ESX 3.5.0. It appears ESXi, ESX4, and other versions of ESX are not impacted.

4 of the 8 patches are rated critical and should be evaluated quickly for application in your virtual infrastructure.

ID: ESX350-200908401-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates forcedeth driver
The forcedeth driver installed on the ESX hosts causes the NVIDIA nForce Network Controller NICs to lose network connectivity until the forcedeth driver is reloaded. This patch addresses the issue.

The affected NICS are:

  • NVIDIA nForce Professional 2200 MCP 1Gbe NIC
  • NVIDIA nForce Professional 2050 I/O companion chip 1Gbe NIC
  • NVIDIA nForce Professional 3600 1Gbe NIC

ID: ESX350-200908402-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates VMware Tools
After performing VMotion between ESX 3.0.x and ESX 3.5 hosts, virtual machines running on ESX 3.5 hosts are restarted in order to upgrade to the latest version of VMware Tools. After applying this fix, VMware Tools function as expected.

ID: ESX350-200908403-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates megaraid and mptscsi drivers
This patch fixes the following issues:

  • When the ESX host boots, the megaraid_sas driver heap gets depleted when claiming 4 LSI SAS RAID controllers on IBM System x3950 M2 Athena servers. This issue might cause the ESX host to stop booting. The fix increases the heap size for the megaraid_sas driver from 8 MB to 16 MB.
  • The mptscsi_2xx driver limits the discovery of targets to 63 SAS devices per LSI Serial Attached SCSI (SAS) host bus adapter (HBA). This fix increases the number of targets to the value returned by the HBA firmware.

ID: ESX350-200908404-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates vmkctl
When N-Port ID Virtualization (NPIV) enabled virtual machines are powered on on ESX hosts, a rescan issued from the VI Client results in an error message stating that the rescan failed, even if the rescan is successful.

ID: ESX350-200908405-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates vmkernel
Running the esxtop command on the service console of the ESX hosts lists high values for the max limited (%MLMTD) parameter for virtual machines when no max limited parameter is set. When the high values are listed, the performance of the virtual machines might be affected. In the VI Client, the max limited parameter is set in the Resources tab for CPU in Virtual Machine properties.

ID: ESX350-200908406-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates vmx
This patch provides the following:

  • Adds support for new SCSI-3 status values in the SCSI emulation for virtual machines.
  • Fixes an issue where powering on customized versions of Ubuntu virtual machines from the ESX hosts might cause the ESX hosts to stop responding.

ID: ESX350-200908407-BG Impact: HostGeneral Release date: 2009-08-31 Products: esx 3.5.0 Updates kernel source and vmnix
This patch updates the service console kernel for the following fixes:

  • The forcedeth driver installed on the ESX hosts causes the NVIDIA nForce Network Controller NICs to lose network connectivity under certain circumstances. The affected NICS are:
    • NVIDIA nForce Professional 2200 MCP 1Gbe NIC
    • NVIDIA nForce Professional 2050 I/O companion chip 1Gbe NIC
    • NVIDIA nForce Professional 3600 1Gbe NIC
  • A bnx2x firmware dump issue.
  • The mptscsi_2xx driver limits the discovery of targets to 63 SAS devices per LSI Serial Attached SCSI (SAS) host bus adapter (HBA). This fix increases the number of targets to the value returned by the HBA firmware.

ID: ESX350-200908408-BG Impact: Critical Release date: 2009-08-31 Products: esx 3.5.0 Updates bnx2x driver
This patch fixes a bnx2x firmware dump issue.

vCalendar Launch

August 19th, 2009 by jason No comments »

vCalendar
Welcome! vCalendar is little something I have been working on in my spare time during the spring and summer months of 2009. I’ve had “Far Side” and “Dilbert” Page-A-Day calendars on my desktop at work for the past 15 years and towards the end of 2008 I thought a virtualization calendar would be nice to have, however, I could find none in existence. So I decided to make my own – and make it available to the virtualization community. vCalendar is a collection of VMware virtualization facts, tips, best practices, configuration maximums, and historical events delivered to your desktop daily. vCalendar was authored by virtualization virtuoso and Senior Systems Engineer Jason Boche.

“This year’s must-have item at VMworld”

-John Troyer, VMware

“Anyone who is in to virtualization HAS to get one of these calendars! They are limited, so they are the ‘must have’ item from VMworld this year! I’ll be going to the Veeam booth to get one for sure!”

-Susan Gudenkauf, VMware, VCP#1

Acknowledgements
I would like to thank all of the contributors in the virtualization community who share their knowledge to assist others. I’d like to thank VMware & their Partners for their products which keep me entertained & employed. I would like to thank my Technical Editors Duncan Epping, Eric Siebert, & Tom Howarth. I would like to thank my family Amy, Allison, Joshua, Snickers, & Duncan for their patience.

Availability
The vCalendar is being made available a few ways:

  1. vCalendar is available for purchase online at The Printed Owl. Select your starting month and year – vCalendars are custom printed to your specification and shipped to your doorstep. Sorry, international shipping is not yet available but I’m working on options.
  2. vCalendar has already arrived in San Francisco and will be available in limited quantities at the VMworld 2009 store.
  3. A special limited edition vCalendar will be available from the Veeam booth at VMworld 2009.
  4. Options are being discussed to run a batch of vCalendars to the upcoming Dutch VMUG.
  5. A vCalendar widget will be voluntarily carried on select virtualization web sites with rotating daily tips. If you are a blogger or if you have a virtualization related web site and you’d like to carry the widget on your blog, contact me.
  6. PowerCLI God, Alan Renouf, has mastered a way to bring vCalendar tips into vCenter as a MOTD.  This is really cool! Check out vTip – A VMware Expert updating your VI

Hidden Virtual CPU Limit Restriction in ESX 3.5

August 18th, 2009 by jason 10 comments »

Here’s something interesting to watch out for if you’re running ESX 3.5 Update 1 or newer clusters. In particular, clusters densely populated with running VMs or VMs with 2-way or 4-way vSMP.

Prior to ESX 3.5 Update 1, the supported and configured maximum number of vCPUs on a host was 128 by default. This meant that VMs totaling up to 128 vCPUs could be powered on within a single host.

With the release of ESX 3.5 Update 1, the supported maximum number of vCPUs on a host was raised to 192. This meant that VMs totaling up to 192 vCPUs could be powered on within a single host. Effectively, VMware is allowing higher consolidation ratios on a single host. However, according to KB article 1006393, ESX 3.5 Update 1 and newer hosts will still be configured to run a maximum of 128 vCPUs! Through my experience, this applies to both new installations of ESX 3.5 Update 1 and newer, as well as ESX 3.5 hosts that have been patched/remediated with Update 1 or newer.

So how does this impact a cluster? As I said in the beginning, you’ll run into problems on highly populated clusters or clusters with large numbers of VMs with vSMP CPUs enabled. You’ll see a few different but closely related scenarios:

  1. VMs will not VMotion onto a host which would cause it to exceed a 128 running vCPU limit
  2. DRS will not move running VMs onto a host which would cause it to exceed a 128 running vCPU limit
  3. Maintenance Mode for a host will never complete if evacuation of the running VMs would cause all other hosts in the cluster to exceed a 128 running vCPU limit
  4. HA will not power on VMs which would cause a host to exceed a 128 running vCPU limit
  5. You will not be able to power on a VM which would cause a host to exceed a 128 running vCPU limit

To configure an ESX 3.5 Update 1 or newer host to support the maximum number of running vCPUs (192), follow the instructions in the KB article above which I will repeat here:

In the VI Client, go to the Configuration tab and choose Advanced Settings.

In the Advanced Settings window, change the value for Misc.RunningVCpuLimit to 192.

The increased maximum limit takes effect immediately and is retained after rebooting the host.

Repeat the steps above for each host.

VMware made a change for the better in vSphere. ESX 4.0 supports a maximum of 512 vCPUs and this is the way the host is configured in a default installation, thus, no hidden restriction as we find in ESX 3.5 Update 1 and newer.

Update 8/19/09: VMTN community member William Lam read this article and published a Perl script which will query a specific cluster and extract out the number of vCPU for the given cluster, each individual host and the advanced configuration Misc.RunningVCpuLimit set for each host. Thanks a lot William!!

vSphere 4 Reference Card now available

August 10th, 2009 by jason No comments »

Forbes Guthrie has done it again! His wildly successful VI3 reference card is now available in vSphere format.  Head over to his site, vReference, and download your copy today.  Be sure to thank him for his hard work! I for one appreciate all that he does. Thanks Forbes and I look forward to meeting you in a few weeks.

8-10-2009 11-05-11 AM

2nd Batch of vSphere Patches Arrive (Critical)

August 7th, 2009 by jason No comments »

The list is short, but critical, and impacts both ESX 4.0 as well as ESXi 4.0 hosts.

ESXi400-200907401-BGCritical http://kb.vmware.com/kb/1013016

This patch fixes the following issue – If the RAID controller battery backup unit is completely discharged after a shutdown, or if a locally attached disk is removed and not returned to the system, data corruption might occur because the RAID controller cache is not cleared while shutting down the server.  See KB 1013016 for more details.  NOTE:  Cisco Nexus 1000v customers using VMware Update Manager to patch ESXi 4.0 should add an additional patch download URL as described in KB 1013134

ESX400-200907401-BGCritical http://kb.vmware.com/kb/1013026

This patch fixes the following issue – If the RAID controller battery backup unit is completely discharged after a shutdown, or if a locally attached disk is removed and not returned to the system, data corruption might occur because the RAID controller cache is not cleared while shutting down the server.  See KB 1013026 for more details.  NOTE:  Cisco Nexus 1000v customers using VMware Update Manager to patch ESXi 4.0 should add an additional patch download URL as described in KB 1013134

It’s coming… vCalendar

August 6th, 2009 by jason No comments »

What could it be?

Update 8/17/09: There have been some fantastic guesses in the comments! Such efforts must be rewarded. The clarity of the image has been improved some. In addition, fellow virtualization blogger Tom Howarth has correctly (and precisely I might add) guessed a letter position via Twitter. The title of this blog post has been updated accordingly. Looking back through the comments once more, it looked like VMDoug was onto something big when he mentioned ‘NDA’. Hmmmm…

Update 8/19/09: Not much activity since the last clues were given two days ago. Did everyone lose interest or they tired of being teased? VMDoug didn’t realize at first how closely his NDA comment was tied to this project. Blog post title has been updated.

Update #2 8/19/09: Cat is out of the bag. vCalendar is launched & avail at VMworld. Congrats @RickVanover & @davikes on solving the mystery! Head on over to the vCalendar page for more details:
http://www.boche.net/blog/index.php/vcalendar/

For those keeping track, 1.5 weeks till VMworld 2009.

VCDX Enterprise Administration Exam

August 5th, 2009 by jason No comments »

In my pursuit for VCDX (VMware Certified Design Expert) certification, I sat this exam on Tuesday July 21st at 8am. I read somewhere that statistically speaking, humans are better test takers in the morning than in the afternoon. This statistic applies to me. I’ve passed every certification exam I’ve sat before noon and the only exam I’ve ever failed was Microsoft Networking Essentials in 1997 and that was an afternoon exam.

Anyway, the Enterprise Administration exam is part written, part hands on lab. Because of the lab piece, it takes VMware 10-12 business days to grade the exam. My exam was no exception. On business day 11, today, I receive the results this morning on my Blackberry while in a VMware BCS support meeting. The initial email read:

Hello Jason,

Thank you for participating in the VMware Certified Design Expert (VCDX) program. Attached you will find your score report for the Enterprise Administration Exam. Please do not hesitate to contact us if you have any questions regarding your score.

Kind Regards
The VMware Certification Team

My initial thought was this doesn’t sound good. Every other blog post I read from people who passed this exam stated they were greeted with “Congratulations” meaning they had passed. My notification basically looked to me like “Thanks for the 400 bucks, your failed score is in the attached .PDF document, contact us if you need suicide prevention”. Don’t take that literally, I’m being dramatic here. The fact is, after taking this exam, I was not at all confident I had passed. In fact, I had returned to the office that morning pretty upset with myself. It was clear to me that on the written exam, I hadn’t drilled down deep enough into some of the blueprint topics, mainly topics dealing with Red Hat Enterprise Linux as well as identifying multiple methods to pull the same information through different CLI commands. As other bloggers have pointed out, the written exam makes up 60% of the passing grade. The hands on lab portion of the exam, for me anyway, was much more familiar although I won’t say it’s a cake walk. It gets into advanced administration and troubleshooting situations which you won’t always find solutions spelled out explicitly in a book. You’re on your own in a live lab to assemble what you feel is the solution. You can get creative and 10 different candidates may solve a lab problem 10 slightly different ways. What counts is what the lab proctor finds in your resulting virtual infrastructure come grading time. Either you satisfied the outlined requirement through your processes, or you didn’t. The lab accounts for the other 40% of the passing grade.

After the first day went by, I was no longer depressed. I focused on my work and other things realizing there was nothing I could do to change my prior performance in the exam room. If I had failed, it is what it is, I accept it, and I clearly knew what areas needed more attention. $400 is a tough learning experience though, for me the cost of this endeavor adds pressure. As the days passed, it became clear VMware really was going to stick to its 10-12 business day grading policy, and I wasn’t going to receive my results early.

So I opened up the attached .PDF score report on my Blackberry. Of course, it opens slowly adding more drama and suspense. However, in the end I was pleased to see:

Congratulations on passing this exam. Passing this exam is the second of the four parts
required to achieve the VMware Certified Design Expert designation. The next
component is the VMware Design Exam. You will be sent registration information to this
exam when it is publicly released in August 2009.

No doubt I was stoked to see I had passed but my focus quickly turned to my score.  I had been thinking that in the remote possibility I had passed this exam, then it was going to be by the hair of my chin. The score report showed I wasn’t nearly as close to the pass/fail threshold as I thought I’d be. I had plenty of buffer with my score and I think I owe my passing grade plus the additional buffer to the lab portion of the exam which I feel I was perfect on. By the way, I had no time to spare on this exam. I was working on the last of the 11 lab problems 5 seconds before time ran out on my exam. I wasted significant time in deep thought during the written part of the exam and almost didn’t leave enough time to complete the labs – I really had to rush through the labs. My lab was also having technical issues which added several minutes of delay and anxiety at one point as a vCenter task sat there hung for several minutes which a successful configuration was dependent on (I don’t believe this was intended to be part of the lab experience).

If you are interested in taking this exam and you’re looking for some study material, Duncan Epping has this recent post which is pretty comprehensive although the first link to the study guide appears to need fixing and the study guide is one of the best resources in the list. Also, as Duncan points out, in hind sight I also feel the DSA course would have been incredibly beneficial in addressing this exam’s objectives.

Now I move on to the Design written exam.  I am waiting for VMware to schedule this for me.  It will probably take a few weeks for that to happen which is just fine because I’m very busy right now at work, at home, and getting things ready for VMworld (I’ve got a little surprise for the VMware community I’ve been working on).  I expect the Design exam to be a little bit more theory and high level concepts rather than detailed CLI commands and switches.  I’ve also heard from one person they felt the Design exam was more difficult than the Enterprise Administration exam.  I’ve got my work cut out for me. At no time did I expect this would be easy.