Hidden Virtual CPU Limit Restriction in ESX 3.5

August 18th, 2009 by jason Leave a reply »

Here’s something interesting to watch out for if you’re running ESX 3.5 Update 1 or newer clusters. In particular, clusters densely populated with running VMs or VMs with 2-way or 4-way vSMP.

Prior to ESX 3.5 Update 1, the supported and configured maximum number of vCPUs on a host was 128 by default. This meant that VMs totaling up to 128 vCPUs could be powered on within a single host.

With the release of ESX 3.5 Update 1, the supported maximum number of vCPUs on a host was raised to 192. This meant that VMs totaling up to 192 vCPUs could be powered on within a single host. Effectively, VMware is allowing higher consolidation ratios on a single host. However, according to KB article 1006393, ESX 3.5 Update 1 and newer hosts will still be configured to run a maximum of 128 vCPUs! Through my experience, this applies to both new installations of ESX 3.5 Update 1 and newer, as well as ESX 3.5 hosts that have been patched/remediated with Update 1 or newer.

So how does this impact a cluster? As I said in the beginning, you’ll run into problems on highly populated clusters or clusters with large numbers of VMs with vSMP CPUs enabled. You’ll see a few different but closely related scenarios:

  1. VMs will not VMotion onto a host which would cause it to exceed a 128 running vCPU limit
  2. DRS will not move running VMs onto a host which would cause it to exceed a 128 running vCPU limit
  3. Maintenance Mode for a host will never complete if evacuation of the running VMs would cause all other hosts in the cluster to exceed a 128 running vCPU limit
  4. HA will not power on VMs which would cause a host to exceed a 128 running vCPU limit
  5. You will not be able to power on a VM which would cause a host to exceed a 128 running vCPU limit

To configure an ESX 3.5 Update 1 or newer host to support the maximum number of running vCPUs (192), follow the instructions in the KB article above which I will repeat here:

In the VI Client, go to the Configuration tab and choose Advanced Settings.

In the Advanced Settings window, change the value for Misc.RunningVCpuLimit to 192.

The increased maximum limit takes effect immediately and is retained after rebooting the host.

Repeat the steps above for each host.

VMware made a change for the better in vSphere. ESX 4.0 supports a maximum of 512 vCPUs and this is the way the host is configured in a default installation, thus, no hidden restriction as we find in ESX 3.5 Update 1 and newer.

Update 8/19/09: VMTN community member William Lam read this article and published a Perl script which will query a specific cluster and extract out the number of vCPU for the given cluster, each individual host and the advanced configuration Misc.RunningVCpuLimit set for each host. Thanks a lot William!!

Advertisement

10 comments

  1. Alan Renouf says:

    Hi Jason, for the PowerCLI fans I have also re-created Williams script in PowerCLI: http://www.virtu-al.net/2009/08/20/powercli-number-of-vcpus-in-a-cluster/

  2. Arpan says:

    in esx 3.5 update 4 how i can change the the max value of cpu more than 4 for vm’s.
    is there any paramater which i can change on server side and will solve this issue.

  3. jason says:

    I just read the other day there is a hidden switch to increase the number of vCPUs in ESX 3 beyond 4. It is of course unsupported and I do not know what it is though. I would bet Duncan Epping knows but won’t tell 🙂

  4. Arpan says:

    Hi Jason
    it would be great if you can tell how to do that.
    and when you said that is unsupported does that mean it not supported by VMware or cm will not support the changes made?

    if its not supported by vmware then i am ok with it 🙂

  5. jason says:

    I have summoned Duncan. Hopefully he will come here and reply.

  6. jcolcombe says:

    To set this within a kickstart script you can use:

    vmware-vim-cmd -n -e “/hostsvc/advopt/update MiscRunningVCpuLimit int 192”

  7. Arpan says:

    While running this command :

    vmware-vim-cmd -n -e “/hostsvc/advopt/update MiscRunningVCpuLimit int 192″

    getting error message as
    Invalid option ‘-n’
    Usage: /usr/lib/vmware/hostd/vmware-vim-cmd [options]… command [cmd_arg1] [cmd _arg2] …
    Options:
    -h Display this help message and exit
    -v Display version information and exit
    -H Host name to connect
    -O Port number to connect
    -U User name to use for login
    -P Password to use for login
    -d Show verbose debug output. (info, verbose, trivia)

    Use the help command to get information on the commands available.

    /usr/lib/vmware/hostd/vmware-vim-cmd help [command]

  8. Arpan says:

    Hello
    i have made the canges for the said parameter using VMware Infrastructure client.
    but after that when i have assigned 8 cpu to virtual machine and i try to start, getting a error as:

    “Virtual machine has 8 virtual CPU’s, but host only supports 4. The number of virtual CPUs may be limited by the guest OS selected for the virtual machine or by the licensing of the host”

    For reference i am using RHEL 5.2 64bit on guest machine. so there should not be an issue with OS

  9. jcolcombe says:

    Apologies, it should have been :
    /usr/bin/vimsh -n -e “/hostsvc/advopt/update Misc.RunningVCpuLimit int 192″

    This sets the maximim running vCPUs to 192, from 128.
    I’ve no idea on incresing max vCPUs per VM from 8.

Leave a Reply