Here’s something interesting to watch out for if you’re running ESX 3.5 Update 1 or newer clusters. In particular, clusters densely populated with running VMs or VMs with 2-way or 4-way vSMP.
Prior to ESX 3.5 Update 1, the supported and configured maximum number of vCPUs on a host was 128 by default. This meant that VMs totaling up to 128 vCPUs could be powered on within a single host.
With the release of ESX 3.5 Update 1, the supported maximum number of vCPUs on a host was raised to 192. This meant that VMs totaling up to 192 vCPUs could be powered on within a single host. Effectively, VMware is allowing higher consolidation ratios on a single host. However, according to KB article 1006393, ESX 3.5 Update 1 and newer hosts will still be configured to run a maximum of 128 vCPUs! Through my experience, this applies to both new installations of ESX 3.5 Update 1 and newer, as well as ESX 3.5 hosts that have been patched/remediated with Update 1 or newer.
So how does this impact a cluster? As I said in the beginning, you’ll run into problems on highly populated clusters or clusters with large numbers of VMs with vSMP CPUs enabled. You’ll see a few different but closely related scenarios:
- VMs will not VMotion onto a host which would cause it to exceed a 128 running vCPU limit
- DRS will not move running VMs onto a host which would cause it to exceed a 128 running vCPU limit
- Maintenance Mode for a host will never complete if evacuation of the running VMs would cause all other hosts in the cluster to exceed a 128 running vCPU limit
- HA will not power on VMs which would cause a host to exceed a 128 running vCPU limit
- You will not be able to power on a VM which would cause a host to exceed a 128 running vCPU limit
To configure an ESX 3.5 Update 1 or newer host to support the maximum number of running vCPUs (192), follow the instructions in the KB article above which I will repeat here:
In the VI Client, go to the Configuration tab and choose Advanced Settings.
In the Advanced Settings window, change the value for Misc.RunningVCpuLimit to 192.
The increased maximum limit takes effect immediately and is retained after rebooting the host.
Repeat the steps above for each host.
VMware made a change for the better in vSphere. ESX 4.0 supports a maximum of 512 vCPUs and this is the way the host is configured in a default installation, thus, no hidden restriction as we find in ESX 3.5 Update 1 and newer.
Update 8/19/09: VMTN community member William Lam read this article and published a Perl script which will query a specific cluster and extract out the number of vCPU for the given cluster, each individual host and the advanced configuration Misc.RunningVCpuLimit set for each host. Thanks a lot William!!
Hi Jason, for the PowerCLI fans I have also re-created Williams script in PowerCLI: http://www.virtu-al.net/2009/08/20/powercli-number-of-vcpus-in-a-cluster/
in esx 3.5 update 4 how i can change the the max value of cpu more than 4 for vm’s.
is there any paramater which i can change on server side and will solve this issue.
I just read the other day there is a hidden switch to increase the number of vCPUs in ESX 3 beyond 4. It is of course unsupported and I do not know what it is though. I would bet Duncan Epping knows but won’t tell 🙂
Hi Jason
it would be great if you can tell how to do that.
and when you said that is unsupported does that mean it not supported by VMware or cm will not support the changes made?
if its not supported by vmware then i am ok with it 🙂
I have summoned Duncan. Hopefully he will come here and reply.
To set this within a kickstart script you can use:
vmware-vim-cmd -n -e “/hostsvc/advopt/update MiscRunningVCpuLimit int 192”
While running this command :
vmware-vim-cmd -n -e “/hostsvc/advopt/update MiscRunningVCpuLimit int 192″
getting error message as
Invalid option ‘-n’
Usage: /usr/lib/vmware/hostd/vmware-vim-cmd [options]… command [cmd_arg1] [cmd _arg2] …
Options:
-h Display this help message and exit
-v Display version information and exit
-H Host name to connect
-O Port number to connect
-U User name to use for login
-P Password to use for login
-d Show verbose debug output. (info, verbose, trivia)
Use the help command to get information on the commands available.
/usr/lib/vmware/hostd/vmware-vim-cmd help [command]
Hello
i have made the canges for the said parameter using VMware Infrastructure client.
but after that when i have assigned 8 cpu to virtual machine and i try to start, getting a error as:
“Virtual machine has 8 virtual CPU’s, but host only supports 4. The number of virtual CPUs may be limited by the guest OS selected for the virtual machine or by the licensing of the host”
For reference i am using RHEL 5.2 64bit on guest machine. so there should not be an issue with OS
Apologies, it should have been :
/usr/bin/vimsh -n -e “/hostsvc/advopt/update Misc.RunningVCpuLimit int 192″
This sets the maximim running vCPUs to 192, from 128.
I’ve no idea on incresing max vCPUs per VM from 8.