With the release of vSphere 4.1, VMware has introduced Multicore Virtual CPU technology to its bare metal flagship hypervisor.  This is an interesting feature which had already existed in current versions of VMware Workstation.  VMware has consistently baked in new features in its Type 2 hypervisor products, such as Workstation, Player, Fusion, etc., more or less as a functionality/stability test before releasing the same features in ESX(i).  VMware highlights this new feature as follows:

User-configurable Number of Virtual CPUs per Virtual Socket: You can configure virtual machines to have multiple virtual CPUs reside in a single virtual socket, with each virtual CPU appearing to the guest operating system as a single core. Previously, virtual machines were restricted to having only one virtual CPU per virtual socket. See the vSphere Virtual Machine Administration Guide.

VMware multicore virtual CPU support lets you control the number of cores per virtual CPU in a virtual machine. This capability lets operating systems with socket restrictions use more of the host CPU’s cores, which increases overall performance.

Using multicore virtual CPUs can be useful when you run operating systems or applications that can take advantage of only a limited number of CPU sockets. Previously, each virtual CPU was, by default, assigned to a single-core socket, so that the virtual machine would have as many sockets as virtual CPUs.

You can configure how the virtual CPUs are assigned in terms of sockets and cores. For example, you can configure a virtual machine with four virtual CPUs in the following ways:

• Four sockets with one core per socket (legacy, this is how we’ve always done it prior to vSphere 4.1)
• Two sockets with two cores per socket (new in vSphere 4.1)
• One socket with four cores per socket (new in vSphere 4.1)

VMware defines a CPU as:

The portion of a computer system that carries out the instructions of a computer program and is the primary element carrying out the computer’s functions.

VMware defines a Core as:

A logical execution unit containing an L1 cache and functional units needed to execute programs. Cores can independently execute programs or threads.

VMware defines a Socket as:

A physical connector on a computer motherboard that accepts a single physical chip. Many motherboards can have multiple sockets that can in turn accept multicore chips.

One of the benefits of multicore which physical computing had was increased density of the hardware.  VMs do not share this advantage as they are virtual to begin with and have no rack footprint to speak of.

VMware’s benefit statement for this feature is a legitimate one and is the primary use case.  It’s the same benefit which applied when multicore (as well as hyperthreading to some extent) technology was introduced to physical servers.  What VMware doesn’t advertise is that the limitation being discussed usually revolves around software licensing - a per-socket license model to be precise which is what many software vendors still use.  For example, if I own a piece of software and I have a single socket license, traditionally I was only able to use this software inside of a single vCPU VM.  With Multicore Virtual CPUs, Virtual Machines have now caught up with their physcial hardware counterparts in that a single socket VM can be created which has 4 cores per socket.  Using the working example, the advantage I have now is that I can run my application inside a VM which still has 1 socket, but 4 cores for a net result of 4 vCPUs instead of just 1 vCPU.  I didn’t have to pay my software vendor additional money for the added CPU power.  To show how this translates into dollars and cents, let’s assume a per socket license cost of my application to be $1,000 and then extrapolate those numbers using VMware’s example above of how CPUs can be assigned in terms of sockets and cores:

Now, all of this said, the responsibility is on the end user to be in license compliance with his or her software vendors.  Just becasue you can do this doens’t mean you’re legally obliged to do so.  Be sure to read your EULA and check with your software vendor or reseller before implementing VMware Multicore Virtual CPUs.

Implementation of Multicore Virtual CPUs was quite straightfoward in VMware Workstation.  Upon creating a new VM or editing an existing VM’s settings, the following interface was presented for configuring vCPUs and cores per vCPU in VMware Workstation.  In this example, a 2xDC (Dual Core) configuration is being applied which results in a total of 4 CPU cores which will serve the VM’s operating system, applications, and users. Note that here, the term “processors” on the first line translates to “sockets”:

Making the same 2xDC CPU configuration in vSphere 4.1 isn’t difficult but nonetheless it is done differently.  Configuring total vCPUs and cores per vCPU is achieved by applying configurations in two different areas of the VM configuration. The combination of the two configurations produces a mathematical calculation which ultimately determines cores per vCPU.

First of all, the total number of cores (processors) is selected in the VM’s CPU configuration.  This hasn’t changed and should be familiar to you.  The number of cores (processors) available for selection here is going to be 1 thru 4 or 1 thru 8 if you have Enterprise Plus licensing.  I’ve purposely included the notation of the VM hardware version 7 which is required. An inconsistency here compared to VMware Workstation is that the term “virtual processors” translates to “cores”, not “sockets”:

Configuring the number of cores per processor is where VMware has deviated from the VMware Workstation implementation.  In ESX and ESXi, this configuration is made as an advanced setting in the .vmx file.  Edit the VM settings, navigate to the Options tab, choose General in the Advanced options list. Click the Configuration Parameters button which allows you to edit the .vmx file on a row by row basis.  Click the Add Row button and add the line item cpuid.coresPerSocket. For the value, your going to supply the number of cores per processor which is generally going to be a value of 2, 4, or 8 (Enterprise Plus licensing required).  Note, using a value of 1 here would serve no practical purpose because it would configure a single core vCPU which is what we’ve had all along up until this point:

As a supplement, here are the requirements for implementing Multicore Virtual CPUs:

• VMware vSphere 4.1 (vCenter 4.1, ESX 4.1 or ESXi 4.1).
• Virtual Machine hardware version 7 is required.
• The VM must be powered off to configure Multicore Virtual CPUs.
• The total number of vCPUs for the VM divided by the number of cores per socket must be a positive integer.
• The cpuid.coresPerSocket value must be a power of 2. The documentation explicitely states a value of 2, 4, or 8 is required, but 1 works as well although as stated before it would serve no practical purpose.
  • 2^0=1 (anything to the power of 0 always equals 1)
  • 2^1=2 (anything to the power of 1 always equals itself)
  • 2^2=4
  • 2^4=8
• When you configure multicore virtual CPUs for a virtual machine, CPU hot Add/Remove is disabled (previously called CPU hot plug).
• You must be in compliance with the requirements of the operating system EULA.

This feature rocks and I think customers have been waiting a long time for it.  Duncan mentioned it quite some time ago but obvioulsy it was unsupported at that time.  I am a little puzzled by the implementation mechanisms, mainly the configuration of the .vmx to specify cores per CPU.  I suppose it lends itself to scriptability and thus automation, but in that sense, we lack the flexibility to configure cores per CPU with guest customization when deploying VMs from a template.  Essentially this means cores per CPU needs to be hard coded in each of my templates or cores per CPU needs to be manually tuned after deploying each VM from a template.  When I take a step back, I guess that’s no different than any other virtual hardware configuration stored in templates, but with the cores per CPU setting being buried in the .vmx as an advanced setting, it’s that much more of a manal/administrative burden to configure cores per CPU for each VM deployed than it is to simply change the number of CPUs or amount of RAM.  It would be nice if the guest customization process offered a quick way to configure cores per processor.

To troubleshoot a vSphere cluster showing Noncompliant on the Profile Compliance tab, check the following:

FT logging NIC speed is at least 1000 Mbps
At least one shared datastore exists
FT logging is enabled
VMotion NIC speed is at least 1000 Mbps
All the hosts in the cluster have the same build for Fault Tolerance
The host hardware supports Fault Tolerance
VMotion is enabled

Read more at: http://kb.vmware.com/kb/1017471

I few weeks ago, I came across a networking issue with VMware ESX(i) 4.0 Update 1.  The issue is that configuring a vSwitch or Portgroup for Failback: No doesn’t work as expected in conjunction with a Network Failover Detection type of Link Status Only. 

For the simplest of examples:

1. Configure a vSwitch with 2 VMNIC uplinks with both NICs in an Active configuration.  I’ll refer to the uplinks as VMNIC0 and VMNIC1.
2. Configure the vSwitch and/or a Portgroup on the vSwitch for Failback: No.
3. Create a few test VMs with outbound TCP/IP through the vSwitch. 
4. Power on the VMs and begin a constant ping to each of the VMs from the network on the far side of the vSwitch.
5. Pull the network cable from VMNIC0.  You should see little to no network connectivity loss on the constant pings.
6. With VMNIC0 failed, at this point, all network traffic is riding over VMNIC1.  When VMNIC0 is recovered, the expected behavior with No Failback is that all traffic will continue to traverse VMNIC1.
7. Now provide VMNIC0 with a link signal by connecting it to a switch port which has no route to the physical network.  For example, simply connect VMNIC0 to a portable Netgear or Linksys switch.
8. What you should see now is that at least one of the VMs is unpingable.  It has lost network connectivity because ESX has actually failed its network path back to VMNIC0.  In the failback mechanism, VMware appears to balance the traffic evenly.  In a 2 VM test, 1 VM will fail back to the recovered VMNIC.  In a 4 VM test, 2 VMs will fail back to the recovered VMNIC.  Etc.

 The impact spreads to any traffic being supported by the vSwitch, not just VM networking.  Thus, the impact includes Service Console, Management Network, IP based storage, VMotion, and FT.  Scope of the bug includes both the standard vSwitch as well as the vSphere Distributed Switch (vDS).

Based on the following VMTN forum thread, it would appear this bug has existed since August of 2008.  Unfortunately, documentation of the bug never made it to VMware support:

http://communities.vmware.com/thread/165302

You may be asking yourself at this point, well who really cares?  At the very least, we have on our hands a feature which does not work as documented on page 41 of the following manual: http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_esxi_server_config.pdf.  Organizations which have made a design decision for no failback have done so for a reason and rely on the feature to work as it is documented.

Why would my VMNIC ever come up with no routing capabilities to the physical network?  Granted, my test was simplistic in nature and doesn’t likely represent an actual datacenter design but the purpose was merely to point out to folks that the problem does exist.  The issue actually does present a real world problem for at least one environment I’ve seen.  Consider an HP C-class blade chassis fitted with redundant 10GbE Flex10 Ethernet modules.  Upstream to the Flex10 modules are Cisco Nexus 5k switches and then Nexus 7k switches.

When a Flex10 module fails and is recovered (say for instance it was rebooted – which you can test yourself if you have one), it has an unfortunate habit of bringing up the blade facing network ports (in this case, VMNIC0 labeled 1 in the diagram) up to 20 seconds before a link is established with the upstream Cisco Nexus 5k (labeled 2 in the diagram) which grants network routing to other infrastructure components.  So what happens here?  VMNIC0 shows a link and ESX fails back traffic to it up to 20 seconds before the link to the Nexus 5k is established.  There is a network outage for Service Console, Management Network, IP based storage, VMotion, and FT.  Perhaps some may say they can tolerate this much of an outage for their VM traffic, but most people I have talked to say even an outage of 2 or more seconds is unacceptable.  And what about IP based storage?  Can you afford the 20 second latency?  What about Management Network and Service Console?  Think HA and isolation response impact.  VMs shutting down as a result.  Etc.  It’s a nasty chain of events.  In such a case, a decision can be made to enable no failback as a policy on the vSwith and Portgroups.  However, due to the VMware bug, this doesn’t work.  Some day you may experience an outage which you did not expect.

As pointed out in the VMTN forum thread above, there is a workaround which I have tested and does work: Force at least one VMNIC to act as Standby.  This is not by VMware design, it just happens to make the no failback behavior work correctly.  The impact with this design decision is of course that now one VMNIC stands idle and there are no load balancing opportunities over this VMNIC.  In addition, with no failback enabled, network traffic will tend to become polarized to one side again impacting network load balancing.

An SR has been opened with VMware on this issue.  They have confirmed it is a bug and will be working to resolve the issue in a future patch or release.

Update 4/27/10:  The no failback issue has been root-caused by VMware and a fix has been constructed and is being tested. It will be triaged for a future release.