Posts Tagged ‘Documentation’

« Older Entries

VMkernel Networks, Jumbo Frames, and ESXi 4

February 12th, 2010

Question:  Can I implement jumbo frames on ESXi 4 Update 1 VMkernel networks?

Answer:  Who in the hell knows?

You see, the ESXi 4.0 Update 1 Configuration Guide states on page 54:

“Jumbo frames are not supported for VMkernel networking interfaces in ESXi.”

Duncan Epping of Yellow Bricks also reports:

“Jumbo frames are not supported for VMkernel networking interfaces in ESXi. (page 54)”

One month after the release of ESXi 4 Update 1, Charu Chaubal of VMware posted on the ESXi Chronicles blog:

“I am happy to say that this is merely an error in the documentation. In fact, ESXi 4.0 DOES support Jumbo Frames on VMkernel networking interfaces. The correction will hopefully appear in a new release of the documentation, but in the meantime, go ahead and configure Jumbo frames for your ESXi 4.0 hosts.”

Shortly after, Duncan Epping of Yellow Bricks confirms Charu Chaubal’s report that jumbo frames are supported on ESXi VMkernel networks.

Now, nearly two months after Charu’s clarification and three months after the release of ESXi 4 Update 1, the documentation remains dubious on page 54 stating that jumbo frames are not supported on ESXi 4 VMkernel networks which is a direct contradition to a VMware ESXi blog.

I opened a Business Critical Support SR with VMware on the question.  I was told by VMware BCS that jumbo frames are NOT supported on ESXi 4 Update 1 VMkernel networks and a reference was made to the documentatation on page 54. 

Our dedicated VMware onsite Engineer escalated and I was then told ESXi 4 Update 1 DOES support jumbo frames on VMkernel networks, making reference to Charu’s article.

Hey VMware, which is it?  If this is a documentation mistake, why are you dragging your feet in getting the documentation updated two months after a VMware employee discovers the error and blogs it?  Waiting for the next release of ESXi?  Unacceptable!  You update the public documentation as soon as you discover the error and be damned sure your BCS support Engineers know the right answer!  Do you know how much companies pay for BCS?  You owe your customers the correct answer.  If misinformation comes as a result of a known documentation error, SHAME ON YOU!  Architecture and design decisions are being made daily on this information or misinformation, which ever it may be.

Update 2/23/10:  Toby Kraft (@vmwarewriter on Twitter) will be updating the documentation by next week.  Thank you Toby!

Update 3/1/10:  VMware has updated their documentation to reflect currently supported configurations.  Thank you VMware (and Toby)!

5 comments »

Posted in Virtualization

Tags:

Saturday Grab Bag

September 12th, 2009

Here’s a collection of quick hits I’ve been meaning to get to. Individually, their content is a bit on the short side for the length I normally like to write so I thought I’d throw them together in a single post and see how it comes out.

Tasks and Events List Lengths

First up is the listing of Tasks and Events in the vSphere Client. Have you ever started troubleshooting an issue in the vSphere client by looking at the Tasks or Events and the chronological listing of events doesn’t go back far enough to the date or time you’re looking for? Not finding the logs you’re looking for in the vSphere Client usually means you need to open a PuTTY session and start sifting through logs in /var/log/ or /var/log/vmware/ in the Service Console. The reason for this is that the vSphere Client, by default, is configured to tail the last 100 entries in the Tasks or Events list. You can find this setting in your vSphere Client by choosing “Edit|Client Settings” then choose the “Lists” tab:

Simply increase the value from 100 to whatever you’d like, with 1,000 being the highest allowable value. Notice that when this number is increased, you will immediately see more history. In other words, you don’t have to necessarily wait for time to pass and more historical events to accumulate to see the additional rows of information. Also note that this is a vSphere Client setting which is retained client side and applies to both vCenter Server and ESX(i) host connections.

Collecting diagnostic information for VMware products

Like any offering from a software or hardware vendor, VMware products aren’t perfect. During your VMware experience, you may run into a problem which requires the intervention of VMware support. More often than not, VMware is going to ask you to generate a support bundle which consists of a collection of diagnostic and configuration files and logs. Following this paragraph is a link to VMware KB1008524 which contains links to creating support bundles for various VMware products. Note that in some cases there are different methods for different versions of the same product. If you choose to create a VMware SR online, it is helpful to have created these log bundles in advance so you can attach them to the SR. If you’ve done VMware support long enough, you already know how to FTP log bundles to VMware after an SR number has been generated.

Collecting diagnostic information for VMware products

New VMware Update Manager won’t download ESX(i) patches

Scenario: You’ve built a new VMware vCenter Server in addition to a new VMware Update Manager Server (VUM). After properly configuring Update Manager as well as the necessary internet, proxy, baseline, and scheduled task settings, VUM proceeds to download Windows, Linux, and application patches, but it won’t download ESX(i) host patches. As I found out by trench experience, the cause is because no ESX(i) hosts have been added to the vCenter Server and thus no hosts are being managed by VUM. You need to add at least one ESX(i) host to vCenter Server before VUM will be triggered to suck down all the host updates. One might then ask why guest patches are being downloaded. The only answer I have for the inconsistent behavior is due the fact that ESX(i) host patches are downloaded from VMware, while guest OS and application patches are downloaded from a completely different source, Shavlik. The mechanics behind the download processes obviously differ between the two.

What vCenter Server is this ESX(i) host managed by?

Scenario: You administer a large VMware virtual infrastructure with many vCenter Servers. You need to manage or configure a host or cluster but haven’t the slightest idea what vCenter Server to connect to. You can easily find out by attempting a Virtual Infrastructure Client connection to the host in question. Shortly after providing the necessary host credentials, the IP address of the vCenter Server managing this host will be revealed:

Now in theory, you could establish a Virtual Infrastructure Client connection to the IP address, however, I don’t like this because it dirties up the cached connection list with IP addresses which are meaningless short of having them all memorized. I prefer to take it a step further by opening a Command Prompt and using the command ping -a <IP_address> to reveal the name of the vCenter Server managing the host:

The command above reveals jarjar.boche.mcse as the vCenter Server which is managing the ESX(i) host I was wanting to manage via the vCenter Server.

I’m sure a PowerShell expert will follow up with a script which makes this process easier but this a good example to follow if you don’t have PowerShell or the VI Toolkit (Power CLI) installed.

1 comment »

Posted in Virtualization

Tags:

Hidden Virtual CPU Limit Restriction in ESX 3.5

August 18th, 2009

Here’s something interesting to watch out for if you’re running ESX 3.5 Update 1 or newer clusters. In particular, clusters densely populated with running VMs or VMs with 2-way or 4-way vSMP.

Prior to ESX 3.5 Update 1, the supported and configured maximum number of vCPUs on a host was 128 by default. This meant that VMs totaling up to 128 vCPUs could be powered on within a single host.

With the release of ESX 3.5 Update 1, the supported maximum number of vCPUs on a host was raised to 192. This meant that VMs totaling up to 192 vCPUs could be powered on within a single host. Effectively, VMware is allowing higher consolidation ratios on a single host. However, according to KB article 1006393, ESX 3.5 Update 1 and newer hosts will still be configured to run a maximum of 128 vCPUs! Through my experience, this applies to both new installations of ESX 3.5 Update 1 and newer, as well as ESX 3.5 hosts that have been patched/remediated with Update 1 or newer.

So how does this impact a cluster? As I said in the beginning, you’ll run into problems on highly populated clusters or clusters with large numbers of VMs with vSMP CPUs enabled. You’ll see a few different but closely related scenarios:

  1. VMs will not VMotion onto a host which would cause it to exceed a 128 running vCPU limit
  2. DRS will not move running VMs onto a host which would cause it to exceed a 128 running vCPU limit
  3. Maintenance Mode for a host will never complete if evacuation of the running VMs would cause all other hosts in the cluster to exceed a 128 running vCPU limit
  4. HA will not power on VMs which would cause a host to exceed a 128 running vCPU limit
  5. You will not be able to power on a VM which would cause a host to exceed a 128 running vCPU limit

To configure an ESX 3.5 Update 1 or newer host to support the maximum number of running vCPUs (192), follow the instructions in the KB article above which I will repeat here:

In the VI Client, go to the Configuration tab and choose Advanced Settings.

In the Advanced Settings window, change the value for Misc.RunningVCpuLimit to 192.

The increased maximum limit takes effect immediately and is retained after rebooting the host.

Repeat the steps above for each host.

VMware made a change for the better in vSphere. ESX 4.0 supports a maximum of 512 vCPUs and this is the way the host is configured in a default installation, thus, no hidden restriction as we find in ESX 3.5 Update 1 and newer.

Update 8/19/09: VMTN community member William Lam read this article and published a Perl script which will query a specific cluster and extract out the number of vCPU for the given cluster, each individual host and the advanced configuration Misc.RunningVCpuLimit set for each host. Thanks a lot William!!

9 comments »

Posted in Virtualization

Tags:

vSphere 4 Reference Card now available

August 10th, 2009

Forbes Guthrie has done it again! His wildly successful VI3 reference card is now available in vSphere format.  Head over to his site, vReference, and download your copy today.  Be sure to thank him for his hard work! I for one appreciate all that he does. Thanks Forbes and I look forward to meeting you in a few weeks.

8-10-2009 11-05-11 AM

1 comment »

Posted in Virtualization

Tags:

VI3 ATDG Book Full Download Available 7/19/09

July 18th, 2009

I have it on good authority that the VMware Infrastructure 3 Advanced Technical Design Guide and Advanced Operations Guide book will be made fully available for download in .PDF format tomorrow. The authors over at vmguru.com had previously been releasing two chapters at a time (one chapter in each of the two sections of the book), but a decision has been made that the next release will include the entire book.

Watch for the release at vmguru.com and grab your copy. If by chance they don’t make the Sunday release date, give them a break, these authors are among the hardest working in the business. I’m sure they’ll have it up very soon.  This is a very generous contribution to the virtualization community as the book is only about a year old.  Kudos to Scott Herold, Ron Oglesby, and Mike Laverick.

2 comments »

Posted in Virtualization

Tags:

VMware ESX Configuration Maximums Comparison Matrix

July 7th, 2009

Some of the this blog’s readers may know that one of my favorite VMware documents is the Configuration Maximums.  I’ve mentioned it a few times.  Sid Smith over at Daily Hypervisor put together a very nice consolidation of configuration maximums for ESX3, ESX3.5, and vSphere (ESX4.0).

The .PDF is assembled in a matrix format making comparison of the various VMware hypervisors easy to differentiate, compare, and contrast.  Sid, you read my mind.  Well done.  Thank you for the early birthday/Christmas present!

2 comments »

Posted in Virtualization

Tags:

VMware vSphere Cheat Sheet

April 22nd, 2009

I’m not exactly sure where this document came from – I was unable to locate it on the internet, but it looks to have been generated by VMware for the partner or sales channels. I’m an end user so I typically don’t have my hands on sales material. The document summarizes vSphere features, licensing, tiers, and more. It’s not marked VMware company confidential so I’m going to go ahead and post it.  Hopefully I won’t find myself begging for forgiveness.

I love the virtualization product comparisons. There’s a lot of smoke in the air coming from all three major virtualization camps. I think the product comparison charts really help answer the questions “Why VMware?” “Why not MS or Citrix?” “Is VMware’s price point worth it?” You bet it is. The data below speaks for itself.

4-22-2009 9-25-40 PM

4-22-2009 9-26-09 PM

4-22-2009 9-26-41 PM

8 comments »

Posted in Virtualization

Tags:

VMware documentation library updates

April 2nd, 2009

Quick note:  In case you missed it (like I did), VMware has updated most of their VMware Infrastructure 3 documentation.  If you’re a documentation junkie (like me), you’ll want to re-download all of VMware’s VI3 documentation.  About 75% of the documents have new file names as well.

http://www.vmware.com/support/pubs/vi_pages/vi_pubs_35u2.html

1 comment »

Posted in Virtualization

Tags:

Setup for Microsoft cluster service

April 1st, 2009

Setting up a Microsoft cluster on VMware used to be a fairly straight forward task with a very minimal set of considerations. Over time, the support documentation has evolved into something that looks like it was written by the U.S Internal Revenue Service. I was an Accountant in my previous life and I remember Alternative Minimum Tax code that was easier to follow than what we have today, a 50 page .PDF representing VMware’s requirements for MSCS support. Even with that, I’m not sure Microsoft supports MSCS on VMware. The Microsoft SVVP program supports explicit versions and configurations of Windows 2000/2003/2008 on ESX 3.5 update 2 and 3, and ESXi 3.5 update 3 but no mention is made regarding clustering. I could not find a definitive answer on the Microsoft SVVP program site other than the following disclaimer:

For more information about Microsoft’s policies for supporting software running in non-Microsoft hardware virtualization software please refer to http://support.microsoft.com/?kbid=897615. In addition, refer to http://support.microsoft.com/kb/957006/ to find more information about Microsoft’s support policies for its applications running in virtual environments.

At any rate, here are some highlights of MSCS setup on VMware Virtual Infrastructure, and by the way, all of this information is fair game for the VMware VCP exam.

Prerequisites for Cluster in a Box

To set up a cluster in a box, you must have:

* ESX Server host, one of the following:

* ESX Server 3 - An ESX Server host with a physical network adapter for the

service console. If the clustered virtual machines need to connect with external

hosts, then an additional network adapter is highly recommended.

* ESX Server 3i - An ESX Server host with a physical network adapter for the

VMkernel. If the clustered virtual machines need to connect with external

hosts, a separate network adapter is recommended.

* A local SCSI controller. If you plan to use a VMFS volume that exists on a SAN, you

need an FC HBA (QLogic or Emulex).

You can set up shared storage for a cluster in a box either by using a virtual disk or by

using a remote raw device mapping (RDM) LUN in virtual compatibility mode

(non‐pass‐through RDM).

When you set up the virtual machine, you need to configure:

* Two virtual network adapters.

* A hard disk that is shared between the two virtual machines (quorum disk).

* Optionally, additional hard disks for data that are shared between the two virtual

machines if your setup requires it. When you create hard disks, as described in this

document, the system creates the associated virtual SCSI controllers.

Prerequisites for Clustering Across Boxes

The prerequisites for clustering across boxes are similar to those for cluster in a box.

You must have:

* ESX Server host. VMware recommends three network adapters per host for public

network connections. The minimum configuration is:

* ESX Server 3 - An ESX Server host configured with at least two physical

network adapters dedicated to the cluster, one for the public and one for the

private network, and one network adapter dedicated to the service console.

* ESX Server 3i - An ESX Server host configured with at least two physical

network adapters dedicated to the cluster, one for the public and one for the

private network, and one network adapter dedicated to the VMkernel.

* Shared storage must be on an FC SAN.

* You must use an RDM in physical or virtual compatibility mode (pass‐through

RDM or non‐pass‐through RDM). You cannot use virtual disks for shared storage.

Prerequisites for Standby Host Clustering

The prerequisites for standby host clustering are similar to those for clustering across

boxes. You must have:

* ESX Server host. VMware recommends three network adapters per host for public

network connections. The minimum configuration is:

* ESX Server 3 - An ESX Server host configured with at least two physical

network adapters dedicated to the cluster, one for the public and one for the

private network, and one network adapter dedicated to the service console.

* ESX Server 3i - An ESX Server host configured with at least two physical

network adapters dedicated to the cluster, one for the public and one for the

private network, and one network adapter dedicated to the VMkernel.

* You must use RDMs in physical compatibility mode (pass‐through RDM).

You cannot use virtual disk or RDM in virtual compatibility mode

(non‐pass‐through RDM) for shared storage.

* You cannot have multiple paths from the ESX Server host to the storage.

* Running third‐party multipathing software is not supported. Because of this

limitation, VMware strongly recommends that there only be a single physical path

from the native Windows host to the storage array in a configuration of

standby‐host clustering with a native Windows host. The ESX Server host

automatically uses native ESX Server multipathing, which can result in multiple

paths to shared storage.

* Use the STORport Miniport driver for the FC HBA (QLogic or Emulex) in the

physical Windows machine.

Cluster in a Box Cluster Across Boxes Standby Host Clustering
Virtual disks Yes No No
Pass-through RDM (physical compatibility mode) No Yes Yes
Non-pass-through RDM (virtual compatibility mode) Yes Yes No

Caveats, Restrictions, and Recommendations

This section summarizes caveats, restrictions, and recommendation for using MSCS in

a VMware Infrastructure environment.

* VMware only supports third‐party cluster software that is specifically listed as

supported in the hardware compatibility guides. For latest updates to VMware

support for Microsoft operating system versions for MSCS, or for any other

hardware‐specific support information, see the Storage/SAN Compatibility Guide for

ESX Server 3.5 and ESX Server 3i.

* Each virtual machine has five PCI slots available by default. A cluster uses four of

these slots (two network adapters and two SCSI host bus adapters), leaving one

PCI slot for a third network adapter (or other device), if needed.

* VMware virtual machines currently emulate only SCSI‐2 reservations and do not

support applications using SCSI‐3 persistent reservations.

* Use LSILogic virtual SCSI adapter.

* Use Windows Server 2003 SP2 (32 bit or 64 bit) or Windows 2000 Server SP4.

VMware recommends Windows Server 2003.

* Use two‐node clustering.

* Clustering is not supported on iSCSI or NFS disks.

* NIC teaming is not supported with clustering.

* The boot disk of the ESX Server host should be on local storage.

* Mixed HBA environments (QLogic and Emulex) on the same host are not

supported.

* Mixed environments using both ESX Server 2.5 and ESX Server 3.x are not

supported.

* Clustered virtual machines cannot be part of VMware clusters (DRS or HA).

* You cannot use migration with VMotion on virtual machines that run cluster

software.

* Set the I/O time‐out to 60 seconds or more by modifying

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Disk\

TimeOutValue.

The system might reset this I/O time‐out value if you recreate a cluster. You must

reset the value in that case.

* Use the eagerzeroedthick format when you create disks for clustered virtual

machines. By default, the VI Client or vmkfstools create disks in zeroedthick

format. You can convert a disk to eagerzeroedthick format by importing,

cloning, or inflating the disk. Disks deployed from a template are also in

eagerzeroedthick format.

* Add disks before networking, as explained in the VMware Knowledge Base article

at http://kb.vmware.com/kb/1513.

phew!

1 comment »

Posted in Virtualization

Tags:

VMware raises the bar on CPU consolidation ratio support

April 1st, 2009

VMware has updated its Configuration Maximums support document (one of my favorite most documents in the VMware document library). Most notable is the increase in number of supported virtual CPUs per core:

  • Previously, ESX and ESXi 3.5 Update 2 and earlier supported 8 virtual CPUs per core and in special cases, 11 virtual CPUs per core if the workloads were VDI
  • The new version of the document shows ESX and ESXi 3.5 Update 3 and later support 20 virtual CPUs per core across the board – with no special circumstances for VDI workloads

One thing to note however is the fact that the number of total virtual CPUs per host or total number of virtual machines per host did not change. They remain at 192 and 170 respectively.

So we’re not increasing the total number of VMs an ESX or ESXi host will support. VMware is saying they can support the same number of VMs and vCPUs on less physical CPU cores. This may be due to more powerful CPUs entering the market (such as the Intel Nehalem). Or maybe VMware is addressing customers who have traditionally light CPU workloads and need to reach higher CPU consolidation ratios. Or maybe it has something to do with blade servers or Cisco’s UCS (or Project California). At any rate, VMware is encouraging the virtualization of more with less. Maybe it’s an economy thing. Who knows. It’s good for us though since VMware still licenses by the socket and not the core. We can power 160 VMs with an 8 core box (dual quads or quad duals).

While we’re on the subject, is anyone coming close to 170 VMs per host? What’s the most impressive consolidation ratio you’ve seen? I’d like to hear about it. As in the Citrix world, I don’t think it’s a matter of “Do we have the hardware today to handle it?” – the answer is yes. It’s more the exposure of 170 VMs on a single host and do we want to go down that road.

4 comments »

Posted in Virtualization

Tags:

VI Toolkit Quick Reference Guide

March 14th, 2009

Virtu-Al (Alan Renouf) has posted a great two-page cheat sheet for the VMware VI Toolkit version 1.5.

This gem of a document is similar to VI3 card created by Forbes Guthrie over at vReference.com. Excellent job gentlemen!

While you’re at Virtu-Al’s site, check out all the sample code and scripts.  Chances are you could implement one or more of these puppies in your environment to configure ESX or ESXi.  Scripting is definitely one of the ways to become more efficient and agile and it’s a great way to ensure consistency across your environment.  PowerShell and VI Toolkit is where’s it at.  I think they are going to be here for a long time.

2 comments »

Posted in Virtualization

Tags:

Microsoft Performance Monitor tweaks

February 17th, 2009

Today I discovered the workarounds to a few issues in Microsoft Performance Monitor that have bugged me for quite a while (read: years).

Issue 1: Vertical lines are displayed in the Sysmon tool that obscure the graph view

2-17-2009 9-41-08 PM

Cause: This behavior occurs when there are more than 100 data points to be displayed in chart view.

Resolution: Microsoft KB article 283110

To enable or disable this behavior:

  1. Start Regedit.exe.
  2. Navigate to the following key:
  3. HKEY_CURRENT_USER\Software\Microsoft\SystemMonitor
  4. On the Edit menu, click New, and then click DWord Value.
  5. Type the following value in the Name box:
  6. DisplaySingleLogSampleValue
  7. Set the value to 1 if you do not want to view the vertical line indicators, or set the value to 0, which is the default setting, to display the vertical indicators.

Result:

2-17-2009 9-47-48 PM

Issue 2: When looking at large numbers in Performance Monitor (Windows XP), comma separators do not exist thus making it difficult to interpret large numbers.

2-17-2009 9-49-26 PM

Cause: Microsoft

Resolution: Microsoft KB article 300884

Follow these steps, and then quit Registry Editor:

  1. Click Start, click Run, type regedit, and then click OK.
  2. Locate and then click the following key in the registry:
  3. HKEY_CURRENT_USER\Software\Microsoft\SystemMonitor\
  4. On the Edit menu, point to New, and then click DWORD Value.
  5. Type DisplayThousandsSeparator, and then press ENTER.
  6. On the Edit menu, click Modify.
  7. Type 1, and then click OK.

Result:

2-17-2009 9-50-51 PM

Extra credit:  Check out Microsoft KB article 281884 for one additional tweak that deals with viewing PIDs in Performance Monitor counters.

4 comments »

Posted in General

Tags:

Virtualization Wiki launched

February 11th, 2009

Rynardt Spies, proprietor of the VirtualVCP blog, has launched VI-Pedia, the Virtualization Open Wiki.

It looks like Rynardt has already begun populating the Wiki with links to VMware’s HCL information.  I think the following information which I posted over at the Petri IT Knowledgebase would also prove to be useful on the Wiki:

Community-Supported Hardware/Software for VMware Infrastructure
http://www.vmware.com/resources/communitysupport/
In 2007, VMware began maintaining a web page of non-HCL hardware that works with VMware ESX. This is a list of hardware and software components that have been reported to work with VMware Infrastructure, either by the community or by the individual vendors themselves. Great for people trying to build a cheap lab out of dubious or whitebox hardware. If your hardware is not on the official VMware HCL, check this list to see if someone has reported that your particular piece of hardware works with ESX.

Additional Resources for Community-Supported Hardware/Software for VMware Infrastructure
http://www.vm-help.com/
http://www.vm-help.com/Whitebox_HCL.php
http://ultimatewhitebox.com/
http://www.vmweekly.com/articles/hardware_recommendations_to_build_cheap_esx_server/1/
http://www.mikedipetrillo.com/mikedvirtualization/2008/10/building-a-500-vmware-esxi-host.html

Thank you for putting this together Rynardt!

1 comment »

Posted in Virtualization

Tags:

VMGURU to release 4 chapters of VI3 book today

February 10th, 2009

Scott Herold of VMGuru.com and co-author of the book VMware Infrastructure 3: Advanced Technical Design Guide and Advanced Operations Guide has announced today the release of four of the book’s chapters in PDF format today.

I’ve read the previous version of this book a few years ago and I’m in the middle of reading the current version.  I HIGHLY recommend this book.  It is worth it’s weight in gold and the fact that the authors are going to begin giving it away for free to the virtualization community is baffling to me but yet at the same time it is a symbol of their generosity and commitment to providing the community with top notch technical and operations detail on VMware virtual infrastructure.

Generally speaking, many technical authors don’t make a pile of money writing books.  Be sure to thank the authors Ron Oglesby, Scott Herold, and Mike Laverick for their hard work and generosity.

More information about this book can be found here and here.  Stay tuned to VMGuru.com for the official release of these chapters which should happen sometime today.

No comments »

Posted in Virtualization

Tags:

Three VirtualCenter security tips Windows administrators should know

January 15th, 2009

Good morning!  I’d like to take the opportunity to talk a bit about something that has been somewhat of a rock in my shoe as a seasoned Windows administrator from the NT 3.5 era:  The VirtualCenter (vCenter Server, VirtualCenter Management Server, VCMS, VC, etc.) security model, or more accurately, its unfamiliar mechanics that can catch Windows administrators off guard and leave them scratching their heads.

Tip #1: The VCMS security model revolves around privileges, roles, and objects.  The more than 100 privileges define rights, roles are a collection of privileges, and roles are assigned to objects which are entities in the virtual infrastructure as shown in the diagram borrowed below:

1-15-2009 11-24-45 AM

Windows administrators will be used to the concept of assigning NTFS permissions to files, folders, and other objects in Active Directory.  It is very common for Windows objects to contain more than one Access Control Entry (ACE) which can be a group (such as “Accounting”, “Marketing”, etc.) or an explicit user (such as “Bob”, Sally”, etc.)  The same holds true for assigning roles to object in VC.

In some instances, which are not uncommon at all, a user may be granted permission to an object by way of more than one ACE.  For example, if both the Accounting and Marketing groups were assigned rights, and Sally was a member of both those groups, Sally would have rights to the object through both of those groups.  Using this same example, if the two ACEs defined different permissions to an object, the end result is a cumulative, so long as the ACE doesn’t contain “deny” which is special:  Sally would have the combined set of permissions.  The same holds true in VC.

Let’s take the above example a step further.  In addition to the two groups, which Sally is a member of, being ACLd to an object, now let’s say Sally’s user account object itself is an explicit ACE in the ACL list.  In the Windows world, the effect is Sally’s rights are still cumulative combining the three ACEs.  This is where the fork in the road lies in the VirtualCenter security model.  Roles explicitly assigned to a user object trump all other assigned or inherited permissions to the same object.  If the explicit ACE defines less permissions, the effective result is Sally will have less permissions than what her group membership would have provided.  If the explicit ACE defines more permissions, the effective result is Sally will have more permissions than what her group membership would have provided.  This is where Windows based VC administrators will be dumbfounded when a user suddenly calls with tales of things gray’d out in VirtualCenter, not enough permissions, etc.  Of course the flip side of the coin is a junior administrator suddenly finds themselves with cool new options in VC.  “Let’s see what this datastore button does”

Moral of the story from a real world perspective:  Assigning explicit permissions to user accounts in VC without careful planning will yield somewhat unpredictable results when inheritance is enabled (which is typical).  To take this to extremes, assigning explicit permissions to user accounts in VC, especially where inheritance in the VC hierarchy is involved, is a security and uptime risk when a user ends up with the wrong permissions accidentally.  For security and consistency purposes, I would avoid assigning permissions explicitly to user accounts unless you have a very clear understanding of the impacts currently and down the road.

Tip #2: Beware the use of the built in role Virtual Machine Administrator.  It’s name is misleading and the permissions it has are downright scary and not much different than the built in Administrator role.  For instance, the Virtual Machine Administrator role:  can modify VC and ESX host licensing, has complete control over the VC folder structure, has complete control over Datacenter objects, has complete control over datastores (short of file management), can remove networks, has complete control over inventory items such as hosts and clusters.  This list goes on and on.  I have three words:  What The Hell?!  I don’t know – the way my brain works is those permissions stretch well beyond the boundaries of what I would delegate for a Virtual Machine Administrator.

Moral of the story from a real world perspective:  Use the Virtual Machine Administrator role with extreme caution.  There is little disparity between the Administrator role and the Virtual Machine Administrator role, minus some items for Update Manager and changing VC permissions themselves. Therefore, any user who has the Virtual Machine Administrator role is practically an administrator.  The Virtual Machine Administrator role should not be used unless you have delegations that would fit this role precisely.  Another option would be clone the role and strip some of the more datacenter impactful permissions out of it.

Tip #3: Audit your effective VirtualCenter permissions on a regular basis, especially if you have large implementation with many administrators “having their hands in the cookie jar” so to speak.  If you use groups to assign roles in VC, then that means you should be auditing these groups as well (above and beyond virtualization conversations, administrative level groups should be audited anyway as a best practice).  This whitepaper has a nice Perl script for dumping VirtualCenter roles and permissions using the VMware Infrastructure Perl Toolkit.  Use of the script will automate the auditing process quite a bit and help transform a lengthy mundane task into a quicker one.  While you’re at it, it wouldn’t be a bad idea to periodically check tasks and events to see who is doing what.  There should be no surprises there.

Moral of the story from a real world perspective:  Audit your VirtualCenter roles and permissions.  When an unexpected datacenter disaster occurs from users having elevated privileges, one of the first questions to be asked in the post mortem meeting will be what your audit process is.  Have a good answer prepared.  Even better, avoid the disaster and down time through the due diligence of auditing your virtual infrastructure security.

For more information about VirtualCenter security, check out this great white paper or download the .pdf version from this link.  Some of the information I posted above I gathered from this document.  The white paper was written by Charu Chaubal, a technical marketing manager at VMware and Ph.D. in numerical modeling of complex fluids, with contributions from Doug Clark, and Karl Rummelhart.

If VirtualCenter security talk really gets your juices flowing, you should check out a new podcast launched by well known and respected VMTN community member/moderator and book author Edward Haletky that starts today called Virtualization Security Round Table.  It is sure to be good!

3 comments »

Posted in Virtualization

Tags: