Archive for January, 2009

Three VirtualCenter security tips Windows administrators should know

January 15th, 2009

Good morning!  I’d like to take the opportunity to talk a bit about something that has been somewhat of a rock in my shoe as a seasoned Windows administrator from the NT 3.5 era:  The VirtualCenter (vCenter Server, VirtualCenter Management Server, VCMS, VC, etc.) security model, or more accurately, its unfamiliar mechanics that can catch Windows administrators off guard and leave them scratching their heads.

Tip #1: The VCMS security model revolves around privileges, roles, and objects.  The more than 100 privileges define rights, roles are a collection of privileges, and roles are assigned to objects which are entities in the virtual infrastructure as shown in the diagram borrowed below:

1-15-2009 11-24-45 AM

Windows administrators will be used to the concept of assigning NTFS permissions to files, folders, and other objects in Active Directory.  It is very common for Windows objects to contain more than one Access Control Entry (ACE) which can be a group (such as “Accounting”, “Marketing”, etc.) or an explicit user (such as “Bob”, Sally”, etc.)  The same holds true for assigning roles to object in VC.

In some instances, which are not uncommon at all, a user may be granted permission to an object by way of more than one ACE.  For example, if both the Accounting and Marketing groups were assigned rights, and Sally was a member of both those groups, Sally would have rights to the object through both of those groups.  Using this same example, if the two ACEs defined different permissions to an object, the end result is a cumulative, so long as the ACE doesn’t contain “deny” which is special:  Sally would have the combined set of permissions.  The same holds true in VC.

Let’s take the above example a step further.  In addition to the two groups, which Sally is a member of, being ACLd to an object, now let’s say Sally’s user account object itself is an explicit ACE in the ACL list.  In the Windows world, the effect is Sally’s rights are still cumulative combining the three ACEs.  This is where the fork in the road lies in the VirtualCenter security model.  Roles explicitly assigned to a user object trump all other assigned or inherited permissions to the same object.  If the explicit ACE defines less permissions, the effective result is Sally will have less permissions than what her group membership would have provided.  If the explicit ACE defines more permissions, the effective result is Sally will have more permissions than what her group membership would have provided.  This is where Windows based VC administrators will be dumbfounded when a user suddenly calls with tales of things gray’d out in VirtualCenter, not enough permissions, etc.  Of course the flip side of the coin is a junior administrator suddenly finds themselves with cool new options in VC.  “Let’s see what this datastore button does”

Moral of the story from a real world perspective:  Assigning explicit permissions to user accounts in VC without careful planning will yield somewhat unpredictable results when inheritance is enabled (which is typical).  To take this to extremes, assigning explicit permissions to user accounts in VC, especially where inheritance in the VC hierarchy is involved, is a security and uptime risk when a user ends up with the wrong permissions accidentally.  For security and consistency purposes, I would avoid assigning permissions explicitly to user accounts unless you have a very clear understanding of the impacts currently and down the road.

Tip #2: Beware the use of the built in role Virtual Machine Administrator.  It’s name is misleading and the permissions it has are downright scary and not much different than the built in Administrator role.  For instance, the Virtual Machine Administrator role:  can modify VC and ESX host licensing, has complete control over the VC folder structure, has complete control over Datacenter objects, has complete control over datastores (short of file management), can remove networks, has complete control over inventory items such as hosts and clusters.  This list goes on and on.  I have three words:  What The Hell?!  I don’t know – the way my brain works is those permissions stretch well beyond the boundaries of what I would delegate for a Virtual Machine Administrator.

Moral of the story from a real world perspective:  Use the Virtual Machine Administrator role with extreme caution.  There is little disparity between the Administrator role and the Virtual Machine Administrator role, minus some items for Update Manager and changing VC permissions themselves. Therefore, any user who has the Virtual Machine Administrator role is practically an administrator.  The Virtual Machine Administrator role should not be used unless you have delegations that would fit this role precisely.  Another option would be clone the role and strip some of the more datacenter impactful permissions out of it.

Tip #3: Audit your effective VirtualCenter permissions on a regular basis, especially if you have large implementation with many administrators “having their hands in the cookie jar” so to speak.  If you use groups to assign roles in VC, then that means you should be auditing these groups as well (above and beyond virtualization conversations, administrative level groups should be audited anyway as a best practice).  This whitepaper has a nice Perl script for dumping VirtualCenter roles and permissions using the VMware Infrastructure Perl Toolkit.  Use of the script will automate the auditing process quite a bit and help transform a lengthy mundane task into a quicker one.  While you’re at it, it wouldn’t be a bad idea to periodically check tasks and events to see who is doing what.  There should be no surprises there.

Moral of the story from a real world perspective:  Audit your VirtualCenter roles and permissions.  When an unexpected datacenter disaster occurs from users having elevated privileges, one of the first questions to be asked in the post mortem meeting will be what your audit process is.  Have a good answer prepared.  Even better, avoid the disaster and down time through the due diligence of auditing your virtual infrastructure security.

For more information about VirtualCenter security, check out this great white paper or download the .pdf version from this link.  Some of the information I posted above I gathered from this document.  The white paper was written by Charu Chaubal, a technical marketing manager at VMware and Ph.D. in numerical modeling of complex fluids, with contributions from Doug Clark, and Karl Rummelhart.

If VirtualCenter security talk really gets your juices flowing, you should check out a new podcast launched by well known and respected VMTN community member/moderator and book author Edward Haletky that starts today called Virtualization Security Round Table.  It is sure to be good!

Save the date: CloudCamp Minneapolis 4/18/08 9am-3pm

January 15th, 2009

The future of the enterprise datacenter is cloud computing. So check this out:

“CloudCamp is an unconference where early adapters of Cloud Computing technologies exchange ideas. With the rapid change occurring in the industry, we need a place we can meet to share our experiences, challenges and solutions. At CloudCamp, you are encouraged you to share your thoughts in several open discussions, as we strive for the advancement of Cloud Computing. End users, IT professionals and vendors are all encouraged to participate.

Our first event in the Twin Cities area will be held Saturday, April 18th at the University of Minnesota.”

Registration is free but there is very limited availability (about 87 tickets left as I write this).

VMware announces details of the vExpert program

January 14th, 2009

This evening, VMware announced a new community award called VMware vExpert!

“The VMware vExpert Awards will be given to individuals who have significantly contributed to the overall community of VMware users over the past year, either online or offline. You might be contributing online to blogs, forums, wikis, or other online sites. You might be organizing VMUG meetings or otherwise getting the word out to local IT professionals. You’re helping spread the word about virtualization and making people successful in deploying this game-changing technology. We want to thank you.”

Although VMware doesn’t prefer the comparison to be made, the program is somewhat similar to the Microsoft MVP program in that it is annually renewable, provides access to exclusive VMware content, and above all provides recognition among peers for hard work.

The nomination form is available.  Go ahead and nominate yourself or someone you know who has dedicated much of their time helping VMware become successful.

Nominate someone today!

If by chance you are interested in nominating me,

my email address is

my VMware communities username is jasonboche.

Opt in for /opt partitioning

January 13th, 2009

I’m currently reading chapters of the VMware Infrastructure 3 Advanced Technical Design Guide in my spare time and I came across their recommended ESX partitioning strategy.  I’d like to think I’ve got a pretty good handle on ESX partitioning.  I’m quite comfortable with it, and thus normally I would breeze quickly through partitioning documentation, verifying along the way that my partitioning was still on par.

My partitioning scheme was still looking good until I came across a new recommendation of creating a dedicated /opt partition.  This is something I hadn’t done before.  Under normal circumstances without a dedicated /opt partition, /opt is going to be a directory off / (root).  Why is this not the greatest idea?  I was enlightened by the fact that some VMware HA logging as well as some hardware agent logging is stored in /opt.  There are enough posts on the VMTN forums describing situations where excessive logging on /opt chewed up all available partition space on / to warrant proactive measures.  As we know, running out of disk space on / is less than ideal and that’s putting it mildly.

Solution:  When building the ESX host, create a dedicated partition for /opt.

Now I’ll be honest, I’ve never run into a situation of excessive logging on /opt, but this is one of those strategies that falls into the category of “an ounce of prevention is worth a pound of cure”.  Learn from the experiences of other VI administrators who I’m sure suffered some downtime as a result.  Don’t wait for this happen to you when it doesn’t need to.  The ESX host and its health is critical for datacenter and production operations.  When the ESX host isn’t happy, the VMs running on it usually aren’t happy either.

That said, here is my updated partitioning scheme for ESX 3.5.0.

Create the following partitions in the following order:

Mount Point Type Size Primary? Notes
/boot ext3 250MB Yes The default from VMware is 97MB.  When we migrated from ESX 2.x to ESX 3.x the partition size grew from 50MB to nearly 100MB.  I came up with 250MB to leave breathing room for future versions of ESX which may need an even larger /boot partition.  This is all a moot point because I don’t do in place upgrades.  I rebuild with new versions.
<swap> 1600MB Yes Twice the maximum amount of allocatable service console memory.  My COS memory allocation is 500MB but if I ever increase COS memory to the 800MB max in the future, I’ve already got enough swap for it without having to rebuild the box to repartition.
/ ext3 4096MB Yes The default from VMware is 3.7GB.  We want plenty of space for this mount point so that we do not suffer the serious consequences of running out.
/home ext3 4096MB Not really needed anymore except as home directory storage for local and the default from VMware is that this partition no longer is created.  For me this is just a carryover from the old ESX days.  And disk space is fairly cheap (unless booting from SAN).  I’ll put this and other custom partitioning out to pasture when I convert to ESXi where we are force fed VMware’s recommended partitioning.
/tmp ext3 4096MB The default from VMware is that it doesn’t exist, rather it creates the /tmp folder under the / mount point.  This is not a great idea.  VMware uses a small portion of /tmp for the installation of the VirtualCenter agent but my philosophy is we should have plenty of sandbox space in /tmp for the unpacking/untarring of 3rd party utils such as HP Systems Insight Manager agents.
/var ext3 4096MB The default from VMware is 1.1GB and additionally VMware makes the mount point /var/log isolating this partition strictly for VMware logs.  We want plenty of space for this mount point so that we do not suffer the serious consequences of running out.  In addition, we want this to be a separate mount point so as not to risk the / mount point by consuming its file system space.  VMware logs and other goodies are stored on this mount point.
/opt ext3 4096 The default from VMware is that it doesn’t exist, rather it creates the /opt folder under the /mount point.  We want enough space for this mount point so that we do not suffer the consequences of running out.  In addition, we want this to be a separate mount point so as not to risk the / mount point by consuming its file system space.  VMware HA logging and sometimes hardware agent logging are stored on this mount point.  This is a VI3 ATDG recommendation.
<vmkcore> 110MB The default from VMware is 100MB.  I got the 110MB recommendation from Ron Oglesby in his RapidApp ESX Server 3.0 Quick Start Guide (this book is a gem by the way; my copy is worn down to the nub).  Although I asked Ron, he never explained to me where he came up with 110MB but let’s just assume the extra 10MB is cushion “just in case”.  This is the VMKernel core dump partition.  The best case scenario is you rarely if ever have a use for this partition although it is required by ESX whether it’s used by a purple screen of death (PSOD) or not.
Leave the remaining space unpartitioned.  This can be partitioned as VMFS-3 later using VirtualCenter for maximum block alignment.

Hyper9 expands beta testing

January 13th, 2009

New readers of my blog may not be aware of Hyper9 and their Google-like Hyper9 VI insight and infiltration product that is being diligently tested in a controlled beta program.

Hyper9 is back from their brief holiday slumber and has announced that they would like to widen their audience of beta testers. This is your opportunity to provide feedback and request features of a hot new product that is a pioneer in VI management tools. If you are interested in joining the beta program, please contact me.

There are a few guidelines and requirements to becoming full members of the Beta experience, and I hope you are able to meet these.

Beta Participant Minimum Environment Requirements

  • VMwareä ESX 3.0+
  • (1) VMware VirtualCenter Instance
  • (2) VMware ESX Host Servers

· (20) Virtual Machines

Additional Requirements

  • If selected, you must download and install the software within five (5) days of receiving the beta software. Can you do this?
  • When you have completed the installation process of Hyper9’s software, we ask that you notify us that this action has been completed. Can you do this?
  • Users from competitor companies are not eligible for participation.
  • Users will have to provide Hyper9 with their company’s name and Web site information.
  • Users will have to provide Hyper9 with their company email address for verification.

VKernel CompareMyVM

January 13th, 2009

VKernel is sponsoring a new virtualization community resource called CompareMyVM.  It’s a forum where virtualization users around the globe can submit and share their VM virtual hardware sizings.  In VKernel’s words:

CompareMyVM is a community site where you learn how to right size your VMs. Compare your virtual machine resource allocations with that of the community at large. You can compare by application category to see how your peers are allocating resources to gain the most efficiency out of their environments. You can also contribute your VM sizings to the community. Your submissions are anonymous.

Things You Can Do On CompareMyVM…

* Browse Community Submissions
* Vote Submissions Up or Down
* Edit Submissions
* Submit Your Own VMs

This resource will prove to be handy for VI administrators of any skill level who are searching for a “known good” virtual hardware configuration for an application that may be new to them, but others have already had experience with.  This is yet another form of knowledge sharing and leveraging knowledge through peers to gain insight and save time in deployment of unfamiliar applications and architecture.

The community has already been seeded with six pages of configurations submitted as of this writing.  One thing that is a little weak at this point though is the ratings (or lack thereof).  If VKernel could somehow encourage more ratings, confidence in the submitted configurations would be bolstered.

Thank you for this resource VKernel.

IT blogger of the week

January 12th, 2009

I was informed by Techtarget’s IT Knowledge Exchange that I am their IT blogger of the week.  Thank you!  I deeply appreciate the recognition!

Similar to HR departments, IT staff loves recognition but usually doesn’t receive a lot of it.  We’re only called when there are problems.  With that in mind, why not surprise your HR department with a nice “thank you” this week for processing your paycheck correctly and on time as they do week after week?  🙂