Posts Tagged ‘vCenter Server’

Putting some money where my VMware mouth is

February 15th, 2009

I came home this afternoon from a Valentines Day wedding in North Dakota to find that my one and only workstation in the house (other than the work laptop) had a belated Valentines Day present for me:  It would no longer boot up.  No Windows.  No POST.  No video signal.  No beep codes.

DSC00473

I was feeling adventurous and I needed a relatively quick and inexpensive fix.  I decided to take one of the thin clients I received from Chip PC via VMworld 2008 plus a freshly deployed Windows XP template on the Virtual Infrastructure and promote this VDI solution to main household workstation status for the next few weeks.  The timing on this could not have been better.  The upcoming Minnesota VMUG on Wednesday March 11th is going to be VDI focused.  I guess I’ll have more to contribute at that meeting than I had originally planned on.  With any luck, Chip PC will be in attendance and we can discuss some things.

The thin client:  Chip PC Xtreme PC NG-6600 (model: EX6600N, part number: CPN04209).

Specs:

  • RMI – Alchemy Au 1550, 500MHz RISC processor (equivalent to 1.2GHz x86 TC processors)
  • 128MB DDR RAM
  • 64MB Disk-On-Chip with TFS
  • 128-bit 3D graphics acceleration engine with separate 2x8MB display memory SDRAM
  • Dual DVI ports each supporting 1920×1200 16-bit color.  Supports quad displays up to 1024×768
  • Audio I/O
  • 4 USB 2.0 ports
  • 10/100 Ethernet NIC
  • Power draw:  3.5W work mode, .35W sleep mode
  • OS:  Enhanced Microsoft Windows CE (6.00 R2 Professional)
  • Integrated applications (Plugins – note plugins are downloaded at no charge from the Chip PC website and are not, by default, embedded or included with the thin client – just enough OS concept)
    • Citrix ICA
    • RDP 5.2 and 6
    • Internet Explorer 6.0
    • VDM Client
    • VDI Client
    • Media Player
    • VPN Client
    • Ultra VNC
    • Pericom (Team Talk) Terminal Emulation
    • LPD Printer
    • ELO Touch Screen
  • Compatibility
    • Citrix WinFrame, MetaFrame, and Presentation Server 4.5
    • MS Windows Server 2000/2003
    • MS Windows NT 4.0 – TS Edition
    • VMware Virtual Desktop Interface using RDP
  • Full support of both local and network printers:  LPD, LPR, SMB, LPT, USB, COM
  • Support for USB mass storage (thumb drives – deal breaker for me)
  • Support for wireless USB NIC (not included)
  • etc. etc. etc.

DSC00474

Truth be told, this isn’t really a promotion in the sense that I had already performed extensive testing on it.  I hadn’t even taken the thing out of the box yet other than to register it for the extended warranty.  I’ve had only a little experience on these devices as I have an identical unit in the lab at work which I’ve spent a total of 30 minutes on.  To the best of my knowledge, this is the Cadillac unit from Chip PC.

I don’t have any fancy VDI brokering solutions here in the home lab and I’m not up to speed on VMware View so the plan is to leverage Thin Client -> RDP -> Windows XP desktop on VMware Virtual Infrastructure 3.5.

I think this is going to be a good test.  A trial by fire of VDI (granted, a fairly simple variation).  I spout a lot about the goodness that is VMware and now I’ll be eating some of my own dog food from the desktop workspace.  I’m a power user.  I’ve got my standard set of applications that I use on a regular basis and I’ve got a few hardware devices such as a flatbed scanner, iPod Shuffle, USB thumb drives, digital cameras, etc.  I should know within a short period of time whether or not this will be a viable solution for the short term.  Also add to the mix my wife’s career.  She uses our home computer to access her servers at work on a fairly regular basis.  Lastly, my wife sometimes works from home while I’m away at the office or traveling.  It’s going to be critical that this solution stays up and running and continues to be viable for my wife while I’m remote and not able to provide computer support.

So where am I at now?  I’ve got the VDI session patched along with my most critical applications installed to get me by in the short term:  Quicken, SnagIt, network printer, and Citrix clients.  I’ll install MS Office later but for now I can use the published application version of Office on my virtualized Citrix servers.  I’ve been listening some Electro House on www.di.fm on the VDI and music quality is as good as it was on my PC before it died, although it doesn’t completely drive my 5.1 surround in the den.  Pretty sure I’m getting 2.1 right now.  Oh well, at least the sub is thumpin.  Shhhh… the thin client is sleeping:

DSC00478

So what else?  As long as I’m throwing caution to the wind, I think it’s time to take the training wheels off VMware DPM (Distributed Power Management) and see what happens in a two node cluster.

2-15-2009 10-53-10 PM

Based on the environment below, what do you think will happen?  CPU load is very low, however, memory utilization is close to being over committed in a one host scenario. Will DPM kick in?

2-15-2009 10-53-59 PM

Most of my infrastructure at home is virtual including all components involving internet access both incoming and outgoing.  If the blog becomes unavailable for a while in the near future, I’ll give you one guess as to what happened.  🙂

No matter what the outcome, vmwarenews.de aka Roman Haug – you are no longer welcomed to republish my blog articles.  Albeit flattering, the fact that you have not even so much as asked in the first place has officially pissed me off.  You publish my content as if it were your own, written by you as indicated by the “by Roman” header preceeding each duplicated post.  Please remove my content from your site and refrain from syndicating my content going forward.  Thank you in advance.

Update: Roman Haug has offered an apology and I believe we have reached an understanding.  Thank you Roman!

VMGURU to release 4 chapters of VI3 book today

February 10th, 2009

Scott Herold of VMGuru.com and co-author of the book VMware Infrastructure 3: Advanced Technical Design Guide and Advanced Operations Guide has announced today the release of four of the book’s chapters in PDF format today.

I’ve read the previous version of this book a few years ago and I’m in the middle of reading the current version.  I HIGHLY recommend this book.  It is worth it’s weight in gold and the fact that the authors are going to begin giving it away for free to the virtualization community is baffling to me but yet at the same time it is a symbol of their generosity and commitment to providing the community with top notch technical and operations detail on VMware virtual infrastructure.

Generally speaking, many technical authors don’t make a pile of money writing books.  Be sure to thank the authors Ron Oglesby, Scott Herold, and Mike Laverick for their hard work and generosity.

More information about this book can be found here and here.  Stay tuned to VMGuru.com for the official release of these chapters which should happen sometime today.

Train Signal training discount through the month of February

January 31st, 2009

Train Signal is offering an astounding 25% off any virtualization product they sell through the month of February 2009.

Here is a short sample of their VMware ESX training video where instructor David Davis talks about templates and cloning virtual machines:

To take advantage of the 25% off, use the code BOCHENET at checkout.

I know first hand that the economy is tough.  Take advantage of this offer and get top shelf training for your dollar.  Train Signal offers a 90 day money back guarantee if you are not completely satisfied.

Three VirtualCenter security tips Windows administrators should know

January 15th, 2009

Good morning!  I’d like to take the opportunity to talk a bit about something that has been somewhat of a rock in my shoe as a seasoned Windows administrator from the NT 3.5 era:  The VirtualCenter (vCenter Server, VirtualCenter Management Server, VCMS, VC, etc.) security model, or more accurately, its unfamiliar mechanics that can catch Windows administrators off guard and leave them scratching their heads.

Tip #1: The VCMS security model revolves around privileges, roles, and objects.  The more than 100 privileges define rights, roles are a collection of privileges, and roles are assigned to objects which are entities in the virtual infrastructure as shown in the diagram borrowed below:

1-15-2009 11-24-45 AM

Windows administrators will be used to the concept of assigning NTFS permissions to files, folders, and other objects in Active Directory.  It is very common for Windows objects to contain more than one Access Control Entry (ACE) which can be a group (such as “Accounting”, “Marketing”, etc.) or an explicit user (such as “Bob”, Sally”, etc.)  The same holds true for assigning roles to object in VC.

In some instances, which are not uncommon at all, a user may be granted permission to an object by way of more than one ACE.  For example, if both the Accounting and Marketing groups were assigned rights, and Sally was a member of both those groups, Sally would have rights to the object through both of those groups.  Using this same example, if the two ACEs defined different permissions to an object, the end result is a cumulative, so long as the ACE doesn’t contain “deny” which is special:  Sally would have the combined set of permissions.  The same holds true in VC.

Let’s take the above example a step further.  In addition to the two groups, which Sally is a member of, being ACLd to an object, now let’s say Sally’s user account object itself is an explicit ACE in the ACL list.  In the Windows world, the effect is Sally’s rights are still cumulative combining the three ACEs.  This is where the fork in the road lies in the VirtualCenter security model.  Roles explicitly assigned to a user object trump all other assigned or inherited permissions to the same object.  If the explicit ACE defines less permissions, the effective result is Sally will have less permissions than what her group membership would have provided.  If the explicit ACE defines more permissions, the effective result is Sally will have more permissions than what her group membership would have provided.  This is where Windows based VC administrators will be dumbfounded when a user suddenly calls with tales of things gray’d out in VirtualCenter, not enough permissions, etc.  Of course the flip side of the coin is a junior administrator suddenly finds themselves with cool new options in VC.  “Let’s see what this datastore button does”

Moral of the story from a real world perspective:  Assigning explicit permissions to user accounts in VC without careful planning will yield somewhat unpredictable results when inheritance is enabled (which is typical).  To take this to extremes, assigning explicit permissions to user accounts in VC, especially where inheritance in the VC hierarchy is involved, is a security and uptime risk when a user ends up with the wrong permissions accidentally.  For security and consistency purposes, I would avoid assigning permissions explicitly to user accounts unless you have a very clear understanding of the impacts currently and down the road.

Tip #2: Beware the use of the built in role Virtual Machine Administrator.  It’s name is misleading and the permissions it has are downright scary and not much different than the built in Administrator role.  For instance, the Virtual Machine Administrator role:  can modify VC and ESX host licensing, has complete control over the VC folder structure, has complete control over Datacenter objects, has complete control over datastores (short of file management), can remove networks, has complete control over inventory items such as hosts and clusters.  This list goes on and on.  I have three words:  What The Hell?!  I don’t know – the way my brain works is those permissions stretch well beyond the boundaries of what I would delegate for a Virtual Machine Administrator.

Moral of the story from a real world perspective:  Use the Virtual Machine Administrator role with extreme caution.  There is little disparity between the Administrator role and the Virtual Machine Administrator role, minus some items for Update Manager and changing VC permissions themselves. Therefore, any user who has the Virtual Machine Administrator role is practically an administrator.  The Virtual Machine Administrator role should not be used unless you have delegations that would fit this role precisely.  Another option would be clone the role and strip some of the more datacenter impactful permissions out of it.

Tip #3: Audit your effective VirtualCenter permissions on a regular basis, especially if you have large implementation with many administrators “having their hands in the cookie jar” so to speak.  If you use groups to assign roles in VC, then that means you should be auditing these groups as well (above and beyond virtualization conversations, administrative level groups should be audited anyway as a best practice).  This whitepaper has a nice Perl script for dumping VirtualCenter roles and permissions using the VMware Infrastructure Perl Toolkit.  Use of the script will automate the auditing process quite a bit and help transform a lengthy mundane task into a quicker one.  While you’re at it, it wouldn’t be a bad idea to periodically check tasks and events to see who is doing what.  There should be no surprises there.

Moral of the story from a real world perspective:  Audit your VirtualCenter roles and permissions.  When an unexpected datacenter disaster occurs from users having elevated privileges, one of the first questions to be asked in the post mortem meeting will be what your audit process is.  Have a good answer prepared.  Even better, avoid the disaster and down time through the due diligence of auditing your virtual infrastructure security.

For more information about VirtualCenter security, check out this great white paper or download the .pdf version from this link.  Some of the information I posted above I gathered from this document.  The white paper was written by Charu Chaubal, a technical marketing manager at VMware and Ph.D. in numerical modeling of complex fluids, with contributions from Doug Clark, and Karl Rummelhart.

If VirtualCenter security talk really gets your juices flowing, you should check out a new podcast launched by well known and respected VMTN community member/moderator and book author Edward Haletky that starts today called Virtualization Security Round Table.  It is sure to be good!

Uptime lost during VMotion

January 11th, 2009

That’s right. We lose uptime during every VMotion. Relax just a bit – I’m not talking about actual uptime/downtime availability of the guest VM in the datacenter. I’m speaking to the uptime performance metric tracked in the VirtualCenter database. It’s a bug that was introduced in VirtualCenter 2.0 and has remained in the code to this day in VirtualCenter 2.5.0 Update 3. Here’s how it works:

We’ve got a VM named Exchange1. VirtualCenter displays its uptime as 28 days is indicated by the screenshots below:

1-11-2009 12-04-06 AM

1-11-2009 12-10-14 AM

Now the VM has been VMotioned from host SOLO to host LANDO. Notice what has happened to uptime. It has disappeared from the console:

1-11-2009 12-07-37 AM

The real proof in what has ultimately happened is we see from the performance chart the latest uptime metric has been reset from 27.99 days as shown above to 0.0026620 days:

1-11-2009 12-10-54 AM

Sometimes the VIC console will show the uptime counter start over at 0 days, then on to 1 days, etc. Other times the uptime counter will remain blank for days or weeks as you can see from my three other VMs in the first screenshot which show no uptime.

This brings us to an interesting discussion. What would you like uptime in the VIC to mean exactly? Following are my observations and thoughts on VMware’s implementation of the uptime metric in VirtualCenter.

In previous versions of VirtualCenter, a soft reboot of the VM inside of the OS would reset the uptime statistic in VirtualCenter. I believe this was a function of VMware Tools that triggered this.

Today in VirtualCenter 2.5.0 Update 3, a soft reboot inside the guest VM does not reset the uptime statistic back to zero.

A VM which has no VMware Tools installed that is soft rebooted inside of the OS (ie. we’re not talking about any VMware console power operation here) does not reset the uptime statistic.

I could see the community take a few different sides on this as there are two variations of the definition of uptime we’re dealing with here. Uptime of the guest VM OS and uptime of the VM’s virtual hardware.

  1. Should uptime translate into how long the VM virtual hardware has been powered on from a virtual infrastructure standpoint?
  2. Or should uptime translate into how long the OS inside the VM has been up, tracked by VMware Tools?

The VMware administrator cares about the length of time a VM has been powered on. It is the powered on VM that consumes resources from the four resource food groups and impacts capacity.

The guest VM OS administrator, whether it be Windows or Linux, cares about uptime of the guest OS. The owner of the OS is held to SLAs by the business lines.

My personal opinion is that the intended use of the Virtual Infrastructure Client is for the VMware administrator and thus should reflect virtual infrastructure information. My preference is that the uptime statistic in VirtualCenter tracks power operations of the VM irregardless of any reboot sequences of the OS inside the VM. In other words, uptime is not impacted by VMware Tools heartbeats or reboots inside the guest VM. The uptime statistic should only be reset when the VM is powered off or power reset (including instances where HA has recovered a VM).

At any rate, due to the bug that uptime has in VirtualCenter 2.0 and above, it’s a fairly unreliable performance metric for any virtual infrastructure using VMotion and DRS. Furthermore, the term itself can be misleading depending on the administrators interpretation of uptime versus what’s written in the VirtualCenter code.

I submitted a post in VMware’s Product and Feature Suggestions forum in January of 2007 recording the uptime reset on VMotion issue. As this problem periodically bugs me, I followed up a few times. Once in a follow up post in the thread above, and at least one time externally requesting someone from VMware take a look at it. Admittedly I do not have an SR open.

VMware, can we get this bug fixed? After all, if the hypervisor has become an every day commodity item leaving the management tools as the real meat and potatoes, you should make sure our management tools work properly.

Thank you,

Jas

Guest blog entry: VMotion performance

January 5th, 2009

Good afternoon VMware virtualization enthusiasts and Hyper-V users whom Microsoft has condoned on your behalf that you don’t have a need for hot migration if you have an intern and $50,000 cash.

Simon Long has shared with us this fantastic article he wrote regarding VMotion performance.  More specifically, fine tuning concurrent VMotions allowed by vCenter.  This one is going in my document repository and tweaks ‘n’ tricks collection.  Thank you Simon and everyone please remember that virtualization is not best enjoyed in moderation!

Simon can be reached via email at contact (at) simonlong.co.uk as well as @SimonLong_ on Twitter.


I’ll set the scene a little….

I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.

I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.

“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.

3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, Chris Dye, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.

After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!

But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once.

Here are the steps to increase the amount of Simultaneous VMotion Migrations per Host.

1. RDP to your vCenter Server.
2. Locate the vpdx.cfg (Default location “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”)
3. Make a Backup of the vpxd.cfg before making any changes
4. Edit the file in using WordPad and insert the following lines between the <vpdx></vpdx> tags;

<ResourceManager>
<maxCostPerHost>12</maxCostPerHost>
</ResourceManager>

5. Now you need to decide what value to give “maxCostPerHost”.

A Cold Migration has a cost of 1 and a Hot Migration aka VMotion has a cost of 4. I first set mine to 12 as I wanted to see if it would now allow 3 VMotions at once, I now permanently have mine set to 24 which gives me 6 simultaneous VMotions per Host (6×4 = 24).

I am unsure on the maximum value that you can use here, the largest I tested was 24.

6. Save your changes and exit WordPad.
7. Restart “VMware VirtualCenter Server” Service to apply the changes.

Now I know how to change the amount of simultaneous VMotions per Host, I decided to run some tests to see if it actually made any difference to the overall VMotion Performance.

I had 2 Host’s with 16 almost identical VM’s. I created a job to Migrate my 16 VM’s from Host 1 to Host 2.

Both Hosts VMotion vmnic was a single 1Gbit nic connected to a CISCO Switch which also has other network traffic on it.


The Network Performance graph above was recorded during my testing and is displaying the “Network Data Transmit” measurement on the VMotion vmnic. The 3 sections highlighted represent the following;

Section 1 – 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 6 simultaneous VMotions.
Time taken = 3.30

Section 2 – This was not a test, I was simply just migrating the VM’s back onto the Host for the 2nd test (Section 3).

Section 3 – 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 2 simultaneous VMotions.
Time taken = 6.36

Time Different = 3.06
3 Mins!! I wasn’t expecting it to be that much. Imagine if you had a 50 Host cluster…how much time would it save you?
I tried the same test again but only migrating 6 VM’s instead of 16.

Migrating off 6 VM’s with only 2 simultaneous VMotions allowed.
Time taken = 2.24

Migrating off 6 VM’s with 6 simultaneous VMotions allowed.
Time taken = 1.54

Time Different = 30secs

It’s still an improvement all be it not so big.

Now don’t get me wrong, these tests are hardly scientific and would never have been deemed as completely fair test but I think you get the general idea of what I was trying to get at.

I’m hoping to explore VMotion Performance further by looking at maybe using multiple physical nics for VMotion and Teaming them using EtherChannel or maybe even using 10Gbit Ethernet. Right now I don’t have the spare Hardware to do that but this is definitely something I will try when the opportunity arises.

Update 4/5/11Limit Concurrent vMotions in vSphere 4.1 by Elias Khnaser.

Update 10/3/12:  Changes to vMotion in vSphere 4.1 per VMware KB 1022851:

In vSphere 4.1:
  • Migration with vMotion and DRS for virtual machines configured with USB device passthrough from an ESX/ESXi host is supported
  • Fault Tolerant (FT) protected virtual machines can now vMotion via DRS. However, Storage vMotion is unsupported at this time.
    Note: Ensure that the ESX hosts are at the same version and build.
In addition to the above, vSphere 4.1 has improved vMotion performance and allows:
  • 4 concurrent vMotion operations per host on a 1Gb/s network
  • 8 concurrent vMotion operations per host on a 10Gb/s network
  • 128 concurrent vMotion operations per VMFS datastore

Note: Concurrent vMotion operation is currently supported only when source and destination hosts are in the same cluster. For further information, see the Configuration Maximums for VMware Sphere 4.1 document.

The vSphere 4.1 configuration maximums above remain true for vSphere 5.x.  Enhanced vMotion operations introduced in vSphere 5.1 also count against the vMotion maximums above as well as the Storage vMotion configuration maximums (8 concurrent Storage vMotions per datastore and 2 concurrent Storage vMotions per host as well as 8 concurrent non-vMotion provisioning operations per host).  Eric Sloof does a good of explaining that here.

VMware product name changes

December 3rd, 2008

Quick update on a news item you may have already heard about. Remember those VMware product/component decoder rings you might have started working on after the VMworld 2008 announcements? It’s time for an update. VMware announced a handful of product name changes on Monday:

  1. VMware VirtualCenter is now VMware vCenter Server
  2. VMware vCenter is the family name for all management products
  3. VMware Lab Manager is now VMware vCenter Lab Manager (since it is in the management products family)
  4. The VMware vCenter prefix applies to the other products in the management products family as well
  5. VMware View is the family name for all VDI/VDM products
  6. VMware VDI is now VMware View
  7. VMware VDM is now VMware View Manager

I’m not real fond of name changes unless there is a good reason behind it. I’ll give VMware the benefit of the doubt that there was good reason to make these changes, although not knowing myself 100% what is up VMware’s sleeve, the timing is somewhat debatable. Couldn’t they have waited until the next generation of Virtual Infrastructure to align the products and components? Citrix did this with Presentation Server when they instantly re-branded it to XenApp. It confused a lot of people, especially the newcomers. I hope confusion among VMware customers is minimized. It’s going to take a little while for these new names to become second nature for me.

What do you think of the name changes? Feedback is always welcomed here.