Posts Tagged ‘vCenter Server’

vCenter Server Linked Mode Configuration Error

October 23rd, 2010

As of vCenter Server 4.1, VMware supports Windows Server 2008 R2 as a vCenter platform (remember 2008 R2 is 64-bit only).  With this, I expect many environments will be configured with vCenter Server on Microsoft’s newest Server operating system.

While working in the lab with vCenter Server 4.1 on Windows Server 2008 R2, I ran into an issue configuring Linked Mode via the vCenter Server Linked Mode Configuration shortcut.  Error 28035. Setup failed to copy LDIFDE.EXE from System folder to ‘%windir%\ADAM’ folder.

SnagIt Capture

After no success in relaxing Windows NTFS permissions, I remembered it’s a Windows Server 2008 R2 permissions issue.  The resolution is quite simple and is often the solution when running into similar errors on Windows 7 and Windows Server 2008 R2.  In addition, I found the workaround documented in VMware KB 1025637.  Rather than launching the vCenter Server Linked Mode Configuration as you normally would by clicking on the icon, instead, right click the shortcut and choose Run as administrator.

SnagIt Capture 

You should find that launching the shortcut in the administrator context grants the installer the permissions necessary to complete Linked Mode configuration.

CPU Ready to %RDY Conversion

October 21st, 2010

Most customers expect x amount of performance out of their virtual machines which is going to be dependent on established Service Level Agreements (SLAs).  Capacity planning, tuning, and monitoring performance all play a role in meeting customer SLAs.  When questioning performance of a physical machine, one of the first ubiquitous metrics that comes to mind is CPU utilization.  Server administrators are naturally inclined to look at this metric on virtual machines as well.  However, when looking at VM performance, Ready time is an additional metric to be examined from a CPU standpoint.  This metric tells us how much time the guest VM is waiting for its share of CPU execution from the host.

I began learning ESX in 2005 on version 2.0.  At that time, the VMware ICM class focused a lot on leveraging the Service Console.  At that time, vCenter Server 1.x was brand new and as such, ESXTOP was king for performance monitoring.  In particular, the %RDY metric in ESXTOP was used to reveal CPU bottlenecks as described above.  %RDY provides statistics in a % format.  I learned what acceptable tolerances were, I learned when to be a little nervous, and I could pretty well predict when the $hit was hitting the fan inside a VM from a CPU standpoint.  Duncan Epping at Yellow Bricks dedicates a page to ESXTOP statistics on his blog and at the very beginning, you’ll see a threshold he has published which you should keep in the back of your mind.

Today, ESXTOP still exists fortunately (it’s one of my favorite old-school-go-to tools).  The Service Console is all but gone, however, you’ll still find resxtop in VMware’s vMA appliance which is used to remotely manage ESXi (and ESX as well).  But what about the vSphere Client and vCenter Server?  With the introduction of vCenter Server, the disappearance of the Service Console, and the inclination of a Windows based administrator to lean on GUI based tools as a preference, notable focus has moved away from the CLI approach in lieu of the vSphere Client (in conjunction with the vCenter Server). 

Focusing on a VM in the vSphere Client, you’ll find a performance metric called CPU Ready.  This is the vSphere Client metric which tells us how much time the guest VM is waiting for its share of CPU execution from the host just as %RDY did in ESXTOP.  But when you look at the statistics, you’ll notice a difference.  %RDY in ESXTOP provides us with metrics in a % format.  CPU Ready in the vSphere Client provides metrics in a millisecond summation format.  I learned way back from the ICM class and through trench experience that ~10% RDY (per each vCPU) is a threshold to watch out for.  How does a % value from ESXTOP translate to a millisecond value in the vSphere Client?  It doesn’t seem to be widely known or published but I’ve found it explained a few places.  A VMware communities document here and a Josh Townsend blog post here.

There’s a little math involved.  To convert the vSphere Client CPU Ready metric to the ESXTOP %RDY metric, you divide the CPU Ready metric by the rollup summation (which are both values in milliseconds).  What does this mean?  Say for instance you’re looking at the overall CPU Ready value for a VM in Real-time.  Real-time is refreshed every 20 seconds and represents a rollup of values over a 20 second period (that’s 20,000 milliseconds).  Therefore…

  • If the CPU Ready value for the VM is, say 500 milliseconds, we divide 500 milliseconds by 20,000 milliseconds and arrive at nearly 3% RDY.  Hardly anything to be concerned about. 
  • If the CPU Ready time were 7,500, we divide 7,500 milliseconds by 20,000 milliseconds and arrive at 37.5% RDY or $hit hitting the fan assuming a 1 vCPU VM. 

What do I mean above by 1 vCPU VM?  The overall VM CPU Ready metric is the aggregate total of CPU Ready for each vCPU.  This should sound familiar – if you know how %RDY works in ESXTOP, then you’re armed with the knowledge needed to understand what I’m explaining.  The %RDY value in ESXTOP is the aggregate total of CPU Ready for each vCPU.  In other words, if you saw a 20% RDY value in ESXTOP for a 4 vCPU VM, the actual %RDY for each vCPU is 5% which is well under the 10% threshold we generally watch for.  In the vSphere Client, not only can you look at the overall aggregate CPU Ready for a particular VM (which should be divided by the number of assigned vCPUs for the VM), but you can also look at the CPU Ready values for the individual vCPUs themselves.  It is the per CPU Ready value which should be compared with published and commonly known thresholds.  When looking at Ready values, it’s important to interpret the data correctly in order to compare the right data to thresholds.

I’ve often heard the conversation of “how do I convert millisecond values in the vSphere Client to % values in ESXTOP?”  I’ve provided a working example using CPU Ready data.  Understand it can be applied to other metrics as well.  Hopefully this helps.

Hardware Status and Maintenance Mode

October 20th, 2010

I’m unable to view hardware health status data while a host is in maintenance mode in my vSphere 4.0 Update 1 environment.

SnagIt Capture

A failed memory module was replaced on a host but I’m skeptical about taking it out of maintenance mode until I am sure it is healthy.  There is enough load on this cluster such that removing the host from maintenance mode will result in DRS moving VM workloads onto it within five minutes.  For obvious reasons, I don’t want VMs running on an unhealthy host.

So… I need to disable DRS at the cluster level, take the host out of maintenance mode, verify the hardware health on the Hardware Status tab, then re-enable DRS.  It’s a round about process, particularly if it’s a production environment which requires a Change Request (CR) with associated approvals and lead time to toggle the DRS configuration. 

Taking a look at KB 1011284, VMware acknowledges the steps above and considers the following a resolution to the problem:

Resolution

By design, the host monitoring agents (IPMI) are not supported while the ESX host is in maintenance mode. You must exit maintenance mode to view the information on the Hardware Status tab. To take the ESX host out of maintenance mode:

1.Right click ESX host within vSphere Client.

2.Click on Exit Maintenance Mode.

Fortunately, this design specification has been improved by VMware in vSphere 4.1 where I have the ability to view hardware health while a host is in maintenance mode.

vCenter Storage Monitoring Plug-in Disabled

October 18th, 2010

Those who have upgraded to vSphere (hopefully most of you by now) may become accustomed to the new tab in vCenter labeled Storage Views. From time to time, you may notice that this tab mysteriously disappears from a view where it should normally be displayed.  If you’re a subscriber to my vCalendar, you’ll find a tip on July 18th which speaks to this:

Is your vSphere Storage Views tab or host Hardware Status tab not functioning or missing? Make sure the VMware VirtualCenter Management Webservices service is running on the vCenter Server.

The solution above is an easy enough resolution, but what if that doesn’t fix the problem?  I ran into another instance of the Storage Views tab disappearing and it was not due to a stopped VMware VirtualCenter Management Webservices service.  After a short investigation, I found a failed or disabled vCenter Storage Monitoring (Storage Monitoring and Reporting) plug-in:

SnagIt Capture

For those who cannot read the screen shot detail above, and for the purposes of Google search, I’ll paste the error code below:

The plug-in failed to load on server(s) <your vCenter Server> due to the following error: Could not load file or assembly ‘VpxClientCommon, Version=4.1.0.0, Culture=neutral, PublicKeyToken=7c8-0a434483c7c50’ or one of its dependencies. The system cannot find the file specified.

I performed some testing in the lab and here’s what I found.  Long story short, installation of the vSphere 4.1 Client on a system which already has the the vSphere 4.0 Update 1 Client installed causes the issue.  The 4.1 Client installs a file called SMS.dll (dated 5/13/2010) into the directory C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Plugins\SMS\ overwriting the previous version (dated 11/7/2009).  While the newer version of the SMS.dll file causes no issues and works fine when connecting to vCenter 4.1 Servers, it’s not backward compatible with vCenter 4.0 Update 1.  The result is what you see in the image above, the plugin is disabled and cannot be enabled.

Furthermore, if you investigate your vSphere Client log files at C:\Users\%username%\AppData\Local\VMware\vpx\ you’ll find another similar entry:

System.IO.FileNotFoundException: Could not load file or assembly ‘VpxClientCommon, Version=4.1.0.0, Culture=neutral, PublicKeyToken=7c80a434483c7c50’ or one of its dependencies. The system cannot find the file specified.

Copying the old version of the SMS.dll file into its proper location resolves the plug-in issue when connecting to a vSphere 4.0 Update 1 vCenter Server, this much I tested, however I’m sure it immediately breaks the plug-in when connecting to a vCenter 4.1 Server (I didn’t go so far as to testing this).

Essentially what this boils down to is a VMware vSphere Client bug which is going to bite people who have both vCenter Server 4.0 and 4.1 in their environment, and the respective clients are installed on the same endpoint machine.  I expect to hear about this more as people start their upgrades from vSphere 4.0 to vSphere 4.1.  Some may not even realize they have the issue, after all, I didn’t notice it until I was looking for the Storage Views tab and it wasn’t there.  After lab testing, I did some looking around on the net to see if anyone had discovered or documented this issue and the only hit I came across was a recently started VMware Communities thread, however, there was no posted solution.  The thread does contain a few hints which would have pointed me in the right direction much quicker had I read it ahead of time.  Nonetheless, time spent in the lab is time well spent as far as I’m concerned.  Unfortunately, there’s no fix here I can offer.  This one is on VMware to fix with a new release of the vSphere 4.1 Client.

Update 12/1/10:  VMware has released KB 1024493 to identify this problem and temporarily address the issue with a workaround:

Installing each Client version in different folders does not work. When you install the first Client you are asked where you want to install it. When you install the second Client, you are not asked for a location. Instead, the installer sees that you have already installed a Client and automatically tries and install the second client in the same directory.

To install vSphere Client 4.0 and 4.1 in separate directories:

  1. Install vSphere Client 4.0 in C:\Client4.0.
  2. Copy C:\Client4.0 to an external drive (such as a share or USB).
  3. Uninstall vSphere Client 4.0. Do not skip this step.
  4. Install vSphere Client 4.1 in C:\Client4.1.
  5. Copy the 4.0 Client folder from the external drive to the machine.
  6. Run vpxClient.exe from the 4.0 or 4.1 folder.

I’m expecting a more permanent fix in the future which addresses the .DLL incompatibility in the 4.1 vSphere Client.

Update 2/15/11:  Through some lab testing, it looks as if VMware has resolved this issue with the release of vSphere 4.1 Update 1 although KB 1024493 has not been updated yet to reflect this.  I uninstalled all vSphere Clients, then installed vSphere Client 4.0 Update 1, then installed vSphere Client 4.1 Update 1.  The result is the vCenter Storage Monitoring plug-in is no longer malfunctioning.  The Storage Views tab is also available.  Both of those items are a positive reflection of a resolution.  The Search function is failing in a different way but I’m not convinced it has anything to do with two installed vSphere Clients because it is also failing on a different machine which has only one vSphere Client installed.

vSphere Upgrade Path

October 11th, 2010

Old habits can be hard to break.  Such was the case today when I called out an individual for producing an ESXi 4.0 Update 2 upgrade document without referencing the requirement to upgrade vCenter 4.0 Update 1 to Update 2 first as a prerequisite. 

Up until the release of vSphere 4.0 Update 1 back in November of 2009, the VMware virtual infrastructure upgrade path was such that the vCenter Server was upgraded to the newer release, then the ESX(i) hosts were upgraded afterward.

As shown in the ESX and vCenter Server Compatibility matrix below, beginning with vSphere 4.0 Update 1, ESX(i) hosts can be upgraded ahead of their vCenter Server counterparts.  In fact, VMware allows a radically wider in versioning variation in that vCenter 4.0 (released May 2009, with no update) is compatible with ESX(i) 4.0 Update 2 which was released in June 2010, over a year later.

SnagIt Capture

After being corrected, I recalled hearing of this new compatibility some time back but the bits had fallen off the platter.  For the record, I’m not always right.  I’m fine with being wrong.  It happens plenty enough.  For me, it’s all about the learning.  Retaining the knowledge is an added benefit but isn’t always guaranteed if not used on a regular basis.

This mantra will provide some flexibility which may be needed to upgrade smaller groups of clusters or hosts (say for troubleshooting purposes) without impacting the centralized vCenter Server which in turn would impact the remaining clusters or hosts it manages by way of agent upgrades blasted out to each attached host.

Before you celebrate in the end zone Dallas Cowboys style, do note from the chart above that the upgrade to vSphere 4.1 reverts back to the old methodology of upgrading the vCenter Server first, and the attached ESX(i) hosts afterward.  In other words, ESX(i) 4.1 is ONLY compatible with vCenter Server 4.1.

Go Vikings!

Open in New Window

September 22nd, 2010

The VMware vSphere Client has a right-click menu option for most objects called Open in New Window

For instance, when right-clicking on a cluster object, the Open in New Window menu item appears:

SnagIt Capture

SnagIt Capture

SnagIt Capture

After choosing Open in New Window, a new vSphere Client window indeed appears.  Like many common tasks in the vSphere Client, this procedure has a shortcut key combination (CTRL + ALT + N).  Does this imply this is a commonly used feature? 

It’s not a commonly used feature by me.  To be honest, I didn’t know this feature existed until this week.  I was intrigued and played around with it for about 15 minutes.  First I tried to understand where this feature was presented.  I found it on most objects.  When I saw this, I looked for a way to exploit it.  The result was a rather anticlimactic failure.

This still left me wondering what the use case was for this feature.  There is one which comes to mind but I’m going to keep that to myself for now.  I’d like to hear from you.  Do you use this feature?  What are the use cases?  If you don’t use the feature, can you imagine a use case?  Open the vSphere Client and give it a try.  Be sure to try different infrastructure views.  There’s really no defined set of correct answers here, I’m looking for practical or creative ideas around the feature.

Respond in the comment section below.  The first responder with a relevant or interesting use case will be the winner of a VMware vSphere video training package from Train Signal.

vCenter Server JVM Memory

September 6th, 2010

For those of you who have installed VMware vCenter Server 4.1, have you noticed anything new during the installation process?  A new screen was introduced at the end of the installation wizard for specifying the anticipated size of the virtual infrastructure which the respective vCenter Server would be managing.  There are three choices here: Small, Medium, & Large.  Sorry, no Supersize available yet.  If you require this option, I’m sure VMware wants to talk to you.

SnagIt Capture

The selection you make from the installation wizard not only defines the Maximum Memory Pool value for the Java Virtual Machine, but also the Initial Memory Pool value.  Following is a chart which takes a look at vCenter Server 4.0 & 4.1 JVM Memory Configuration comparisions:

vCenter/JVM Initial Memory Pool Max Memory Pool Thread Stack Size
4.0 128MB 1024MB 1024KB
4.1 Small (<100 hosts, default) 256MB 1024MB 1024KB
4.1 Medium (100-400 hosts) 256MB 2048MB 1024KB
4.1 Large (> 400 hosts) 512MB 4096MB 1024KB

As noted by the table above, in vCenter Server 4.0, the JVM Maximum Memory Pool was configured by default at 1024MB.  The vCenter Server 4.1 installation also defaults to 1024MB (Small <100 hosts) if left unchanged. One other comparison – pay attention to the difference in Initial Memory Pool. By default, vCenter 4.1 uses twice the amount of RAM out of the gate than previous versions.

Although the installation wizard JVM tuning component is new in 4.1, the ability to tune the JVM for vCenter is not.  The Configure Tomcat application has been available in previous versions of vCenter.  Some organizations with growing infrastructures may have been instructed by VMware support to tune the JVM values to overcome a vCenter issue having to do with scaling or some other issue.

SnagIt Capture

SnagIt Capture

Judging from the table, one can assume that the 1024MB value was appropriate for managing less than 100 hosts in vCenter 4.0.  As a point of reference, the Configuration Maximums document states that 300 hosts can be managed by vCenter 4.0.  This would imply that managing 100 hosts or more with vCenter 4.0 requires an adjustment to the out of box setting for the JVM Maximum Memory Pool (change from 1024MB to 2048MB). 

With vCenter 4.1, VMware has improved scaling in terms of the number of hosts a vCenter Server can manage.  The Configuration Maximums document specifies vCenter 4.1 can manage 400 hosts but the table above implies VMware may be preparing to support more than 400 hosts in the near future.  And that’s awesome because vCenter Server sprawl sucks. Period.

So have fun tuning the JVM but before you go, a few parting tips:

  • The Initial Memory Pool value defines the memory footprint (Commit Size) of the Tomcat process when the service is first started.  The Maximum Memory Pool defines the memory footprint which the Tomcat process is allowed to grow to.  Make sure you have sufficient RAM installed in your server to accommodate both of these values.
  • Setting the Initial Memory Pool to a value greater than the Maximum Memory Pool will prevent the Tomcat VJM from starting.  I thought I’d mention that before you spend too much time pulling your hair out.
  • If you would like to learn more about tuning Tomcat, vast resources exist on the internet.  This looks like a good place to start.