Posts Tagged ‘vSphere’

« Older Entries

vSphere 4.0 Quick Start Guide Released on Amazon

November 23rd, 2009

What a great way to kick off the new week – The highly anticipated book, vSphere 4.0 Quick Start Guide: Shortcuts down the path of Virtualization, has arrived at Amazon.com! I look at this new release as the 2nd edition or vSphere edition of RapidApp’s Quick Start Guide to ESX 3.0 which is still available and was a huge success.

The vSphere 4.0 Quick Start Guide was written by a lineup of new authors who are well known rock stars in the virtualization community: Bernie Baker, Thomas Bryant, Duncan Epping, Dave Mischenko, Stewart Radnidge, and Alan Renouf. I obtained a preview copy of this book at VMworld 2009 in San Francisco and I can tell you that this it is absolutely amazing. Nowhere else will you find as much information in such a small and convenient footprint. Its small size allows you to put it in your pocket and take it virtually anywhere: On the plane, on the bus, into a meeting, or into the datacenter. As with the first edition, there are several blank pages in this book which allow you space to write down notes, command line information, configuration maximum changes, information about your environment, helpful URLs, etc. The authors did a great job on this book and considering the cumulative years of experience and combined expertise packed into this book, you can’t beat the price. I don’t think a better value exists. My copy has been traveling with me daily in my laptop bag. I give it two thumbs up.

1 comment »

Posted in Virtualization

Tags:

Old vCenter Server Name Shown In Title Bar; Update Manager Plugin Fails

November 22nd, 2009

I recently rebuilt a vCenter Server on a new Windows host having a different name than the vCenter Server host used previously. Wanting to maintain my existing datacenter configuration and layout, I chose to connect to and preserve the existing SQL database back end.

The installation went well and my existing datacenter configuration was in tact, however, I noticed one anomaly having two symptoms. After establishing a vSphere Client connection to the new vCenter Server named vc40.boche.mcse, the vSphere Client title bar showed jarjar.boche.mcse which was the old vCenter Server name.

Furthermore, the Update Manager plugin was failing to load because it could not establish a connection to jarjar.boche.mcse. I wasn’t surprised a connection could not be made since jarjar was retired and no longer on the network. But why was the legacy vCenter Server name persisting in my new installation?

At first, I thought there was some funky DNS reverse lookup going on but I was able to quickly rule that out when I remembered that I had assigned a new IP address to the new vCenter Server host.

I quickly came to the conclusion that there was a row in the SQL database tattooed with the old vCenter Server name which was showing up in the vSphere Client. With that thought in mind, I used the vSphere Client to access the Administration|vCenter Server Settings menu option.

There it was, under Runtime Settings, the old name of the vCenter Server from the original installation. I was able to simply change the Name from jarjar.boche.mcse to vc40.boche.mcse

Afterwards, the vSphere Client title bar was updated with the correct name of the vCenter Server vc40.boche.mcse. No reboot or recycling of any services needed. The Update Manager plugin had also followed suit, making its connection to the correct vCenter Server name instead of the old one which no longer existed.

Simple stuff but I thought I’d write it up in case anyone else ran into this and was pulling their hair out.

No comments »

Posted in Virtualization

Tags:

Create a 32-bit vCenter DSN on a 64-bit Operating System

November 21st, 2009

As I had pointed out in this blog post, VMware hints that 64-bit may be the future for vCenter Server. I decided that for my upgrade to vCenter 4.0 Update 1 this weekend, I would take the opportunity to rebuild my vCenter server from Windows Server 2003 32-bit to Windows Server 2008 64-bit.

Once the 64-bit base operating system build was complete, I installed the 64-bit Microsoft SQL Server Native Client drivers (downloadable here) since my back end database is Microsoft SQL Server 2005 on a remote server. A key thing to remember about this installation is that it installs both 64-bit and 32-bit DSN drivers.

The next step is to create the vCenter ODBC DSNs. Although vCenter Server runs on 64-bit operating systems, it currently requires a 32-bit ODBC DSN. This is important to remember because the Windows Start Menu launches the 64-bit ODBC DSN tool, not the 32-bit version I needed.  The vCenter Server (and Update Manager) installation will not complete without a 32-bit DSN.

To create a 32-bit DSN on a 64-bit operating system, run the following executable:

[WindowsDir]\SysWOW64\odbcad32.exe

Once the utility opens, you’ll be greeted by all the legacy 32-bit ODBC DSNs you’ve likely seen for years working with tiered Windows platforms. If using Microsoft SQL Server 2005 like me, be sure to select the SQL Native Client driver towards the bottom of the list, and not Driver da Microsoft para arquivos texto highlighted below:

Proceed with the creation of the vCenter Server and Update Manager ODBC DSNs and complete the vCenter Server and Update Manager installations.

This information and much more can be found in the ESX and vCenter Server Installation Guide, page 73.

2 comments »

Posted in Virtualization

Tags:

64-bit vCenter Server Coming? VMFS-2 Support Going Away?

November 20th, 2009

Looking at the VMware vCenter 4.0 Update 1 Release Notes, it appears we may be looking at 64-bit only versions of vCenter Server in the future:

“Future releases of VMware vCenter Server might not support installation on 32-bit Windows operating systems. VMware recommends installing vCenter Server on a 64-bit Windows operating system. If you have VirtualCenter 2.x installed, see the vSphere Upgrade Guide for instructions on installing vCenter Server on a 64-bit operating system and preserving your VirtualCenter database.”

I’m hoping this won’t be a deal breaker for anyone as we should all have 64-bit hardware in the datacenter by now. However, we may be temporarily inconvenienced with Windows Server platform upgrades from 32 to 64-bit.

In the same document, I also noticed verbiage about VMFS-2 support going away in future vSphere releases. If you’re not completely rid of ESX2 and VMFS-2 in your environment by now, I’d start planning on it soon.

2 comments »

Posted in Virtualization

Tags:

Tame Electrical and Heating Costs with CPU Power Management

November 11th, 2009

A casual Twitter tweet about my power savings through the use of VMware Distributed Power Management (DPM) found its way to VMware Senior Product Manager for DPM, Ulana Legedza, and Andrei Dorofeev. Ulana was interested in learning more about my situation. I explained how VMware DPM had evaluated workloads between two clustered vSphere hosts in my home lab, and proceeded to shut down one of the hosts for most of the month of October, saving me more than $50 on my energy bill.

Ulana and Andrei took the conversation to the next level and asked me if I was using vSphere’s Advanced CPU Power Management feature (See vSphere Resource Management Guide page 22). I was not, in fact I was unaware of its existence. Power Management is a new feature in ESX(i)4 available to processors supporting Enhanced Intel SpeedStep or Enhanced AMD PowerNow! power management technologies. To quote the .PDF article:

“To improve CPU power efficiency, you can configure your ESX/ESXi hosts to dynamically switch CPU frequencies based on workload demands. This type of power management is called Dynamic Voltage and Frequency Scaling (DVFS). It uses processor performance states (P-states) made available to the VMkernel through an ACPI interface.”

A quick look at the Quad Core AMD Opteron 2356 processors in my HP DL385 G2 showed they support Enhanced AMD PowerNow! Power Management Technology:

There are two steps to enabling this power management feature. The first step is to ensure it is enabled in the server BIOS. On an HP DL385 G2, CPU power management is enabled by default. In this particular server model, it is configured via the BIOS by hitting <F9> at the end of the POST (would require a reboot obviously)

A slightly easier method might be to verify and/or configure the policy through HP’s out of band (OOB) iLO 2, however, a reboot will be requested by the iLO 2 for a policy change to take effect. On an HP server, configure for OS Control mode, but again, this appears to be the default for the HP DL385 G2 so hopefully no reboot is required for you to implement this power saving measure in your environment:

After enabling power management in the BIOS, the second step is to modify the Power Management Policy on each ESX(i) host from the default of static to dynamic. The definitions of these two settings can be found in the .PDF linked above and are as follows:

static – The default. The VMkernel can detect power management features available on the host but does not actively use them unless requested by the BIOS for power capping or thermal events.

dynamic – The VMkernel optimizes each CPU’s frequency to match demand in order to improve power efficiency but not affect performance. When CPU demand increases, this policy setting ensures that CPU frequencies also increase.

You might be asking yourself by this point “Ok, this is nice, but what’s the trade off?” Note the wording in the dynamic definition above “improves power efficiency but does not affect performance”. This is a win/win configuration change!

This step can be performed one of a few ways on each host (again, no reboot required for this change):

  1. Using the vSphere Client, change the Advanced host setting Power.CpuPolicy from static to dynamic
  2. Scriptable: Via the ESX service console, PuTTY, or script, issue the command esxcfg-advcfg -s dynamic /Power/CpuPolicy

The impact on my home lab was quite visible. After 12 hours, the blue area in the following 24 hour graph reflects average electrical consumption was reduced from an average 337 Watts down to 292 Watts. All things being equal and CPU loads balanced by DRS, that’s a reduction in energy consumption of over 13% per host:

An alternate graph shows Btu output dropped from 1,135 Btu to about 1,000 Btu. All things being equal, a reduction of about 135 Btu per host:

A Btu is heat – explained more at wiseGEEK’s What is a Btu? Heat is a byproduct of technology in the datacenter and in most cases is viewed as overhead expense because it requires cooling (additional costs) to maintain optimal operating conditions for the equipment running in the environment. If we can eliminate heat, we eliminate the associated cost of removing the heat. This is known as cost avoidance.

Eliminating heat is as much of an interest to me as reducing my energy bill. The excessive heat generated in the basement eventually finds its way upstairs causing the rest of the house to be a little uncomfortable. The air conditioner in my home wasn’t manufactured to handle the excessive heat. Now, I live in the midwest where we have some frigid winters. Heat in the home is welcomed during the winter months. I could turn off CPU Power Management raising the Btu index as well as my energy bill, in favor of reducing my natural gas heating bill. I don’t know which is more expensive. This could be a great experiment for the January/February time frame.

In summary, we can attack operating costs from two sides by using VMware CPU Power Management:

  1. Reduction in excess electricity used by idle CPU cycles
  2. Reduction in cooling costs by reducing Btu output

I’m excited to see what next month’s energy bill looks like.

Update 11-17-09:  I was just made aware that Simon Seagrave wrote an earlier article on CPU power management here.  Sorry Simon, I was unaware of your article and I did not intentionally copy your topic.  Your article covered the topic well.  I hope we’re still friends :)

24 comments »

Posted in Virtualization

Tags:

Lab Manager 4 Installation Fails With vSphere VMXNET 3 NIC

November 7th, 2009

A few of the networking requirements for installing a VMware Lab Manager server are:

  1. At least one network card
  2. A static TCP/IP configuration (no DHCP)

Failure to meet the above requirements will result in error message # 5014 during the “Valid NIC Requirement” prerequisite check:

Lab Manager servers make fine virtualization candidates, therefore, it makes sense to deploy them as VMs on existing VMware virtual infrastructure so that they can take advantage of all the benefits VMware brings into the datacenter.

I ran into a new issue installing Lab Manager 4 in a vSphere VM which I configured with a VMXNET 3 virtual NIC. Already aware of the networking requirements, I had configured the virtual NIC with a static TCP/IP address, subnet mask, default gateway, and DNS servers. However, I was surprised to find out that my installation was failing the Valid NIC Requirement prerequisite.

So I resorted to what any certified professional would in this situation: GOOGLE. A quick search revealed scarce results but thankfully one solution. In this VMTN forums thread, a short discussion reveals that the VMXNET 3 virtual NIC is unexpectedly not compatible with the Lab Manager 4 installation prerequisites check. VMTN user MLaskowski012 explains:

“Talked to support and they said they are seeing the same issues. I guess nobody tested LabManager4 with the new hardware. BUT I think I figured out the trick. In device manager under NIC / Advanced if you change the Speed / Duplex from Auto Negotiation 10GB to 1GB Full, run the pre-check it will pass. Then right after you finish the install you can switch back to Auto or 10Gb. Not sure if there are any issues pass that…”

Viola! The trick works. Thank you MLaskowski012 for doing the legwork on this one. Unfortunately, no KB article from VMware yet on this (that I could find), but once again, as it has millions of times in the past, the VMTN community has fulfilled one of its primary purposes: technical support for the community, by the community.

Update 11/8/09: Via lab testing, the same failure and workaround applies to Lab Manager 3 installations with a VMXNET 3 virtual network adapter as well.

2 comments »

Posted in Virtualization

Tags:

TrainSignal vSphere Training DVD 1 Completed

October 23rd, 2009

This evening I finished viewing the first of three TrainSignal vSphere Training DVDs authored by VCP and CCIE David Davis. Having viewed TrainSignal’s last VMware Virtual Infrastructure training on VI3, I knew I was in for some good stuff.

DVD 1 starts off with introductions to the video’s instructor as well as a hypothetical company which is used as a focus and discussion point throughout the video series. Practical application of technologies to a role played scenario, the Wired Brain Coffee Company in this case, serves as positive reinforcement to the lessons being taught and is an effective method for knowledge retention, especially if the student is following along and working hands on in their own lab through the examples.

The video then sets a beginner’s pace as it covers VMware certification, virtualization basics. Moving on, it compares and contrasts VMware, Microsoft, and Citrix hypervisors. Beyond this comparison, the focus from here on out is on VMware products where a closer look is taken at the different components and tiers of vSphere.

Half way through the DVD, we’re finally to the point where we’re installing and configuring the vSphere products. One valuable offering from the video is a lesson describing the steps needed to install ESX and ESXi in VMware Workstation. This is what is called a nested hypervisor – an ESX(i) type 1 (bare metal) hypervisor running on top of a VMware Workstation type 2 (hosted) hypervisor. Nested hypervisors are not supported in production environments but they are quite helpful in lab, test, and portable environments.

Towards the end, lesson 17 provides a nice demonstration of a VMware Tools installation in a Linux guest operating system which isn’t nearly as straight forward as a VMware Tools on Windows installation. The last two lessons begin touching on some of the new advanced features that vSphere offers: Hot Add/Hot Plug virtual hardware and Host Profiles.

Thus far my feeling is this training is geared towards the beginner to intermediate level. I’m looking forward to DVD 2 where the instructor dives into more of the advanced design, configuration, and operational topics of VMware vSphere. I’ve attended VMware’s vSphere What’s New (2 day) and VMware’s vSphere Quick Start (5 day) classes. With approximately 150 new features making their debut in vSphere, I’ve yet to see anyone cover them all – that would be a tall order.

DVD 1 Lessons:

  1. Meet Your Instructor
  2. Our Scenario with the Wired Brain Coffee Company
  3. VMware Certification – Preparing for the VCP and VCDX
  4. Introduction to Virtualization
  5. Virtualization Products Compared
  6. VMware ESXi4 Free Edition for the SMB
  7. VMware vSphere 4 and ESX Essentials
  8. vSphere Management Options
  9. Installing the VMware vSphere Client
  10. Navigating vSphere Using the vSphere Client
  11. Running VMware ESX 4 in Workstation
  12. Installing VMware ESX 4
  13. Installing VMware ESXi Version 4
  14. Installing VMware vCenter 4
  15. vCenter4 – Configuring Your New Virtual Infrastructure
  16. Creating & Modifying Virtual Guest Machines
  17. Installing and Configuring VMware Tools
  18. Adding Virtual Machine Hardware with vSphere Hot Plug
  19. Using vSphere Host Profiles

6 comments »

Posted in Virtualization

Tags:

After enabling FT on a VM – subtleties to expect

September 16th, 2009

While using VMware vSphere, you may encounter a situation where you cannot edit the memory resource settings (shares, reservations, and limits) for a particular VM on the resources tab. The memory resource settings section will be completely grayed out. In addition, a label will clearly state “Memory resources-cannot edit” as shown below:

In this particular instance, the underlying cause for this condition is VMware Fault Tolerance (FT) has been enabled on the FT “primary” VM. The fact that the memory resource settings cannot be modified is by design and is used as a means to help guarantee the FT “secondary” VM stays in vLockstep with the primary. What has actually happened is that when FT was enabled on the VM, a memory reservation was set equal to the amount of memory configured for the VM. This eliminates VMkernel swap file for the VM managed by the host in all cases, not just for FT enabled VMs.

What other subtle changes can you expect when you enable VMware Fault Tolerance (FT) on a VM?

DRS will be disabled for the FT enabled primary VM, although it may be VMotioned manually in cases where maintenance needs to be performed on the ESX(i) host. FT secondaries may also be migrated by right clicking on the FT primary VM and choosing the Fault Tolerance menu item to “Migrate Secondary”:

Thin provisioned disks will be converted to a Thick type:

A FT “secondary” VM will be created on another host in the cluster which will consume CPU and memory on the secondary host. It will share VM storage with the FT “primary”. VM networking is disabled on the FT “secondary” to eliminate the obvious problem of a duplicate machine on the network, however, other considerable host based network traffic will be consumed for two purposes:

  1. Initial creation of the FT “secondary” – dedicated VMotion network is used
  2. Continuous FT logging traffic – dedicated FT logging network is used

If hardware MMU feature exists in the host CPUs (AMD RVI/Intel EPT), the feature is disabled in the VM. This will force a power off of the VM before FT can be enabled.

Storage vMotion will be disabled for the FT enabled VM.

The hypervisor may slow down execution of the FT “primary” VM if the FT “secondary” is not able to keep pace with the FT “primary” using vLockstep technology.

Snapshotting functionality will be disabled. Furthermore and maybe more importantly, backups requiring snapshot technology won’t work.

Virtual hardware that is not compatible with FT will be disabled (ie. USB, Sound, etc.)

vSMP (multiple CPUs) in the VM is not supported. FT can’t be enabled.

Physical RDMs in the VM are not supported. FT can’t be enabled.

For more information on VMware Fault Tolerance, see VMware vSphere™ 4 Fault Tolerance: Architecture and Performance, VMware vSphere Availability Guide, and Xtravirt’s Disaster Recovery and VMware vSphere 4.0 Fault Tolerance whitepaper,

3 comments »

Posted in Virtualization

Tags:

vSphere 4 Reference Card now available

August 10th, 2009

Forbes Guthrie has done it again! His wildly successful VI3 reference card is now available in vSphere format.  Head over to his site, vReference, and download your copy today.  Be sure to thank him for his hard work! I for one appreciate all that he does. Thanks Forbes and I look forward to meeting you in a few weeks.

8-10-2009 11-05-11 AM

1 comment »

Posted in Virtualization

Tags:

2nd Batch of vSphere Patches Arrive (Critical)

August 7th, 2009

The list is short, but critical, and impacts both ESX 4.0 as well as ESXi 4.0 hosts.

ESXi400-200907401-BG – Critical http://kb.vmware.com/kb/1013016

This patch fixes the following issue – If the RAID controller battery backup unit is completely discharged after a shutdown, or if a locally attached disk is removed and not returned to the system, data corruption might occur because the RAID controller cache is not cleared while shutting down the server.  See KB 1013016 for more details.  NOTE:  Cisco Nexus 1000v customers using VMware Update Manager to patch ESXi 4.0 should add an additional patch download URL as described in KB 1013134

ESX400-200907401-BG – Critical http://kb.vmware.com/kb/1013026

This patch fixes the following issue – If the RAID controller battery backup unit is completely discharged after a shutdown, or if a locally attached disk is removed and not returned to the system, data corruption might occur because the RAID controller cache is not cleared while shutting down the server.  See KB 1013026 for more details.  NOTE:  Cisco Nexus 1000v customers using VMware Update Manager to patch ESXi 4.0 should add an additional patch download URL as described in KB 1013134

No comments »

Posted in Virtualization

Tags:

Not All FT Compatible CPUs Are Created Equal

July 10th, 2009

Hopefully you are aware that to enable VMware vSphere’s FT (Fault Tolerance), you need FT compatible CPUs from Intel or AMD.  VMware KB article 1008027 Processors and guest operating systems that support VMware Fault Tolerance outlines both the Intel and AMD CPU requirements to use FT.  I had read this article months ago and on that basis I purchased FT compatible AMD Opteron 2356 Barcelona Quad Core processor upgrades for the HP DL385 G2 servers in my lab.

While trying to enable FT on powered on VMs in the lab, I ran into issues.  The following error was thrown in the vSphere Client:

“The Fault Tolerance configuration of the entity <vm name> has an issue: The virtual machine’s current configuration does not support Fault Tolerance.”

In the vCenter Server log, the more verbose error displayed was reason = “replayNotSupported”

7-10-2009 8-55-47 PM

Oddly enough, I was able to configure FT on powered off VMs.

What I hadn’t noticed, or possibly what didn’t exist in earlier versions of this KB article was a chart at the bottom of the page with more verbose information that explained specific FT behavior based on the processor architecture.  My AMD Barcelona processors do in fact support FT, however, the chart confirms that with my processors, the VMs must be powered off first before enabling FT, whereas Intel Xeon 45nm Core 2 processors I’ve worked with in other labs allow FT to be enable while a VM is running live.  Also note in the chart below that there are FT support guidelines for specific Operating Systems as well.  For instance, a Windows 2000 VM may never be FT enabled while running, and Windows 2000 is not an FT compatible guest OS on my AMD Barcelona processors.

Following is a copy and paste of the KB article above with the FT support specifics:

Processors and guest operating systems that support VMware Fault Tolerance

Details

VMware Fault Tolerance (FT) requires specific processors and guest operating systems.

Solution

Processors

VMware collaborated with AMD and Intel in providing an efficient VMware FT capability on modern x86 processors. The collaboration required changes in both the performance counter architecture and virtualization hardware assists of both Intel and AMD. These changes could only be included in recent processors from both vendors: 3rd-Generation AMD Opteron based on the AMD Barcelona, Budapest and Shanghai processor families; and Intel Xeon processors based on the Penryn and Nehalem microarchitectures and their successors.

Download the VMware SiteSurvey (http://www.vmware.com/download/shared_utilities.html) utility to check if your configuration can run VMware FT.

For VMware FT to be supported, the servers that host the virtual machines must each use a supported processor from the same category as documented below:

Intel Xeon based on 45nm Core 2 Microarchitecture Category:

    • 3100 Series
    • 3300 Series
    • 5200 Series (DP)
    • 5400 Series
    • 7400 Series

Intel Xeon based on Core i7 Microarchitecture Category:

    • 5500 Series

AMD 3rd Generation Opteron Category:

    • 1300 Series
    • 2300 Series (DP)
    • 8300 Series (MP)

Guest Operating Systems The following table displays guest operating system support for VMware FT. For specific guest operating system version information, see the Guest Operating System Installation Guide at http://www.vmware.com/pdf/GuestOS_guide.pdf.   The following values appear in the table:

  • Yes – Virtual machine can be FT-enabled while powered on.
  • Yes/Off - Virtual machine must be powered off before FT is enabled
  • No – Not supported by VMware FT.
Guest Operating System Fault Tolerance Support with Intel Xeon Based on 45nm Core 2 Microarchitecture Fault Tolerance Support with Intel Xeon Based on Core i7 Microarchitecture Fault Tolerance Support with AMD 3rd Generation Opteron
Windows Server 2008 Yes Yes/Off Yes/Off
Windows Vista Yes Yes/Off Yes/Off
Windows Server 2003 (64 bit) Yes Yes/Off Yes/Off
Windows Server 2003 (32 bit) Yes Yes/Off Yes/Off (Requires Service Pack 2 or greater)
Windows XP (64 bit) Yes Yes/Off Yes/Off
Windows XP (32 bit) Yes Yes/Off No
Windows 2000 Yes/Off Yes/Off No
Windows NT 4.0 Yes/Off Yes/Off No
Linux (all ESX-supported distributions) Yes Yes/Off Yes/Off
Netware Server Yes/Off Yes/Off Yes/Off
Solaris 10 (64-bit) Yes Yes/Off Yes/Off (Requires Solaris U1)
Solaris 10 (32-bit) Yes Yes/Off No
FreeBSD (all ESX-supported distributions) Yes Yes/Off Yes/Off

Note: System vendors are certifying that their systems work with FT. You can find details on the FT-certified systems at http://www.vmware.com/resources/compatibility. More systems are being certified all the time, so check back if your platform is not currently listed.

You can also check processor, operating system, and virtual machine configuration compliance with FT by downloading and running the VMware SiteSurvey utility from http://www.vmware.com/download/shared_utilities.html. It highlights compliance issues and describes how to correct them.

3 comments »

Posted in Virtualization

Tags:

FT Problem Decoder Chart

July 10th, 2009

Receiving errors while trying to configure FT (Fault Tolerance) on a VM and stumped as to the reason why? This may help. Take a look at the vCenter server log in your vSphere Client and find the entry when the FT error occurred (the vCenter server log lists events in chronological order from oldest to newest, be sure to choose the correct log file as there are several to choose from). More specifically, look for the line reason = “blah blah blah”. In this case, the reason is “replayNotSupported”.

7-10-2009 8-55-47 PM

Next, open your web browser and surf to this vSphere API 4.0 link titled “Enum – VmFaultToleranceConfigIssueReasonForIssue”. This is a cross reference chart that lists common sense explanations for the “reason” code above. According to the chart, “replayNotSupported” is explained as:

“It is not possible to turn on Fault Tolerance on this powered-on VM. The support for record/replay should be enabled or Fault Tolerance turned on, when this VM is powered off.”

The root cause for the example shown above is the processors support FT, however, not while the VM is powered on. For the AMD Opteron 2356 Barcelona processors, FT is supported but the VMs must be in a powered off state to enable FT, which leads me to my next blog entry…

Here is a copy of the chart:

Name Description
ftSecondaryVm The virtual machine is a fault tolerance secondary virtual machine
ftUnsupportedHardware The host ftSupported flag is not set because of hardware issues
ftUnsupportedProduct The host ftSupported flag is not set because of it is a VMware Server 2.0
haNotEnabled HA is not enabled on the cluster
hasLocalDisk The virtual machine has one or more disks on local datastore
hasSnapshots The virtual machine has one or more snapshots
hostInactive The host is not active
missingFTLoggingNic FT logging nic is not configured on the host
missingVMotionNic No VMotion license or VMotion nic is not configured on the host
moreThanOneSecondary There is already a secondary virtual machine for the primary virtual machine
multipleVCPU The virtual machine has more than one virtual CPU
noConfig No configuration information is available for the virtual machine
recordReplayNotSupported The virtual machine does not support record/replay. Vm::Capability.RecordReplaySupported is false.
replayNotSupported It is not possible to turn on Fault Tolerance on this powered-on VM. The support for record/replay should be enabled or Fault Tolerance turned on, when this VM is powered off.
templateVm The virtual machine is a template
thinDisk The virtual machine has thin provisioned disks
verifySSLCertificateFlagNotSet The “check host certificate” flag is not set

1 comment »

Posted in Virtualization

Tags:

First vSphere Patches Released by VMware

July 10th, 2009

Approximately six weeks after the vSphere launch, the first batch of ESX/ESXi 4.0 patches have downloaded by vSphere Update Manager. I was notified this morning at 3am via an Email from vSphere Update Manager. Here is the patch list:

The number of patch definitions downloaded:

10 critical

16 total

ESX:

ID: ESX400-200906401-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates VMX

ID: ESX400-200906402-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates ESX Scripts

ID: ESX400-200906403-BG Impact: HostGeneral Release date: 2009-07-09 Products: esx 4.0.0
Updates VMware Tools

ID: ESX400-200906404-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates CIM

ID: ESX400-200906405-SG Impact: HostSecurity Release date: 2009-07-09 Products: esx 4.0.0
Updates krb5 and pam_krb5

ID: ESX400-200906406-SG Impact: HostSecurity Release date: 2009-07-09 Products: esx 4.0.0
Updates sudo

ID: ESX400-200906407-SG Impact: HostSecurity Release date: 2009-07-09 Products: esx 4.0.0
Updates curl

ID: ESX400-200906408-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates SCSI Driver for QLogic FC

ID: ESX400-200906409-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates LSI storelib Library

ID: ESX400-200906410-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates hostd

ID: ESX400-200906411-SG Impact: HostSecurity Release date: 2009-07-09 Products: esx 4.0.0
Updates udev

ID: ESX400-200906412-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates esxupdate

ID: ESX400-200906413-BG Impact: Critical Release date: 2009-07-09 Products: esx 4.0.0
Updates vmkernel iSCSI Driver

ESXi:

ID: ESXi400-200906401-BG Impact: Critical Release date: 2009-07-09 Products: embeddedEsx 4.0.0
Updates Firmware

ID: ESXi400-200906402-BG Impact: Critical Release date: 2009-07-09 Products: embeddedEsx 4.0.0
Updates Tools

ID: VEM400-200906002-BG Impact: HostGeneral Release date: 2009-07-09 Products: embeddedEsx 4.0.0
Cisco Nexus 1000V VEM

2 comments »

Posted in Virtualization

Tags:

vSphere Virtual Machine Performance Counters Integration into Perfmon

July 8th, 2009

VMware introduced the VMware Descheduled Time Accounting Service as a new VMware Tools component in ESX 3.0. The goal was to account for inconsistent CPU cycles allocated to the guest VM by the VMkernel to provide accurate performance statistics using standard performance monitoring tools within the guest VM. Although the service was not installed and enabled with VMware Tools by default, nor did it ever escape the bonds of experimental support status, I found the service to be both stable and reliable and it was a standard installation component in one of my production datacenters. One caveat was that the service only supported uniprocessor guest VMs having a single vCPU.

The VMware Descheduled Time Accounting Service was deprecated in VMware vSphere. More accurately, it was sort of replaced by a new vSphere feature called Virtual Machine Performance Counters (Integrated into Perfmon). To quote VMware:

“Virtual Machine Performance Counters Integration into Perfmon — vSphere 4.0 introduces the integration of virtual machine performance counters such as CPU and memory into Perfmon for Microsoft Windows guest operating systems when VMware Tools is installed. With this feature, virtual machine owners can do accurate performance analysis within the guest operating system. See the vSphere Client Online Help.”

The vSphere Client Online Help has this to say about Virtual Machine Performance:

“In a virtualized environment, physical resources are shared among multiple virtual machines. Some virtualization processes dynamically allocate available resources depending on the status, or utilization rates, of virtual machines in the environment. This can make obtaining accurate information about the resource utilization (CPU utilization, in particular) of individual virtual machines, or applications running within virtual machines, difficult. VMware now provides virtual machine-specific performance counter libraries for the Windows Performance utility. Application administrators can view accurate virtual machine resource utilization statistics from within the guest operating system’s Windows Performance utility.”

Did you notice the explicit statement about Perfmon? Perfmon is Microsoft Windows Performance Monitor or perfmon.exe for short. Whereas the legacy VMware Descheduled Time Accounting Service supported both Windows and Linux guest VMs, its successor currently supports Perfmon ala Windows guest VMs only. It seems we’ve gone backwards in functionality from a Linux guest VM perspective. Another pie in the face for shops with Linux guest VMs.

Rant…

I understand that Windows guest VMs are the low hanging fruit for software development and features, but VMware needs to make sure some love is spread through the land of Linux as well. Folks with Linux shops are still struggling with basic concepts such as Linux guest customization as well as flexibility and automation of VMware Tools installation in the Linux guest OS. If VMware is going to tout their support for Linux guest VMs, I’d like to see more of a commitment than what is currently being offered. There’s more to owning a virtualized infrastructure than powering on instances on top of a hypervisor. Building it is the easy part. Managing it can be much more difficult without the right tools. Flexibility and ease with in the management tools is critical, especially as virtual infrastructures grow.

/Rant…

So, taking a look at a VMware vSphere Windows VM with current VMware Tools, I launched Perfmon. The installation of VMware Tools installs two new Performance Objects along with various associated counters:

  • VM Memory
    • Memory Active in MB
    • Memory Ballooned in MB
    • Memory Limit in MB
    • Memory Mapped in MB
    • Memory Overhead in MB
    • Memory Reservation in MB
    • Memory Shared in MB
    • Memory Shared Saved in MB
    • Memory Shares
    • Memory Swapped in MB
    • Memory Used in MB
  • VM Processor
    • % Processor Time
    • Effective VM Speed in MHz
    • Host processor speed in MHz
    • Limit in MHz
    • Reservation in MHz
    • Shares

Observing some of the counter names, it’s interesting to see that VMware has given us direct insight into the hypervisor resource configuration settings via Performance Monitor from inside the guest VM. While this may be useful for VI Administrators who manage both the VI as well as the guest operating systems, it may be disservice to VI Administrators in environments where guest OS administration is delegated to another support group. The reason why I say this is that some of these new counters disclose an “over commit” or “thin provisioning” of virtual hardware resources which I’d rather not reveal to other supports groups. What they don’t know won’t hurt them. Revealing some of the tools in our bag of virtualization tricks may bring about difficult discussions we don’t really want to get into or perhaps provoke the finger of blame to be perpetually pointed in our direction whenever a guest OS problem is encountered.

I’ve grabbed a few screen shots from my lab which show the disparity between native Perfmon metrics and the new vSphere Virtual Machine Performance Counters. In this example, I compare %Processor Time from the Perfmon native Processor object against the %Processor Time from the VM Processor object which was injected into the VM during the vSphere VMware Tools installation. It’s interesting to note, and you should be able to clearly see it in the graph, that the VM Processor %Processor time is consistently double that of the Perfmon native Processor % Processor Time counter. Consider this when you are providing performance information for a guest VM or one of its applications. If you choose the native Perfmon counter, you could be reporting performance data with 100% margin of error as shown in the case below. This is significant and if used for capacity planning purposes could lead to all sorts of problems.

7-8-2009 9-15-20 PM

7-8-2009 10-17-02 PM

One other important item to note is that you may recall I said towards the beginning that the legacy VMware Descheduled Time Accounting Service only supported uniprocessor VMs. The same appears to be true for the new vSphere Virtual Machine Performance Counters. In the lab I took a single CPU VM which had the vSphere Virtual Machine Performance Counters, and I adjusted the vCPU count to 4. After powering on with the new vCPU count, the vSphere Virtual Machine Performance Counters disappeared from the pulldown list. VMware needs to address this shortcoming. Performance statistics on vSMP VMs are just as important, if not more important, than performance statistics on uniprocessor VMs. vSMP VM resource utilization needs to be watched more closely for vSMP justification purposes.

So VMware, in summary, here is what needs work with vSphere Virtual Machine Performance Counters:

  1. Must support vSMP VMs
  2. Must support Linux VMs
  3. Support for Solaris VMs would also be nice
  4. More objects: VM Disk and VM Networking
Update: On Friday July 11th, 2009, I received the following email response from Praveen Kannan, a VMware Product Manager. Praveen has given me permission to reprint the response here. It is an encouraging read:

Hi Jason,

I read your recent blog post on the Perfmon integration in vSphere 4.0. I’m the product manager for the feature and wanted to reach out to on your findings and feedback regarding the feature.

First off, thanks for the detailed post on the intricacies of the feature and the screenshots. I think this post would be very helpful to the community! Much appreciated…

1) note on vmdesched

We’ve deprecated vmdesched in vSphere 4.0 because it was primarily an experimental feature that we didn’t recommend putting in production. More importantly, vmdesched adds overhead to the guest and is not compatible with some of the newer kernels out there and so the Perfmon integration is our answer to improve on the current state and provide accurate CPU accounting to VM owners that can be deployed in production and is integrated well with VMware Tools for out-of-box functionality.

2) Linux support for accurate counters

The Perfmon integration in vSphere 4.0 leverages the guest SDK API to get to the accurate counters from the hypervisor and that is available on Linux GOS as well. All you need is to have the VMware Tools installed to get access to the guest SDK interface. We couldn’t provide something like Perfmon on Linux since there aren’t many broadly used tools/APIs that we can standardize on.

There are some discussions internally to solve the accounting issue on Linux guests in a much simplified manner but I can’t go into the specific details at this time. Rest assured, I can tell you that we are looking into the problem for Linux workloads.

On a side note, the Perfmon implementation exposes the two new counter groups through WMI (you can almost think of the Perfmon integration as a WMI provider that sits on top of the guest SDK interface and provide access to the counters). What this means is any in-guest agent, benchmarking, reporting tool etc. can quickly adapt to use these “accurate” counters using WMI

So for Linux guests, you can refer to the guest SDK documentation on how someone can modify their Linux agents, tools etc. to talk to the “accurate” counters. The programming guide for vSphere guest SDK 4.0 is available at http://www.vmware.com/support/developer/guest-sdk/. The list of available perf counters is in Page 11 of the PDF (Accessor functions for VM data).

You can in fact use the older 3.5 version of the guest SDK API as well if you want to implement something that works with existing VI3 environments (yes, this SDK has been around for a while!). The only difference is that the vSphere version of the API has a few extra counters but you will get access to the important counters such as CPU utilization in the older API itself.

3) over commit, thin provisioning counters

Interesting feedback that I’ll take back to engineering :) This is something that we need to think about for sure

4) uni-processor Perfmon?

I’m really surprised with your observations after moving to a 4 vCPUs. Not sure what’s going on but AFAIK, we report the _Total (aggregate) of all CPU utilization in one metric in the “VM Processor” counter group in Perfmon. What that means is regardless of how many CPUs in-guest, we do provide the _Total of CPU Utilization. Maybe you may have run into a bug. I’ll check with engineering on this anyways to confirm my understanding.

Just so you know we have a “standalone” version of the Perfmon tool that works with existing VI 3.5 environments. We’ve posted details about this experimental tool and the binaries on our performance blog here:

http://communities.vmware.com/blogs/drummonds/2009/06/18/using-perfmon-for-accurate-esx-performance-counters

The reason I mentioned the standalone version is because on my test box running 3.5 with the standalone version of Perfmon, I was able to see the _Total on a 2 vCPU VM. I haven’t yet tested your findings on a vSphere test box yet but I look into it…

So to help us investigate this, could you please do the following?

a. re-install VMware tools on a test Windows VM after switching to 4 vCPUs and check if the problem is reproducible

b. if you have the 3.5 version of VMware tools running on a VI3 setup, download the standalone version of the Perfmon tool and install it on a Windows VM and check if the 4-vCPU problem is observed. I haven’t tested the same standalone version of Perfmon on a vSphere 4.0 setup (with 4.0 version of the tools) but I wouldn’t be surprised if the standalone version does work. You may want to snapshot the VM before you attempt this though so you can rollback.

5) more counters such as disk and networking

Some background…our main focus in 4.0 was to solve the immediate customer pain-point, namely the CPU accounting issue inside the guest for VM owners. Also, what we heard is that VI admins didn’t want to give out VI client access to VM owners whenever they wanted to look at “accurate” counters for CPU utilization. In fact, the memory counters in Perfmon were sort of a bonus since it was already available in the guest SDK interface :)

Importantly, other counters when measured inside the guest such as Memory, Disk and Network don’t really suffer from accounting problems (i.e. they are accurate) as compared to CPU utilization numbers captured over a period of time (which may be accounted different due to the scheduling and de-scheduling the hypervisor does). So the numbers for Disk, Memory and Network when captured inside the Windows guest will be the same as the VI client.

However, I do recognize that as more and more customers start using this integration, there will soon be a need for providing disk and network counters as well. This is definitely on my radar to address in a future release.

Hope the information I provided helps in better understanding the Perfmon integration in vSphere 4.0 and also answer some of your questions in the blog post.

Looking forward to your findings with the 4 vCPU VMs. LMK if you have any questions in the interim.

P.S: Do feel free to use the information discussed here for your blog where you deem useful…

Have a good weekend…

–
Praveen Kannan
Product Manager
VMware, Inc.

After some more investigation in another test VM, I replied to Praveen with the following information:

Praveen,

In my previous test, I had a 1 vCPU Windows Server 2003 VM. The VM Memory
and VM Processor objects were listed in the pulldown lits in perfmon. After
upgrading the VM to 4 vCPUs, the VM Memory and VM Processor objects were no
longer listed in the pulldown list in perfmon. So you see, the objects were
not available thus the counters (including _Total) were not available.

Today, I deployed a 1 vCPU Windows Server 2003 VM from a 1 vCPU template.
When I ran perfrmon, the VM Memory and VM CPU objects were missing (VMware
Tools was up to date). I closed perfmon and reopened it. Then the 2 VM
objects were there.

Then I upgraded the VM to a 4 vCPU VM. I ran perfmon and both the VM
objects were there.

Following that, I encountered more problems. I was able to choose the VM Processor object, but the counters for the object were all missing. Definitely a bug somewhere with these. Please advise.


6 comments »

Posted in Virtualization

Tags:

VMware ESX Configuration Maximums Comparison Matrix

July 7th, 2009

Some of the this blog’s readers may know that one of my favorite VMware documents is the Configuration Maximums.  I’ve mentioned it a few times.  Sid Smith over at Daily Hypervisor put together a very nice consolidation of configuration maximums for ESX3, ESX3.5, and vSphere (ESX4.0).

The .PDF is assembled in a matrix format making comparison of the various VMware hypervisors easy to differentiate, compare, and contrast.  Sid, you read my mind.  Well done.  Thank you for the early birthday/Christmas present!

1 comment »

Posted in Virtualization

Tags: