Archive for the ‘Virtualization’ category

Happy Easter!

April 4th, 2010

Allison vEGG from Jason Boche on Vimeo.

VMware Disks Moving Back To .DSK File Name Format

April 1st, 2010

VMware Administrators who use VMware Update Manager (and I highly recommend its use), will have noticed that 10 new patches were released for VMware vSphere on 4/1/2010.  A few for the Cisco Nexus 1000V VEM, and the remainder of the updates are for ESX(i) 4.0.

Two of the patches which I’d like to highlight, ESX400-201003401-BG and ESXi400-201003401-BG, make some required changes to the virtual infrastructure which you should be aware of.  The .VMDK file naming convention, introduced originally in ESX 2.x, is being retired in favor of the original naming convention .DSK which existed prior to ESX 2.x.  The reasoning for this is not yet known but as the saying goes, “You can’t make an omelette without breaking a few eggs.”  So suck it up; it’s not all about you.

How does this affect you?  The impacts will vary depending on your role with VMware Virtual Infrastructure:

  • VI Administrators will need to update any scripts which may be impacted.
  • 3rd party tool vendors will need to update any code which references .VMDK files.
  • Book authors would be advised to perform a mass “find and replace” before their next printing.  For tactical advice on how to do this, speak to Scott Herold or Ron Oglesby as they have experience in this.  As for books and articles already published on VMware disk technologies using the .VMDK naming scheme, refer to the omelette statement above.
  • Lab Manager and View environments are not yet compatible with the file naming convention change.  An updated release for each of these products should be available by Q4 this year, Q1 2011 at the latest.

I am completely in favor of this change.  I never did adapt fully to the migration from .DSK to .VMDK, so this will be just like old times for me.  We need more radical ideas like this to break the chains of complacency.  From a sales perspective, VMware can totally pitch changes like these as innovation which keeps them several steps ahead of their competitors.  So what’s next?  Inside sources tell me HA naming is going back to its roots:  Hello Dynamic Availability Service!

For more information, please follow this link. 😀

Windows 2008 R2 and Windows 7 on vSphere

March 28th, 2010

If you run Windows Server 2008 R2 or Windows 7 as a guest VM on vSphere, you may be aware that it was advised in VMware KB Article 1011709 that the SVGA driver should not be installed during VMware Tools installation.  If I recall correctly, this was due to a stability issue which was seen in specific, but not all, scenarios:

If you plan to use Windows 7 or Windows 2008 R2 as a guest operating system on ESX 4.0, do not use the SVGA drivers included with VMware Tools. Use the standard SVGA driver instead.

Since the SVGA driver is installed by default in a typical installation, it was necessary to perform a custom installation (or scripted perhaps) to exclude the SVGA driver for these guest OS types.  Alternatively, perform a typical VMware Tools installation and remove the SVGA driver from the Device Manager afterwards.  What you ended up with, of course, is a VM using the Microsoft Windows supplied SVGA driver and not the VMware Tools version shown in the first screenshot.  The Microsoft Windows supplied SVGA driver worked and provided stability as well, however one side effect was that mouse movement via VMware Remote Console felt a bit sluggish.

Beginning with ESX(i) 4.0 Update 1 (released 11/19/09), VMware changed the behavior and revised the above KB article in February, letting us know that they now package a new version of the SVGA driver in VMware Tools in which the bits are populated during a typical installation but not actually enabled:

The most effective solution is to update to ESX 4.0 Update 1, which provides a new WDDM driver that is installed with VMware Tools and is fully supported. After VMware Tools upgrade you can find it in C:\Program Files\Common Files\VMware\Drivers\wddm_video.

After a typical VMware Tools installation, you’ll still see a standard SVGA driver installed.  Following the KB article, head to Windows Device Manager and update the driver to the bits located in C:\Program Files\Common Files\VMware\Drivers\wddm_video:

    

The result is the new wddm driver, which ships with the newer version of VMware Tools, is installed: 

After a reboot, the crisp and precise mouse movement I’ve become accustomed to over the years with VMware has returned.  The bummer here is that while the appropriate VMware SVGA drivers get installed in previous versions of Windows guest operating systems, Windows Server 2008 R2 and Windows 7 require manual installation steps, much like VMware Tools installation on Linux guest VMs.  Add to this the fact that the automated installation/upgrade of VMware Tools via VMware Update Manager (VUM) does not enable the wddm driver.  In short, getting the appropriate wddm driver installed for many VMs will require manual intervention or scripting.  One thing you can do is to get the wddm driver installed in your Windows Server 2008 R2 and Windows 7 VM templates.  This will ensure VMs deployed from the templates have the wddm driver installed and enabled.

The wddm driver install method from VMware is helpful for the short term, however, it’s not the scalable and robust long term solution.  We need an automated solution from VMware to get the wddm driver installed.  It needs to be integrated with VUM.  I’m interested in finding out what happens with the next VMware Tools upgrade – will the wddm driver persist, or will the VMware Tools upgrade replace the wddm version with the standard version?  Stay tuned.

Update 11/6/10:  While working in the lab tonight, I noticed that with vSphere 4.1, the correct wddm video driver is installed as part of a standard VMware Tools installation on Windows 7 Ultimate x64 – no need to manually replace the Microsoft video driver with VMware’s wddm version as this is done automatically now.

Update 12/10/10: As a follow up to these tests, I wanted to see what happens when the wddm driver is installed under ESX(i) 4.0 Update 1 and its corresponding VMware Tools, and then the VM is moved to an ESX(i) 4.1 cluster and the VMware Tools are upgraded.  Does the wddm driver remain in tact, or will the 4.1 tools upgrade somehow change the driver?  During this test, I opted to use Windows 7 Ultimate 32-bit as the guest VM guinea pig.  A few discoveries were made, one of which was a surprise:

1.  Performing a standard installation of VMware Tools from ESXi 4.0 Update 1 on Windows 7 32-bit will automatically install the wddm driver, version 7.14.1.31 as shown below.  No manual steps were required to install this driver forcing a 2nd reboot.  I wasn’t counting on this.  I expected the Standard VGA Graphics Adapter driver to be installed as seen previously.  This is good.

SnagIt Capture

After moving the VM to a 4.1 cluster and performing the VMware Tools upgrade, the wddm driver was left in tact, however, its version was upgraded to 7.14.1.40.  This is also good in that the tools ugprade doesn’t negatively impact the desired results of leveraging the wddm driver for best graphics performance.

SnagIt Capture

More conclusive testing should be done with Windows 7 and Windows Server 2008 R2 64-bit to see if the results are the same.  I’ll save this for a future lab maybe.

Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters

March 25th, 2010

This is one of those “I’m documenting it for my own purposes” articles.  Yes I read my own blog once in a while to find information on past topics.  Here I’m basically copying a VMware KB article but I’ll provide a brief introduction.

So your wondering if you should use VMware Paravirtual SCSI?  I’ve gotten this question a few times.  PVSCSI is one of those technologies where “should I implement it” could be best answered with the infamous consulting reply “it depends”.  One person asked if it would be good to use as a default configuration for all VMs.  One notion that I would agree on by and large is that I feel the support complexity increases when using PVSCSI and it should only be used as needed for VMs which need an additional bit of performance squeezed from the disk subsystem.  This is not a technology I would implement by default on all VMs.  Dissecting the practical beneifts and ROI of implementing PVSCSI should be performed, but before that, your valuable time may be better spent finding out if your environment will support it to begin with.  Have a look at VMware KB Article 1010398 which is where the following information comes from, verbatim.

It’s important to identify the guest VMs which support PVSCSI:

Paravirtual SCSI adapters are supported on the following guest operating systems:

  • Windows Server 2008
  • Windows Server 2003
  • Red Hat Enterprise Linux (RHEL) 5

It’s important to further identify more ambiguous type situations where PVSCSI may or may not not fit:

Paravirtual SCSI adapters also have the following limitations:

  • Hot add or hot remove requires a bus rescan from within the guest.
  • Disks with snapshots might not experience performance gains when used on Paravirtual SCSI adapters or if memory on the ESX host is overcommitted.
  • If you upgrade from RHEL 5 to an unsupported kernel, you might not be able to access data on the virtual machine’s PVSCSI disks. You can runvmware-config-tools.pl with the kernel-version parameter to regain access.
  • Because the default type of newly hot-added SCSI adapter depends on the type of primary (boot) SCSI controller, hot-adding a PVSCSI adapter is not supported.
  • Booting a Linux guest from a disk attached to a PVSCSI adapter is not supported. A disk attached using PVSCSI can be used as a data drive, not a system or boot drive. Booting a Microsoft Windows guest from a disk attached to a PVSCSI adapter is not supported in versions of ESX prior to ESX 4.0 Update 1

For more information on PVSCSI, including installation steps, see VMware KB Article 1010398.  One more important thing to note is that in some operating system types, to install PVSCSI, you need to create a virtual machine with the LSI controller, install VMware Tools, then change to the drives to paravirtualized mode.

Meet the VMware Certified Design Experts

March 22nd, 2010

VMware Education Services has unveiled a new site called Meet the VMware Certified Design Experts.  The site serves as a directory of individuals who have achieved VMware VCDX certification, currently VMware’s premier level achievement.  Here you’ll find a short profile and a photo of each of the 39 existing VMware Certified Design Experts around the world.

VMware Education describes the site as follows: 

VMware Certified Design Experts have achieved the VMware Certified Design Expert (VCDX) certification by building on their VMware Certified Professional (VCP) certification. Each expert has taken the extra steps of passing both the Enterprise Administration Exam and the Design Exam, as well as successfully defending a VMware infrastructure design and implementation plan.

VMware Certified Design Experts are part of an elite group of architects leading virtualization implementations around the world. Join this community of experts by earning your VCDX certification.

VCDX’s can use the site to meet others around the world who have been through the process.  Pursuant to VCDX Tip #38, customers, employers, or recruiters can use the site to validate a candidate’s credentials for a project or employment.  For verification purposes, keep in mind that the site is not automated or updated the moment a candidate is minted.  There could be a delay of a few weeks before a certified individual appears on the site.

VMware is working hard to expand the pool of VCDX certified Architects and Engineers.  Many candidates have already satisfied the three written exam requirements and are waiting for their shot at a design submission and defense panel.  VMware typically holds defense panels at major VMware events in the US and Europe.  Just last week, VMware held VCDX defense panels in Munich, Germany, breaking the mold of holding the panels around major VMware events.  Design submissions and defense panels are proctored by existing VCDX’s in the pool.  As the pool grows, VMware should be able to handle larger volumes of candidates.  At that point the system will be pretty well primed and hopefully efficient from a candidate’s perspective.

The VMware Certified Professional Program is designed for individuals who want to demonstrate their expertise in virtual infrastructure and increase the potential for career advancement.  For additional information, you can learn more about the process here.

vpxd.cfg Advanced Configuration

March 13th, 2010

vpxd.cfg is an XML formatted file which can be modified to alter the native behavior of the VMware vCenter Server.  Sparse references on the internet document the changes that can be made in this environment.  Inspired by Ulli Hankeln, the purpose of this blog post is to collect and document all known, unknown, supported, and unsupported vpxd.cfg modifications in a centralized location. 

If you have any to add, please provide feedback in the form of a blog comment along with a link pointing to a reference and I’ll update the post.

**Disclaimer**
As with anything found on this site and much of the internet in general, information is provided “as is” without warranty.  Modify settings at your own risk.  I suggest thoroughly researching the changes first and also checking with VMware Support.

The vpxd.cfg file is located on the VMware vCenter Server by default at %ALLUSERPROFILE%\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

  • On Windows Server 2008, this would generally be C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
  • On Windows Server 2003, this would generally be C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

This collection of vpxd.cfg settings has been sourced from various places.  The parameters will generally apply to a version of vCenter Server ranging from 2.0 through 4.x.  A given parameter can apply to several or even all versions.  However, one thing I didn’t do was specify which version of vCenter Server the parameter applies to – too much work – sorry – you’ll have to experiment in your lab or DEV environment.  I do think it’s safe to say that most of these parameters focus on the latest releases of vCenter Server – 2.5 and 4.0.

Remember to restart the VMware VirtualCenter Server service in the Server Manager for changes to vpxd.cfg to take effect.

Tag:  blockingTimeoutSeconds
Nested In:  vmomi, soapStubAdapter
What It Does:  Defines the timeout value in seconds for SOAP layer blocking.  Use cases for increasing: slow connections, low bandwidth, or high latency between virtual infrastructure components.  Read more here and here.
Example:

<vmomi>
<soapStubAdapter>
<blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
</soapStubAdapter>
</vmomi>

Tag:  calls
Nested In:  trace, vmomi
What It Does:  Unknown.  Read more here.
Example:

<trace>
<vmomi>
<calls>true</calls>
</vmomi>
</trace>

Tag:  cipherList
Nested In:  vmacore, ssl
What It Does:  Reverts to cipher suites used in previous versions of vCenter Server (2.5u3 and earlier) for browser/SSL compatibility issues.  Read more here.
Example:

<vmacore>
<ssl>
<cipherList>DEFAULT</cipherList>
</ssl>
</vmacore>

Tag:  compressOnRoll
Nested In:  log
What It Does:  Defines whether or not vCenter Server vpxd log files are rolled up and compressed into .gz files.  Read more here.
Example:

<log>
<compressOnRoll>false</compressOnRoll>
</log>

Tag:  cpuFeatureMask
Nested In:  guestOSDescriptor, esx-2-x-x, all-versions, all-guests
What It Does:  Masks CPU features to force VMotion compatibility between hosts. VMware neither supports nor recommends modifying the VMotion constraints for CPU features.  Read more here.
Example:

<guestOSDescriptor>
<esx-2-x-x>
<all-versions>
<all-guests>
<cpuFeatureMask>Elements and mask definition go in here</cpuFeatureMask>
</all-guests>
</all-versions>
</esx-2-x-x>
</guestOSDescriptor>

Tag:  directory
Nested In:  log
What It Does:  Defines the location for the vCenter logs.  Read more here.
Example:

<log>
<directory>D:\VC_Logs</directory>
</log>

Tag:  dontstartconsolidation
Nested In:  vcp2v
What It Does:  May resolve an issue where the Consolidation button is missing in the Virtual Infrastructure Client.  Read more here.
Example:

<vcp2v>
<dontstartconsolidation>true</dontstartconsolidation>
</vcp2v>

Tag:  filterOverheadLimitIssues
Nested In:  vpxd
What It Does:  Unknown.
Example:

<vpxd>
<filterOverheadLimitIssues>true</filterOverheadLimitIssues>
</vpxd>

Tag:  hostRescanFilter
Nested In:  unknown
What It Does:  Defines the behavior of mass ESX(i) host rescans of vmHBAs.  Read more here.
Example:

<hostRescanFilter>true</hostRescanFilter>

Tag:  IoMax
Nested In:  vmacore, threadpool
What It Does:  Unknown but my guess is it defines the maximum I/O for the vpxd.exe process (vCenter Server service). Influenced by TaskMax.
Example:

<vmacore>
<threadpool>
<IoMax>200</IoMax>
</threadpool>
</vmacore>

Tag:  level
Nested In:  log
What It Does:  Defines the logging level for vCenter logs.  Read more here.
Example:

 <log>
<level>trivia</level>
</log>

Tag:  logLevel
Nested In:  trace, vmomi
What It Does:  Enables debug logging level for vmomi?  Read more here.
Example:

<trace>
<vmomi>
<logLevel>verbose</logLevel>
</vmomi>
</trace>

Tag:  loglevel
Nested In:  nfc
What It Does:  Enables debug logging level for the NFC process.  Read more here.
Example:

<nfc>
<loglevel>debug</loglevel>
</nfc>

Tag:  managedIP
Nested In:  unknown
What It Does:  Defines the managed IP address used in vCenter Server Heartbeat.  Read more here.
Example:

<managedIP>10.10.0.1</managedIP>

Tag:  maxCostPerHost
Nested In:  ResourceManager
What It Does:  Defines the number of simultaneous VM migrations (both hot and cold) per ESX(i) host.  Read more here.
Example:

<ResourceManager>
<maxCostPerHost>8</maxCostPerHost>
</ResourceManager>

Tag:  maxFileNum
Nested In:  log
What It Does:  Defines the maximum number of log files for vCenter logs.  Read more here.
Example:

<log>
<maxFileNum>50</maxFileNum>
</log>

Tag:  maxFileSize
Nested In:  log
What It Does:  Defines the maximum log file size in Bytes and thus rollover interval for vCenter logs.  Read more here.
Example:

<log>
<maxFileSize>10485760</maxFileSize>
</log>

Tag:  name
Nested In:  log
What It Does:  Defines the log file prefix name for vCenter logs.  Read more here.
Example:

<log>
<name>vpxd</name>
</log>

Tag:  notRespondingTimeout
Nested In:  heartbeat
What It Does:  Defines the heartbeat timeout value in seconds between ESX(i) hosts and vCenter Server.  Use case would be to increase the value if remote ESX(i) hosts frequently go into a not responding state in vCenter Server due to WAN bandwidth or latency issues.  Read more here.
Example:

<heartbeat>
<notRespondingTimeout>60</notRespondingTimeout>
</heartbeat>

Tag:  portReserveTimeout
Nested In:  dvs
What It Does:  Defines the timeout value in minutes for unused dvPort reservations.  Lowering the value temporarily is helpful for unlocking dvPorts to remove a vDS or dvPort group.  Read more here.
Example:

<dvs>
<portReserveTimeout>10</portReserveTimeout>
</dvs>

Tag:  serializeadds
Nested In:  vpxd, das
What It Does:  Unknown but if I had to guess I’d say it defines the behavior of how the HA agent is installed on cluster hosts.
Example:

<vpxd>
<das>
<serializeadds>true</serializeadds>
</das>
</vpxd>

Tag:  slotCpuMinMHz
Nested In:  vpxd, das
What It Does:  Defines the minimum CPU calculation of a HA cluster slot size when there are no CPU reservations. Read more here.
Example:

<vpxd>
<das>
<slotCpuMinMHz>256</slotCpuMinMHz>
</das>
</vpxd>

Tag:  slotMemMinMB
Nested In:  vpxd, das
What It Does:  Defines the minimum memory calculation of a HA cluster slot size when there are no memory reservations. Read more here.
Example:

<vpxd>
<das>
<slotMemMinMB>0</slotMemMinMB>
</das>
</vpxd>

Tag:  sspiProtocol
Nested In:  unknown
What It Does:  Defines the authentication mechanism used with passthrough authentication between the Virtual Infrastructure Client and vCenter Server.  Read more here.
Example:

<sspiProtocol>Kerberos</sspiProtocol>

Tag:  TaskMax
Nested In:  vmacore, threadpool
What It Does:  Defines the number of worker threads for the vpxd.exe process (vCenter Server service). Influences IoMax.
Example:

<vmacore>
<threadpool>
<TaskMax>30</TaskMax>
</threadpool>
</vmacore>

Tag:  timeout
Nested In:  task
What It Does:  Defines the timeout value in seconds for long tasks.  Read more here.
Example:

<task>
<timeout>10800</timeout>
</task>

Tag:  verbose
Nested In:  trace, db
What It Does:  Enables database tracing.  Enables database logging in the vpxd log.  Read more here and here.
Example:

<trace>
<db>
<verbose>true</verbose>
</db>
</trace>

Tag:  verbosity
Nested In:  trace, vmomi
What It Does:  Unknown.  Read more here.
Example:

<trace>
<vmomi>
<verbosity>verbose</verbosity>
</vmomi>
</trace>

Tag:  verboseObjectSize
Nested In:  trace, vmomi
What It Does:  Unknown.  Read more here.
Example:

<trace>
<vmomi>
<verboseObjectSize>40</verboseObjectSize>
</vmomi>
</trace>

Tag:  VMOnVirtualIntranet
Nested In:  migrate, test, CompatibleNetworks
What It Does:  Setting to false enables VMotion for VMs connected to an internal vSwitch. Setting to false will turn off the internal vSwitch restriction on VMotion events. Useful for servers behind a firewall virtual appliance deployed in bridged networking mode.  Read more here.
Example:

<migrate>
<test>
<CompatibleNetworks>
<VMOnVirtualIntranet>false</VMOnVirtualIntranet>
</CompatibleNetworks>
</test>
</migrate>

Tag:  VMOverheadGrowthLimit
Nested In:  cluster
What It Does:  Defines the growth rate cap in terms of MB per minute for VM memory overhead at the cluster level. Can be adjusted to resolve high CPU utilization in guest VMs introduced in ESX(i) 3.5 and vCenter 2.5.  Read more here.
Example:

<cluster>
<VMOverheadGrowthLimit>5</VMOverheadGrowthLimit>
</cluster>

 

Slightly related, the vCenter Server process (vpxd.exe) can be launched at a command prompt on the vCenter Server (instead of starting as a service) for troubleshooting purposes.  The executable is located at:

<Install Directory>\VMware\Infrastructure\VirtualCenter Server>vpxd.exe

Usage: vpxd.exe [FLAGS]
Flags:
-r Register VMware VirtualCenter Server
-u Unregister VMware VirtualCenter Server
-s Run as a standalone server rather than a Service
-c Print vmdb schema to stdout
-b Recreate database repository
-f cfg Use the specified file instead of the default vpxd.cfg
-l licenseKey Store license key in ldap and assign it to VirtualCenter
-e feature Set the feature to be in use for VirtualCenter. This option takes only one feature at a time.
-p Reset the database password
-v Print the version number to stdout

Perf Charts Service Experienced An Internal Error

March 12th, 2010

Happy Friday evening y’all.  Tonight’s blog post comes from a former colleague of mine whom I will call “Paul Berg”.  Paul came across an error in VMware vSphere which he was able to resolve and he would like to share the solution with the VMware community. 

Paul uses an Oracle database to back end vCenter. When viewing the performance charts in Performance tab | Overview button, he received the following error:

Perf Charts service experienced an internal error.

Message:  Report application initialization is not completed successfully.  Retry in 60 seconds.

You can probably guess what followed… missing data in the charts.  No joy whatsoever.

Following is the resolution:

1. Get the fully qualified domain name or the global name of the TNS service from the Oracle database. This can be found in the file named tnsnames.ora on the Oracle database server

2. Add this FQDN to the registry key HKLM\Software\ODBC\ODBC.INI\VirtualCenter\ServerName on the VC server.

3. Restart the VMware VirtualCenter Server service.

For us, the database was listed as VMDB in the registry. We have moved to an Oracle RAC configuration so I needed to change the entry to VMDB.GLOBAL to match what was in the tnsnames.ora listing. I wasn’t aware that VMDB.GLOBAL was considered the FQDN for an Oracle DB.

The following VMware KB Article 1012812 documents the issue as well as a few different approaches to a resolution depending on root cause.  Again, this issue is specific to Oracle database environments.

Performance Overview charts fail with the error: STATs Report Service internal error

Thank you for sharing Paul.  I’ve got one more in the queue from you – I’ll try to get it out in the next couple of weeks.  Here’s a teaser: Poor vSphere performance on Nehalem processors.  Ouch!