Posts Tagged ‘vCenter Server’

« Older Entries

OVF? OVA? WTF?

July 2nd, 2010

If you’ve worked with recent versions of VMware virtual infrastructure, Converter, or Workstation, you may be familiar with the fact that these products have the native ability to work with virtual machines in the Open Virtualization Format, or OVF for short.  OVF is a Specification governed by the DMTF (Distributed Management Task Force) which to me sounds a lot like RFCs which provide standards for protocols and communication across compute platforms – basically SOPs for how content is delivered on the internet as we know it today.

So if there’s one standard, why is it that when I choose to create an OVF (Export OVF Template in the vSphere Client), I’m prompted to create either an OVF or an OVA?  If the OVF is an OVF, then what’s an OVA?

 7-2-2010 8-00-01 PM

Personally, I’ve seen both formats, typically when deploying packaged appliances.  The answer is simple: Both the OVF and the OVA formats roll up into the Specification defined by the DMTF.  The difference between the two is in the presentation and encapsulation.  The OVF is a construct of a few files, all of which are essential to its definition and deployment.  The OVA on the other hand is a single file with all of the necessary information encapsulated inside of it.  Think of the OVA as an archive file.  The single file format provides ease in portability.  From a size or bandwidth perspective, there is no advantage between one format or the other as they each tend to be the same size when all is said and done.

7-2-2010 8-13-26 PM

The DMTF explains the two formats on pages 12 through 13 in the PDF linked above:

An OVF package may be stored as a single file using the TAR format. The extension of that file shall be .ova (open virtual appliance or application).

An OVF package can be made available as a set of files, for example on a standard Web server.

Do keep in mind that which ever file type you choose to work with, if you plan on hosting them on a web server, MIME types will need to be set up for .OVF, OVA, or both, in order for a client to download them for deployment onto your hypervisor.

At 41 pages, the OVF Specification contains a surprising amount of detail.  There’s more to it than you might think, and for good reason:

The Open Virtualization Format (OVF) Specification describes an open, secure, portable, efficient and extensible format for the packaging and distribution of software to be run in virtual machines.

Open, meaning cross platform (bring your own hypervisor).  Combined with Secure and Portable attributes, OVF may be one of the key technologies for intracloud and intercloud mobility.  The format is a collaborative effort spawned from a variety of contributors:

Simon Crosby, XenSource
Ron Doyle, IBM
Mike Gering, IBM
Michael Gionfriddo, Sun Microsystems
Steffen Grarup, VMware (Co-Editor)
Steve Hand, Symantec
Mark Hapner, Sun Microsystems
Daniel Hiltgen, VMware
Michael Johanssen, IBM
Lawrence J. Lamers, VMware (Chair)
John Leung, Intel Corporation
Fumio Machida, NEC Corporation
Andreas Maier, IBM
Ewan Mellor, XenSource
John Parchem, Microsoft
Shishir Pardikar, XenSource
Stephen J. Schmidt, IBM
René W. Schmidt, VMware (Co-Editor)
Andrew Warfield, XenSource
Mark D. Weitzel, IBM
John Wilson, Dell

Take a look at the OVF Specifications document as well as some of the other work going on at DTMF. 

Have a great and safe July 4th weeekend, and congratulations to the Dutch on their win today in World Cup Soccer.  I for one will be glad when it’s all over with and our Twitter APIs can return to normal again.

7 comments »

Posted in Virtualization

Tags:

vSphere Upgrade Prerequisites Checklist

May 27th, 2010

Upgrading your virtual infrastructure to vSphere?  Be sure to check out this handy reference from VMware:  vSphere Upgrade Prerequisites Checklist.  There are several areas which need to be considered and this document covers them all, including both requirements and recommendations.  If you’re a consultant who visits new customer sites on a regular basis, it wouldn’t be a bad idea to bring this with to each engagement, or at least a condensed version of it.

No comments »

Posted in Virtualization

Tags:

Happy Birthday vSphere!

May 21st, 2010

birthday-cake

I was reminded by today’s vCalendar page that vSphere was launched by VMware one year ago today.  Happy Birthday Buddy – you set the bar which all other hypervisors aspire to be at one day.

On this day in 2009, VMware vSphere, the next generation datacenter virtualization product and successor to Virtual Infrastructure 3 (VI3), was released boasting approximately 150 new features, new license tiers, and an amazing 350,000 I/O operations per second (IOPS). vSphere is a 64-bit only ESX host OS.

Don’t have a vCalendar yet?  Get one!

1 comment »

Posted in Virtualization

Tags:

vpxd.cfg Advanced Configuration

March 13th, 2010

vpxd.cfg is an XML formatted file which can be modified to alter the native behavior of the VMware vCenter Server.  Sparse references on the internet document the changes that can be made in this environment.  Inspired by Ulli Hankeln, the purpose of this blog post is to collect and document all known, unknown, supported, and unsupported vpxd.cfg modifications in a centralized location. 

If you have any to add, please provide feedback in the form of a blog comment along with a link pointing to a reference and I’ll update the post.

**Disclaimer**
As with anything found on this site and much of the internet in general, information is provided “as is” without warranty.  Modify settings at your own risk.  I suggest thoroughly researching the changes first and also checking with VMware Support.

The vpxd.cfg file is located on the VMware vCenter Server by default at %ALLUSERPROFILE%\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

  • On Windows Server 2008, this would generally be C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
  • On Windows Server 2003, this would generally be C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

This collection of vpxd.cfg settings has been sourced from various places.  The parameters will generally apply to a version of vCenter Server ranging from 2.0 through 4.x.  A given parameter can apply to several or even all versions.  However, one thing I didn’t do was specify which version of vCenter Server the parameter applies to – too much work – sorry – you’ll have to experiment in your lab or DEV environment.  I do think it’s safe to say that most of these parameters focus on the latest releases of vCenter Server – 2.5 and 4.0.

Remember to restart the VMware VirtualCenter Server service in the Server Manager for changes to vpxd.cfg to take effect.

Tag:  blockingTimeoutSeconds
Nested In:  vmomi, soapStubAdapter
What It Does:  Defines the timeout value in seconds for SOAP layer blocking.  Use cases for increasing: slow connections, low bandwidth, or high latency between virtual infrastructure components.  Read more here and here.
Example:

<vmomi>
<soapStubAdapter>
<blockingTimeoutSeconds>10800</blockingTimeoutSeconds>
</soapStubAdapter>
</vmomi>

Tag:  calls
Nested In:  trace, vmomi
What It Does:  Unknown.  Read more here.
Example:

<trace>
<vmomi>
<calls>true</calls>
</vmomi>
</trace>

Tag:  cipherList
Nested In:  vmacore, ssl
What It Does:  Reverts to cipher suites used in previous versions of vCenter Server (2.5u3 and earlier) for browser/SSL compatibility issues.  Read more here.
Example:

<vmacore>
<ssl>
<cipherList>DEFAULT</cipherList>
</ssl>
</vmacore>

Tag:  compressOnRoll
Nested In:  log
What It Does:  Defines whether or not vCenter Server vpxd log files are rolled up and compressed into .gz files.  Read more here.
Example:

<log>
<compressOnRoll>false</compressOnRoll>
</log>

Tag:  cpuFeatureMask
Nested In:  guestOSDescriptor, esx-2-x-x, all-versions, all-guests
What It Does:  Masks CPU features to force VMotion compatibility between hosts. VMware neither supports nor recommends modifying the VMotion constraints for CPU features.  Read more here.
Example:

<guestOSDescriptor>
<esx-2-x-x>
<all-versions>
<all-guests>
<cpuFeatureMask>Elements and mask definition go in here</cpuFeatureMask>
</all-guests>
</all-versions>
</esx-2-x-x>
</guestOSDescriptor>

Tag:  directory
Nested In:  log
What It Does:  Defines the location for the vCenter logs.  Read more here.
Example:

<log>
<directory>D:\VC_Logs</directory>
</log>

Tag:  dontstartconsolidation
Nested In:  vcp2v
What It Does:  May resolve an issue where the Consolidation button is missing in the Virtual Infrastructure Client.  Read more here.
Example:

<vcp2v>
<dontstartconsolidation>true</dontstartconsolidation>
</vcp2v>

Tag:  filterOverheadLimitIssues
Nested In:  vpxd
What It Does:  Unknown.
Example:

<vpxd>
<filterOverheadLimitIssues>true</filterOverheadLimitIssues>
</vpxd>

Tag:  hostRescanFilter
Nested In:  unknown
What It Does:  Defines the behavior of mass ESX(i) host rescans of vmHBAs.  Read more here.
Example:

<hostRescanFilter>true</hostRescanFilter>

Tag:  IoMax
Nested In:  vmacore, threadpool
What It Does:  Unknown but my guess is it defines the maximum I/O for the vpxd.exe process (vCenter Server service). Influenced by TaskMax.
Example:

<vmacore>
<threadpool>
<IoMax>200</IoMax>
</threadpool>
</vmacore>

Tag:  level
Nested In:  log
What It Does:  Defines the logging level for vCenter logs.  Read more here.
Example:

 <log>
<level>trivia</level>
</log>

Tag:  logLevel
Nested In:  trace, vmomi
What It Does:  Enables debug logging level for vmomi?  Read more here.
Example:

<trace>
<vmomi>
<logLevel>verbose</logLevel>
</vmomi>
</trace>

Tag:  loglevel
Nested In:  nfc
What It Does:  Enables debug logging level for the NFC process.  Read more here.
Example:

<nfc>
<loglevel>debug</loglevel>
</nfc>

Tag:  managedIP
Nested In:  unknown
What It Does:  Defines the managed IP address used in vCenter Server Heartbeat.  Read more here.
Example:

<managedIP>10.10.0.1</managedIP>

Tag:  maxCostPerHost
Nested In:  ResourceManager
What It Does:  Defines the number of simultaneous VM migrations (both hot and cold) per ESX(i) host.  Read more here.
Example:

<ResourceManager>
<maxCostPerHost>8</maxCostPerHost>
</ResourceManager>

Tag:  maxFileNum
Nested In:  log
What It Does:  Defines the maximum number of log files for vCenter logs.  Read more here.
Example:

<log>
<maxFileNum>50</maxFileNum>
</log>

Tag:  maxFileSize
Nested In:  log
What It Does:  Defines the maximum log file size in Bytes and thus rollover interval for vCenter logs.  Read more here.
Example:

<log>
<maxFileSize>10485760</maxFileSize>
</log>

Tag:  name
Nested In:  log
What It Does:  Defines the log file prefix name for vCenter logs.  Read more here.
Example:

<log>
<name>vpxd</name>
</log>

Tag:  notRespondingTimeout
Nested In:  heartbeat
What It Does:  Defines the heartbeat timeout value in seconds between ESX(i) hosts and vCenter Server.  Use case would be to increase the value if remote ESX(i) hosts frequently go into a not responding state in vCenter Server due to WAN bandwidth or latency issues.  Read more here.
Example:

<heartbeat>
<notRespondingTimeout>60</notRespondingTimeout>
</heartbeat>

Tag:  portReserveTimeout
Nested In:  dvs
What It Does:  Defines the timeout value in minutes for unused dvPort reservations.  Lowering the value temporarily is helpful for unlocking dvPorts to remove a vDS or dvPort group.  Read more here.
Example:

<dvs>
<portReserveTimeout>10</portReserveTimeout>
</dvs>

Tag:  serializeadds
Nested In:  vpxd, das
What It Does:  Unknown but if I had to guess I’d say it defines the behavior of how the HA agent is installed on cluster hosts.
Example:

<vpxd>
<das>
<serializeadds>true</serializeadds>
</das>
</vpxd>

Tag:  slotCpuMinMHz
Nested In:  vpxd, das
What It Does:  Defines the minimum CPU calculation of a HA cluster slot size when there are no CPU reservations. Read more here.
Example:

<vpxd>
<das>
<slotCpuMinMHz>256</slotCpuMinMHz>
</das>
</vpxd>

Tag:  slotMemMinMB
Nested In:  vpxd, das
What It Does:  Defines the minimum memory calculation of a HA cluster slot size when there are no memory reservations. Read more here.
Example:

<vpxd>
<das>
<slotMemMinMB>0</slotMemMinMB>
</das>
</vpxd>

Tag:  sspiProtocol
Nested In:  unknown
What It Does:  Defines the authentication mechanism used with passthrough authentication between the Virtual Infrastructure Client and vCenter Server.  Read more here.
Example:

<sspiProtocol>Kerberos</sspiProtocol>

Tag:  TaskMax
Nested In:  vmacore, threadpool
What It Does:  Defines the number of worker threads for the vpxd.exe process (vCenter Server service). Influences IoMax.
Example:

<vmacore>
<threadpool>
<TaskMax>30</TaskMax>
</threadpool>
</vmacore>

Tag:  timeout
Nested In:  task
What It Does:  Defines the timeout value in seconds for long tasks.  Read more here.
Example:

<task>
<timeout>10800</timeout>
</task>

Tag:  verbose
Nested In:  trace, db
What It Does:  Enables database tracing.  Enables database logging in the vpxd log.  Read more here and here.
Example:

<trace>
<db>
<verbose>true</verbose>
</db>
</trace>

Tag:  verbosity
Nested In:  trace, vmomi
What It Does:  Unknown.  Read more here.
Example:

<trace>
<vmomi>
<verbosity>verbose</verbosity>
</vmomi>
</trace>

Tag:  verboseObjectSize
Nested In:  trace, vmomi
What It Does:  Unknown.  Read more here.
Example:

<trace>
<vmomi>
<verboseObjectSize>40</verboseObjectSize>
</vmomi>
</trace>

Tag:  VMOnVirtualIntranet
Nested In:  migrate, test, CompatibleNetworks
What It Does:  Setting to false enables VMotion for VMs connected to an internal vSwitch. Setting to false will turn off the internal vSwitch restriction on VMotion events. Useful for servers behind a firewall virtual appliance deployed in bridged networking mode.  Read more here.
Example:

<migrate>
<test>
<CompatibleNetworks>
<VMOnVirtualIntranet>false</VMOnVirtualIntranet>
</CompatibleNetworks>
</test>
</migrate>

Tag:  VMOverheadGrowthLimit
Nested In:  cluster
What It Does:  Defines the growth rate cap in terms of MB per minute for VM memory overhead at the cluster level. Can be adjusted to resolve high CPU utilization in guest VMs introduced in ESX(i) 3.5 and vCenter 2.5.  Read more here.
Example:

<cluster>
<VMOverheadGrowthLimit>5</VMOverheadGrowthLimit>
</cluster>

 

Slightly related, the vCenter Server process (vpxd.exe) can be launched at a command prompt on the vCenter Server (instead of starting as a service) for troubleshooting purposes.  The executable is located at:

<Install Directory>\VMware\Infrastructure\VirtualCenter Server>vpxd.exe

Usage: vpxd.exe [FLAGS]
Flags:
-r Register VMware VirtualCenter Server
-u Unregister VMware VirtualCenter Server
-s Run as a standalone server rather than a Service
-c Print vmdb schema to stdout
-b Recreate database repository
-f cfg Use the specified file instead of the default vpxd.cfg
-l licenseKey Store license key in ldap and assign it to VirtualCenter
-e feature Set the feature to be in use for VirtualCenter. This option takes only one feature at a time.
-p Reset the database password
-v Print the version number to stdout

4 comments »

Posted in Virtualization

Tags:

Perf Charts Service Experienced An Internal Error

March 12th, 2010

Happy Friday evening y’all.  Tonight’s blog post comes from a former colleague of mine whom I will call “Paul Berg”.  Paul came across an error in VMware vSphere which he was able to resolve and he would like to share the solution with the VMware community. 

Paul uses an Oracle database to back end vCenter. When viewing the performance charts in Performance tab | Overview button, he received the following error:

Perf Charts service experienced an internal error.

Message:  Report application initialization is not completed successfully.  Retry in 60 seconds.

You can probably guess what followed… missing data in the charts.  No joy whatsoever.

Following is the resolution:

1. Get the fully qualified domain name or the global name of the TNS service from the Oracle database. This can be found in the file named tnsnames.ora on the Oracle database server

2. Add this FQDN to the registry key HKLM\Software\ODBC\ODBC.INI\VirtualCenter\ServerName on the VC server.

3. Restart the VMware VirtualCenter Server service.

For us, the database was listed as VMDB in the registry. We have moved to an Oracle RAC configuration so I needed to change the entry to VMDB.GLOBAL to match what was in the tnsnames.ora listing. I wasn’t aware that VMDB.GLOBAL was considered the FQDN for an Oracle DB.

The following VMware KB Article 1012812 documents the issue as well as a few different approaches to a resolution depending on root cause.  Again, this issue is specific to Oracle database environments.

Performance Overview charts fail with the error: STATs Report Service internal error

Thank you for sharing Paul.  I’ve got one more in the queue from you – I’ll try to get it out in the next couple of weeks.  Here’s a teaser: Poor vSphere performance on Nehalem processors.  Ouch!

1 comment »

Posted in Virtualization

Tags:

Old vCenter Server Name Shown In Title Bar; Update Manager Plugin Fails

November 22nd, 2009

I recently rebuilt a vCenter Server on a new Windows host having a different name than the vCenter Server host used previously. Wanting to maintain my existing datacenter configuration and layout, I chose to connect to and preserve the existing SQL database back end.

The installation went well and my existing datacenter configuration was in tact, however, I noticed one anomaly having two symptoms. After establishing a vSphere Client connection to the new vCenter Server named vc40.boche.mcse, the vSphere Client title bar showed jarjar.boche.mcse which was the old vCenter Server name.

Furthermore, the Update Manager plugin was failing to load because it could not establish a connection to jarjar.boche.mcse. I wasn’t surprised a connection could not be made since jarjar was retired and no longer on the network. But why was the legacy vCenter Server name persisting in my new installation?

At first, I thought there was some funky DNS reverse lookup going on but I was able to quickly rule that out when I remembered that I had assigned a new IP address to the new vCenter Server host.

I quickly came to the conclusion that there was a row in the SQL database tattooed with the old vCenter Server name which was showing up in the vSphere Client. With that thought in mind, I used the vSphere Client to access the Administration|vCenter Server Settings menu option.

There it was, under Runtime Settings, the old name of the vCenter Server from the original installation. I was able to simply change the Name from jarjar.boche.mcse to vc40.boche.mcse

Afterwards, the vSphere Client title bar was updated with the correct name of the vCenter Server vc40.boche.mcse. No reboot or recycling of any services needed. The Update Manager plugin had also followed suit, making its connection to the correct vCenter Server name instead of the old one which no longer existed.

Simple stuff but I thought I’d write it up in case anyone else ran into this and was pulling their hair out.

No comments »

Posted in Virtualization

Tags:

Create a 32-bit vCenter DSN on a 64-bit Operating System

November 21st, 2009

As I had pointed out in this blog post, VMware hints that 64-bit may be the future for vCenter Server. I decided that for my upgrade to vCenter 4.0 Update 1 this weekend, I would take the opportunity to rebuild my vCenter server from Windows Server 2003 32-bit to Windows Server 2008 64-bit.

Once the 64-bit base operating system build was complete, I installed the 64-bit Microsoft SQL Server Native Client drivers (downloadable here) since my back end database is Microsoft SQL Server 2005 on a remote server. A key thing to remember about this installation is that it installs both 64-bit and 32-bit DSN drivers.

The next step is to create the vCenter ODBC DSNs. Although vCenter Server runs on 64-bit operating systems, it currently requires a 32-bit ODBC DSN. This is important to remember because the Windows Start Menu launches the 64-bit ODBC DSN tool, not the 32-bit version I needed.  The vCenter Server (and Update Manager) installation will not complete without a 32-bit DSN.

To create a 32-bit DSN on a 64-bit operating system, run the following executable:

[WindowsDir]\SysWOW64\odbcad32.exe

Once the utility opens, you’ll be greeted by all the legacy 32-bit ODBC DSNs you’ve likely seen for years working with tiered Windows platforms. If using Microsoft SQL Server 2005 like me, be sure to select the SQL Native Client driver towards the bottom of the list, and not Driver da Microsoft para arquivos texto highlighted below:

Proceed with the creation of the vCenter Server and Update Manager ODBC DSNs and complete the vCenter Server and Update Manager installations.

This information and much more can be found in the ESX and vCenter Server Installation Guide, page 73.

5 comments »

Posted in Virtualization

Tags:

64-bit vCenter Server Coming? VMFS-2 Support Going Away?

November 20th, 2009

Looking at the VMware vCenter 4.0 Update 1 Release Notes, it appears we may be looking at 64-bit only versions of vCenter Server in the future:

“Future releases of VMware vCenter Server might not support installation on 32-bit Windows operating systems. VMware recommends installing vCenter Server on a 64-bit Windows operating system. If you have VirtualCenter 2.x installed, see the vSphere Upgrade Guide for instructions on installing vCenter Server on a 64-bit operating system and preserving your VirtualCenter database.”

I’m hoping this won’t be a deal breaker for anyone as we should all have 64-bit hardware in the datacenter by now. However, we may be temporarily inconvenienced with Windows Server platform upgrades from 32 to 64-bit.

In the same document, I also noticed verbiage about VMFS-2 support going away in future vSphere releases. If you’re not completely rid of ESX2 and VMFS-2 in your environment by now, I’d start planning on it soon.

2 comments »

Posted in Virtualization

Tags:

Virtualizing vCenter With vDS Catch-22

October 9th, 2009

I’ve typically been a fan of virtualizing the vCenter management server in most situations. VMware vCenter and Update Manager both make fine virtualization candidates as long as the underlying infrastructure for vCenter stays up. Loss of vCenter in a blackout situation can make things a bit of a hassle, but one can work through it with the right combination of patience and knowledge.

A few nights ago I had decided to migrate my vCenter VM to my vSphere virtual infrastructure. Because my vCenter VM was on a standalone VMware Server 2.0 box, I had to shut down the vCenter VM in order to cold migrate it to one of the ESX4 hosts directly, transfer the files to the SAN, upgrade virtual hardware, etc. Once the files were migrated to the vSphere infrastructure, it was time to configure the VM for the correct network and power it up. This is where I ran into the problem.

vCenter was shut down and unavailable, therefore, I had connected my vSphere client directly to the ESX4 host in which I transferred the VM to. When trying to configure the vCenter VM to use the vNetwork Distributed Switch (vDS) port group I had set up for all VM traffic, it was unavailable in the dropdown list of networks. The vCenter server was powered down and thus the vDS Control Plane was unavailable, eliminating my view of vDS networks.

This is a dilemma. Without a network connection, the vCenter server will not be able to communicate with the back end SQL database on a different box running SQL. This will cause the vCenter server services to not start and thus I’ll never have visibility to the vDS. Fortunately I have a fairly flat network in the lab with just a few subnets. I was able to create a temporary vSwitch and port group locally on the ESX4 host which would grant the vCenter VM the network connectivity it needed so I could then modify the network, changing from a local to a vDS port group on the fly.

Once the vCenter server was back up, I further realized that vDS port groups are still unable to be seen when the vSphere client is connected directly to an ESX4 host. The ability configure a VM to utilize vDS networking requires both that the vCenter server be functional, as well as a vSphere client connected to said vCenter server and not a managed host.

The situation I explained above is the catch-22 – the temporary inability to configure VMs for vDS networking while the vCenter server is unavailable. One might call my situation a convergence of circumstances, but with an existing virtualized vCenter server that you’re looking to migrate to a vDS integrated vSphere infrastructure, the scenario is very real. I’d like to note all VMs that had been running on a vDS port continued to run without a network outage as the vDS Data Plane is maintained on each host and remained in tact.

21 comments »

Posted in Virtualization

Tags:

SQL 2005 SP2 End of Support to Force Rapid vSphere Upgrade?

October 1st, 2009

The way I read it, the Microsoft Support Lifecycle for SQL Server 2005 tells me that SQL Server 2005 SP2 support ends on 12/15/2009. That’s about 10 weeks from today.

Why should you care? If you’re utilizing VMware vCenter Server 2.5 in your production datacenter, you’ve got about 10 weeks to upgrade to vSphere to stay within a VMware supported configuration. The VMware Virtual Infrastructure Compatibility Matrixes reveal on page 10 that vCenter 2.5 is only compatible with SQL Server 2005 up to Service Pack 2. SP3 is not supported.

To make the jump to SQL Server 2005 SP3 or SQL Server 2008 requires upgrading to vSphere to stay within a VMware supported configuration.

I would venture to guess that a lot of VI customers are not ready for the jump to vSphere, especially those who wish to take advantage of the new features and the design considerations which must be evaluated and planned before deployment. Not to mention the licensing considerations which are tied to the new features. While we’re on the subject of licensing, keep in mind Enterprise licensing is retired mid December 2009. To keep existing Enterprise features in the virtual infrastructure will require Enterprise Plus licensing after the mid December Enterprise license retirement date.

With the SQL 2005 SP2 retirement date approaching, I’ll be looking for VMware modify their support stance to support SQL Server 2005 SP3. A lot of customers are going to need this to keep within support.

Speaking of SQL Server 2008, beware a caveat that Orchestrator 4.0 is not supported on SQL 2008 (yet).

9 comments »

Posted in Virtualization

Tags:

Saturday Grab Bag

September 12th, 2009

Here’s a collection of quick hits I’ve been meaning to get to. Individually, their content is a bit on the short side for the length I normally like to write so I thought I’d throw them together in a single post and see how it comes out.

Tasks and Events List Lengths

First up is the listing of Tasks and Events in the vSphere Client. Have you ever started troubleshooting an issue in the vSphere client by looking at the Tasks or Events and the chronological listing of events doesn’t go back far enough to the date or time you’re looking for? Not finding the logs you’re looking for in the vSphere Client usually means you need to open a PuTTY session and start sifting through logs in /var/log/ or /var/log/vmware/ in the Service Console. The reason for this is that the vSphere Client, by default, is configured to tail the last 100 entries in the Tasks or Events list. You can find this setting in your vSphere Client by choosing “Edit|Client Settings” then choose the “Lists” tab:

Simply increase the value from 100 to whatever you’d like, with 1,000 being the highest allowable value. Notice that when this number is increased, you will immediately see more history. In other words, you don’t have to necessarily wait for time to pass and more historical events to accumulate to see the additional rows of information. Also note that this is a vSphere Client setting which is retained client side and applies to both vCenter Server and ESX(i) host connections.

Collecting diagnostic information for VMware products

Like any offering from a software or hardware vendor, VMware products aren’t perfect. During your VMware experience, you may run into a problem which requires the intervention of VMware support. More often than not, VMware is going to ask you to generate a support bundle which consists of a collection of diagnostic and configuration files and logs. Following this paragraph is a link to VMware KB1008524 which contains links to creating support bundles for various VMware products. Note that in some cases there are different methods for different versions of the same product. If you choose to create a VMware SR online, it is helpful to have created these log bundles in advance so you can attach them to the SR. If you’ve done VMware support long enough, you already know how to FTP log bundles to VMware after an SR number has been generated.

Collecting diagnostic information for VMware products

New VMware Update Manager won’t download ESX(i) patches

Scenario: You’ve built a new VMware vCenter Server in addition to a new VMware Update Manager Server (VUM). After properly configuring Update Manager as well as the necessary internet, proxy, baseline, and scheduled task settings, VUM proceeds to download Windows, Linux, and application patches, but it won’t download ESX(i) host patches. As I found out by trench experience, the cause is because no ESX(i) hosts have been added to the vCenter Server and thus no hosts are being managed by VUM. You need to add at least one ESX(i) host to vCenter Server before VUM will be triggered to suck down all the host updates. One might then ask why guest patches are being downloaded. The only answer I have for the inconsistent behavior is due the fact that ESX(i) host patches are downloaded from VMware, while guest OS and application patches are downloaded from a completely different source, Shavlik. The mechanics behind the download processes obviously differ between the two.

What vCenter Server is this ESX(i) host managed by?

Scenario: You administer a large VMware virtual infrastructure with many vCenter Servers. You need to manage or configure a host or cluster but haven’t the slightest idea what vCenter Server to connect to. You can easily find out by attempting a Virtual Infrastructure Client connection to the host in question. Shortly after providing the necessary host credentials, the IP address of the vCenter Server managing this host will be revealed:

Now in theory, you could establish a Virtual Infrastructure Client connection to the IP address, however, I don’t like this because it dirties up the cached connection list with IP addresses which are meaningless short of having them all memorized. I prefer to take it a step further by opening a Command Prompt and using the command ping -a <IP_address> to reveal the name of the vCenter Server managing the host:

The command above reveals jarjar.boche.mcse as the vCenter Server which is managing the ESX(i) host I was wanting to manage via the vCenter Server.

I’m sure a PowerShell expert will follow up with a script which makes this process easier but this a good example to follow if you don’t have PowerShell or the VI Toolkit (Power CLI) installed.

1 comment »

Posted in Virtualization

Tags:

Not All FT Compatible CPUs Are Created Equal

July 10th, 2009

Hopefully you are aware that to enable VMware vSphere’s FT (Fault Tolerance), you need FT compatible CPUs from Intel or AMD.  VMware KB article 1008027 Processors and guest operating systems that support VMware Fault Tolerance outlines both the Intel and AMD CPU requirements to use FT.  I had read this article months ago and on that basis I purchased FT compatible AMD Opteron 2356 Barcelona Quad Core processor upgrades for the HP DL385 G2 servers in my lab.

While trying to enable FT on powered on VMs in the lab, I ran into issues.  The following error was thrown in the vSphere Client:

“The Fault Tolerance configuration of the entity <vm name> has an issue: The virtual machine’s current configuration does not support Fault Tolerance.”

In the vCenter Server log, the more verbose error displayed was reason = “replayNotSupported”

7-10-2009 8-55-47 PM

Oddly enough, I was able to configure FT on powered off VMs.

What I hadn’t noticed, or possibly what didn’t exist in earlier versions of this KB article was a chart at the bottom of the page with more verbose information that explained specific FT behavior based on the processor architecture.  My AMD Barcelona processors do in fact support FT, however, the chart confirms that with my processors, the VMs must be powered off first before enabling FT, whereas Intel Xeon 45nm Core 2 processors I’ve worked with in other labs allow FT to be enable while a VM is running live.  Also note in the chart below that there are FT support guidelines for specific Operating Systems as well.  For instance, a Windows 2000 VM may never be FT enabled while running, and Windows 2000 is not an FT compatible guest OS on my AMD Barcelona processors.

Following is a copy and paste of the KB article above with the FT support specifics:

Processors and guest operating systems that support VMware Fault Tolerance

Details

VMware Fault Tolerance (FT) requires specific processors and guest operating systems.

Solution

Processors

VMware collaborated with AMD and Intel in providing an efficient VMware FT capability on modern x86 processors. The collaboration required changes in both the performance counter architecture and virtualization hardware assists of both Intel and AMD. These changes could only be included in recent processors from both vendors: 3rd-Generation AMD Opteron based on the AMD Barcelona, Budapest and Shanghai processor families; and Intel Xeon processors based on the Penryn and Nehalem microarchitectures and their successors.

Download the VMware SiteSurvey (http://www.vmware.com/download/shared_utilities.html) utility to check if your configuration can run VMware FT.

For VMware FT to be supported, the servers that host the virtual machines must each use a supported processor from the same category as documented below:

Intel Xeon based on 45nm Core 2 Microarchitecture Category:

    • 3100 Series
    • 3300 Series
    • 5200 Series (DP)
    • 5400 Series
    • 7400 Series

Intel Xeon based on Core i7 Microarchitecture Category:

    • 5500 Series

AMD 3rd Generation Opteron Category:

    • 1300 Series
    • 2300 Series (DP)
    • 8300 Series (MP)

Guest Operating Systems The following table displays guest operating system support for VMware FT. For specific guest operating system version information, see the Guest Operating System Installation Guide at http://www.vmware.com/pdf/GuestOS_guide.pdf.   The following values appear in the table:

  • Yes – Virtual machine can be FT-enabled while powered on.
  • Yes/Off - Virtual machine must be powered off before FT is enabled
  • No – Not supported by VMware FT.
Guest Operating System Fault Tolerance Support with Intel Xeon Based on 45nm Core 2 Microarchitecture Fault Tolerance Support with Intel Xeon Based on Core i7 Microarchitecture Fault Tolerance Support with AMD 3rd Generation Opteron
Windows Server 2008 Yes Yes/Off Yes/Off
Windows Vista Yes Yes/Off Yes/Off
Windows Server 2003 (64 bit) Yes Yes/Off Yes/Off
Windows Server 2003 (32 bit) Yes Yes/Off Yes/Off (Requires Service Pack 2 or greater)
Windows XP (64 bit) Yes Yes/Off Yes/Off
Windows XP (32 bit) Yes Yes/Off No
Windows 2000 Yes/Off Yes/Off No
Windows NT 4.0 Yes/Off Yes/Off No
Linux (all ESX-supported distributions) Yes Yes/Off Yes/Off
Netware Server Yes/Off Yes/Off Yes/Off
Solaris 10 (64-bit) Yes Yes/Off Yes/Off (Requires Solaris U1)
Solaris 10 (32-bit) Yes Yes/Off No
FreeBSD (all ESX-supported distributions) Yes Yes/Off Yes/Off

Note: System vendors are certifying that their systems work with FT. You can find details on the FT-certified systems at http://www.vmware.com/resources/compatibility. More systems are being certified all the time, so check back if your platform is not currently listed.

You can also check processor, operating system, and virtual machine configuration compliance with FT by downloading and running the VMware SiteSurvey utility from http://www.vmware.com/download/shared_utilities.html. It highlights compliance issues and describes how to correct them.

3 comments »

Posted in Virtualization

Tags:

FT Problem Decoder Chart

July 10th, 2009

Receiving errors while trying to configure FT (Fault Tolerance) on a VM and stumped as to the reason why? This may help. Take a look at the vCenter server log in your vSphere Client and find the entry when the FT error occurred (the vCenter server log lists events in chronological order from oldest to newest, be sure to choose the correct log file as there are several to choose from). More specifically, look for the line reason = “blah blah blah”. In this case, the reason is “replayNotSupported”.

7-10-2009 8-55-47 PM

Next, open your web browser and surf to this vSphere API 4.0 link titled “Enum – VmFaultToleranceConfigIssueReasonForIssue”. This is a cross reference chart that lists common sense explanations for the “reason” code above. According to the chart, “replayNotSupported” is explained as:

“It is not possible to turn on Fault Tolerance on this powered-on VM. The support for record/replay should be enabled or Fault Tolerance turned on, when this VM is powered off.”

The root cause for the example shown above is the processors support FT, however, not while the VM is powered on. For the AMD Opteron 2356 Barcelona processors, FT is supported but the VMs must be in a powered off state to enable FT, which leads me to my next blog entry…

Here is a copy of the chart:

Name Description
ftSecondaryVm The virtual machine is a fault tolerance secondary virtual machine
ftUnsupportedHardware The host ftSupported flag is not set because of hardware issues
ftUnsupportedProduct The host ftSupported flag is not set because of it is a VMware Server 2.0
haNotEnabled HA is not enabled on the cluster
hasLocalDisk The virtual machine has one or more disks on local datastore
hasSnapshots The virtual machine has one or more snapshots
hostInactive The host is not active
missingFTLoggingNic FT logging nic is not configured on the host
missingVMotionNic No VMotion license or VMotion nic is not configured on the host
moreThanOneSecondary There is already a secondary virtual machine for the primary virtual machine
multipleVCPU The virtual machine has more than one virtual CPU
noConfig No configuration information is available for the virtual machine
recordReplayNotSupported The virtual machine does not support record/replay. Vm::Capability.RecordReplaySupported is false.
replayNotSupported It is not possible to turn on Fault Tolerance on this powered-on VM. The support for record/replay should be enabled or Fault Tolerance turned on, when this VM is powered off.
templateVm The virtual machine is a template
thinDisk The virtual machine has thin provisioned disks
verifySSLCertificateFlagNotSet The “check host certificate” flag is not set

1 comment »

Posted in Virtualization

Tags:

Update Manager does not download host updates

July 1st, 2009

Scenario: You build a brand new vCenter and Update Manager server. After the installation is complete, you decide to get a jump on things by starting the download of all the ESX/ESXi host updates. You force Update Manager to download updates and the task completes surprisingly fast for the amount of ESX/ESXi content expected to be downloaded:

7-1-2009 8-54-08 PM

A problem is discovered in that Update Manager has downloaded metadata for guest OS updates (Windows, Linux, applications, etc.), but no ESX/ESXi update information is downloaded. The baselines are verified as OK, internet connectivity and proxy configuration checks out OK. What is the problem?

Cause: There are no ESX/ESXi hosts in vCenter Server. Per VMware KB 1008308, ESX/ESXi hosts must be present in vCenter Server before Update Manager will download the update metadata and the updates themselves.

7-1-2009 9-00-41 PM

This is one of those embarrassing forehead slapper type problems, however, Windows administrators who are used to working with and relying on the predicable behavior of WSUS are likely to encounter this at some point in time and are exempt from chastising. Swallow your pride and don’t tell anyone. :)

VMware is entitled to their opinion on how their software should function but to me this is a UI/usability issue that doesn’t make a lot of sense. What adds to the confusion is the inconsistent behavior in that in the absence of both hosts and guests in vCenter Server, guest OS update information appears in Update Manager but host update information does not. Yes I’m aware that host updates come from VMware and guest updates come from Shavlik.  No that’s not an acceptable excuse.

While we’re on the subject, there are a handful of other reasons why Update Manager may malfunction. Take a look at VMware’s KB index and use your browser search to find all instances of “Update Manager”. There you’ll find all known solutions to Update Manager issues as well as some best practices and port requirements.

1 comment »

Posted in Virtualization

Tags:

vSphere upgrade experience, day 1

June 24th, 2009

A few nights ago, I began the VI3 to vSphere upgrade in my home lab and I thought I would share a few experiences. This day 1 post will cover vSphere management tools (vCenter, Update Manager, etc.) and not the hypervisor itself (ESX or ESXi).

My VI3 environment has been through some wear and tear over the years, including a few unexpected power outages which could have caused corruption on the vCenter server or the back end databases. Although the part of me which desires peace of mind wanted to start “clean” with an empty database, I knew that I must go the upgrade route, maintaining my existing data because frankly this is the method I will likely be using to deploy most vSphere installations.

I run a lot of what I would consider “production workloads” on my home lab, including domain controllers, messaging servers for registered domains, web servers, Citrix servers, this blog, etc. Failure is an option as well as a good learning experience (after all, this is a lab), however, long term outage of my production workloads is not an option. My vCenter server is virtualized on VMware Server 2.x so I started out by shutting down its OS and taking a VMware snapshot. After the vCenter shutdown, I also captured a full backup of my SQL server databases (both the vCenter database as well as the Update Manager database). I now have a solid backout plan which does not incorporate crash consistent data.

I powered the vCenter VM back up. I then copied over the vCenter 4.0 .zip package and extracted it into a temp directory on the vCenter server. This was the first mistake I made. Not thinking clearly about my snapshotted VM, I had just inflated the VM’s delta file by a few GB. What I should have done is to perform the vCenter copy and extraction before the snapshot. This is not the end of the world. After the installation of vCenter 4 and Update Manager, the snapshot would have grown by several hundred MBs if not a few GBs anyway. The .zip file and extracted contents were just a lot of extra non-contiguous I/O baggage.

I’m going to perform an upgrade of the databases, but I don’t care to actually “upgrade” vCenter and all of its components from 2.5 Update 4 to version 4.0. I’ve never ever had good luck with vCenter upgrades. My method, therefore, is complete uninstall of vCenter and all components, then a new installation of vCenter while attaching to the existing database which will in turn be upgraded. During the uninstall of vCenter, I typically find that the vCenter uninstall routine leaves bits and pieces behind in folder structures as well as the registry. I manually deleted the remaining pieces and rebooted the vCenter server for good measure and a clean start for the vCenter 4.0 installation. In retrospect, the manual deletion of left over files and uninstall of the vCenter license server will turn out to be my second and third mistakes which I’ll talk about shortly.

After the reboot, I began the vCenter 4.0 installation. I also made sure my vCenter SQL account had DBO rights to the MSDB database, the vCenter database, and the Update Manager database. This is a new requirement during the installation of vSphere. I wasn’t too far into the installation when I ran into failure and the installation routine rolled back. The error message was rather cryptic and I’m sorry I don’t have a screenshot but it had to do with the installer’s inability to properly install and configure the local ADAM instance which I believe is used for vCenter federation (linked vCenter servers). I quickly found a long thread on the VMTN forums which pointed me to the solution in VMware’s knowledgebase. KB1010938 talks about NETWORK SERVICE NTFS directory permissions (READ) that are required on the root of the drive where vCenter is being installed. A quick check showed I lacked the necessary permissions. I resolved this quickly and re-ran the installation.

During the re-installation, I ran into my second problem (self inflicted). Way back when, I had set up SSL certificates for VI3. The certificate files are required to be present during the database upgrade because the certs are tattooed to the database as well. During my “cleanup” process I spoke about above, I had deleted the SSL folder containing the certificate files VMware had left behind. Turns out this was by design. Thankfully when I performed the cleanup, all files and folders went to the recycle bin and I was able to quickly retrieve them. Without the certificate files, I would have been looking at a new database installation which would have deleted all vCenter data including performance history.

After restoring the certificate files, I reran the installation a third time. The installation of vCenter Server and all of its components was successful. I was able to open the vSphere client and connect to the vCenter server. My hosts, VMs, and all data was present. All looked to be successful until I tried a VMotion. The ESX hosts which were still on VI3 were no longer licensed. Refer to my comment further up about uninstalling the license server being a mistake. vCenter 4.0 license keys do not license VI3 legacy hosts. A VI3 license server or host based license keys must be plugged into vCenter 4.0 in order to properly license VI3 legacy hosts. I resolved the issue by re-installing the VI3 license server on some junk VM in the interim and then plugged the license server name into vCenter 4.0′s licensing configuration. Viola! The ESX3 and ESXi3 hosts are now licensed and everything is working properly. After feeling confident in the installation, I removed the vCenter snapshot.

This was enough change for one night. The ESX host upgrades (rebuilds) will come a few days later. If I uncover any gotchas during host installations, I’ll be sure to share but I expect those to be fairly uneventful. I’ve installed a lot of ESX4/ESXi4 hosts during the vSphere beta and it’s straight forward, not unlike ESX3 /ESXi3 installations. Most of the ~150 changes in vSphere will be evident in vCenter and the various components. There’s a few enhancements in the hypervisor installation but nothing that hasn’t already been pointed out in various other blogs and installation videos.

6 comments »

Posted in Virtualization

Tags: