Posts Tagged ‘vCenter Server’

Watch VMware Raise the Bar on July 12th

July 11th, 2011

On Tuesday July 12th, VMware CEO Paul Maritz and CTO Steve Herrod are hosting a large campus and worldwide event where they plan to make announcements about the next generation of cloud infrastructure.

The event kicks off at 9am PDT and is formally titled “Raising the Bar, Part V”. You can watch it online by registering here.  The itinerary is as follows:

  • 9:00-9:45 Paul and Steve present – live online streaming
  • 10:00-12:00 five tracks of deep dive breakout sessions
  • 10:00-12:00 live Q&A with VMware cloud and virtualization experts
    • Eric Siebert
    • David Davis
    • Bob Plankers
    • Bill Hill

In addition, by attending live you also have the chance to win a free VMworld pass.  More details on that and how to win here.

I’m pretty excited both personally and for VMware.  This is going to be huge!

Performance Overview charts fail with STATs Report Service internal error

May 11th, 2011

A few months ago I was troubleshooting a problem with the Overview charts in the Performance tab of the vSphere Client.  This was a vSphere 4.0 Update 1 environment but I believe the root cause will impact other vSphere versions as well.

Instead of displaying the dashboard of charts in the Overview display, an error was displayed:

STATs Report service internal error
or
STATs Report application initialization is not completed successfully

One unique aspect of this environment was that the vCenter database was hosted on a Microsoft SQL Server which used a port other than the default of TCP 1433.  VMware KB Article 1012812 identified this as the root cause of the issue.

To resolve the issue, I was required to stop the vCenter Server service and modify the statsreport.xml file located on the vCenter Server in the \Program Files\VMware\Infrastructure\tomcat\conf\Catalina\localhost\ directory by inserting the line in bold.  Note the italicized components will vary and are environment specific based on the SQL server name, database name, alternate TCP port in use, and authentication method (SQL/false or Windows integrated/true):

<Resource auth=”Container”
   name=”jdbc/StatsDS”
   type=”javax.sql.DataSource”
   factory=”org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory”
   initialSize=”3″
  maxActive=”10″
  maxIdle=”3″
  maxWait=”10000″
  defaultReadOnly=”true”
  defaultTransactionIsolation=”READ_COMMITTED”
  removeAbandoned=”true”
  removeAbandonedTimeout=”60″
  url=”jdbc:sqlserver://sqlservername:1601;instanceName=sqlservername;
     databaseName=sqldatabasename;integratedSecurity=false;”
/>

Don’t forget to restart the vCenter Server service after saving the statsreport.xml file.

vSphere Integration With EMC Unisphere

February 14th, 2011

SnagIt CaptureIf you manage EMC unified storage running at least FLARE 30 and DART 6, or if you’re using a recent version of the UBER VSA, or if you’re one of the fortunate few who have had your hands on the new VNX series, then chances are you’re familiar with or you’ve at least experienced Unisphere, which is EMC’s single pane of glass approach to managing its multi protocol arrays.  For what is essentially a 1.0 product, I think EMC did a great job with Unisphere.  It’s modern.  It’s fast.  It has a cool sleek design and flows well.  They may have cut a few corners where it made sense (one can still see a few old pieces of Navisphere code here and there) but what counts for me the most at the end of the day is the functionality and efficiency gained by a consolidation of tools.

You’re probably reading this because you have a relationship with VMware virtualization.  Anyone who designs, implements, manages, or troubleshoots VMware virtual infrastructure also has a relationship with storage, most often shared storage.  Virtualization has been transforming the datacenter, and not just it’s composition.  The way we manage and collaborate from a technology perspective is also evolving.  Virtualization has brought about an intersection of technologies which is redefining roles and delegation of responsibilities.  One of the earlier examples of this was virtual networking.  With the introduction of 802.1Q VST in ESX, network groups found themselves fielding requests for trunked VLANs to servers and having to perform the associated design, capacity, and security planning.  Managing access to VLANs was a shift in delegated responsibility from the network team to the virtualization platform team.  Some years later, implementation of the Cisco Nexus 1000V in vSphere pulled most of the network related tasks back under the control of the network team.

Storage is another broad reaching technology upon which most of today’s computing relies upon, including virtualization.  Partners work closely with VMware to develop tools which provide seamless integration of overlapping technologies.  Unisphere is one of several products in the EMC portfolio which boasts this integration.  Granted, some of these VMware bits existed in Unisphere’s ancestor Navisphere.  However, I think it’s still worth highlighting some of the capabilities found in Unisphere.  EMC has been on an absolute virtualization rampage.  I can only imagine that with their commitment, these products will get increasingly better.

So what does this Unisphere/vSphere integration look like?  Let’s take a look…

In order to bring vSphere visibility into Unisphere, we need to make Unisphere aware of our virtual environment.  From the Host Management menu pane in Unisphere, choose Hypervisor Information Configuration Wizard:

SnagIt Capture

Classic welcome to the wizard.  Next:

SnagIt Capture

Select the EMC array in which to integrate a hypervisor configuration:

SnagIt Capture

In the following screen, we’re given the option to integrate either standalone ESX(i) hosts, vCenter managed hosts, or both.  In this case, I’ll choose vCenter managed hosts:

SnagIt Capture

Unisphere needs the IP address of the vCenter Server along with credentials having sufficient permissions to collect virtual infrastructure information.  FQDN of virtual infrastructure doesn’t work here (Wish list item), however, hex characters are accepted which tells me it’s IPv6 compatible:

SnagIt Capture

I see your infrastructure.  Would you like to add or remove items?

SnagIt Capture

Last step.  This is the virtual infrastructure we’re going to tie into.  Choose Finish:

SnagIt Capture

Congratulations.  Success.  Click Finish once more:

SnagIt Capture

Once completed, I see that the vCenter server I added has nested in the ESX host which it manages.  Again we see only the IP address representing a vCenter Server, rather than the FQDN itself.  This could get a little hairy in larger environments where a name is more familiar and friendlier than an IP address.  However, in Unisphere’s defense, at the time of adding a host we do have the option of adding a short description which would show up here.  Highlighting the ESX host reveals the VMs which are running on the host.  Nothing Earth shattering yet, but the good stuff lies ahead:

SnagIt Capture

Let’s look at the ESX host properties.  Here’s where the value starts to mount (storage pun intended).  The LUN Status tab reveals information of LUNs in use by the ESX host, as well as the Storage Processor configuration and status.  This is useful information for balance and performance troubleshooting purposes:

SnagIt Capture

Moving on to the Storage tab, more detailed information is provided about the LUN characteristics and how the LUNs are presented to the ESX host:

SnagIt Capture

The Virtual Machines tab is much the same as the VMware Infrastructure summary screen with the information that it provides.  However, it does provide the ability to drill down to specific VM information by way of hyperlinks:

SnagIt Capture

Let’s take a look at the VM named vma41 by clicking on the vma41 hyperlink from the window above.  The General tab provides some summary information about the VM and the storage, but nothing that we probably don’t already know at this point.  Onward:

SnagIt Capture

The LUN Status tab provides the VM to storage mapping and Storage Processor.  Once again, this is key information for performance troubleshooting.  Don’t get me wrong.  This information alone isn’t necessarily going to provide conclusive troubleshooting data.  Rather, it should be combined with other information collected such as  storage or fabric performance reports:

SnagIt Capture

Similar to the host metrics, the Storage tab from the VM point of view provides more detailed information about the datastore as well as the VM disk configuration.  Note the Type column which shows that the VM was thinly provisioned:

SnagIt Capture

There are a few situations which can invoke the age old storage administrator’s question: “What’s using this LUN?”  From the Storage | LUNs | Properties drill down (or from Storage | Pools/RAID Groups), Unisphere ties in the ESX hosts connected to the LUN as well as the VMs  living on the LUN.  Example use cases where this information is pertinent would be performance troubleshooting, storage migration or expansion, replication and DR/BCP planning.

SnagIt Capture

VM integration also lends itself to the Unisphere Report Wizard.  Here, reports can be generated for immediate display in a web browser, or they can be exported in .CSV format to be massaged further.

SnagIt Capture

If you’d like to see more, EMC has made available a three minute EMC Unisphere/VMware Integration Demo video which showcases integration and the flow of information:

In addition to that, you can download the FREE UBER VSA and give Unisphere a try for yourself.  Other EMC vSpecialist demos can be found at Everything VMware At EMC.

With all of this goodness and as with any product, there is room for improvement.  I mentioned before that by and large the vSphere integration code appears to be legacy which came from Navisphere.  Navisphere manages CLARiiON block storage only (fibre channel and native CLARiiON iSCSI).  What this means is that there is a gap in Unisphere/vSphere integration with respect to Celerra NFS and iSCSI.  For NFS, EMC has a vSphere plugin which Chad Sakac introduced about a year ago on his blog here and here.  While it’s not Unisphere integration, it does do some cool and useful things which are outlined in this product overview

In medium to large sized environments where teams can be siloed, it’s integration like this which can provide a common language, bridging the gap between technologies which have close dependencies with one another.  These tools work in the SMB space as well where staff will have both virtualization and storage areas of responsibility.  vSphere integration with Unisphere can provide a fair amount insight and efficiency.  I think this is just a slight representation of what future integration will be capable of.  VMware’s portfolio of virtualization, cloud, and data protection products continues to expand.  Each and every product VMware delivers is dependent on storage.  There is a tremendous opportunity to leverage each of these attach points for future integration.

VMware Releases vSphere 4.1 Update 1

February 10th, 2011

I’ve just been informed by my VMware Update Manager (VUM) that VMware has released vSphere 4.1 Update 1, including:

  • vCenter 4.1 Update 1
  • ESXi 4.1 Update 1
  • ESX 4.1 Update 1
  • vShield Zones 4.1 Update 1??

Will this be the last release of the ESX hypervisor in history?  Thus far, I haven’t seen that the HP, IBM, and Dell versions of ESXi 4.1 Update 1 are available for download yet.  They typically follow the VMware GA release by a few weeks.

Grab your copy now!

SnagIt Capture

The number of patch definitions downloaded (15 critical/28 total):

ID: ESX410-201101201-SG  Impact: HostSecurity  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 Core and CIM components

ID: ESX410-201101202-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 VMware-webCenter

ID: ESX410-201101203-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 esxupdate library

ID: ESX410-201101204-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 mptsas device driver

ID: ESX410-201101206-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 bnx2xi device driver

ID: ESX410-201101207-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 bnx2x device driver

ID: ESX410-201101208-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 sata device driver

ID: ESX410-201101211-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 VMware-esx-remove-rpms

ID: ESX410-201101213-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 net-enic device driver

ID: ESX410-201101214-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 qla4xxx device driver

ID: ESX410-201101215-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 net-nx-nic device driver

ID: ESX410-201101216-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 vaai plug-in

ID: ESX410-201101217-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 e1000e device driver

ID: ESX410-201101218-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 net-cdc-ether driver

ID: ESX410-201101219-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 e1000 device driver

ID: ESX410-201101220-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 igb, tg3, scsi-fnic

ID: ESX410-201101221-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 HP SAS Controllers

ID: ESX410-201101222-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 mptsas, mptspi drivers

ID: ESX410-201101223-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0

3w-9xxx: scsi driver for VMware ESX

ID: ESX410-201101224-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0

vxge: net driver for VMware ESX

ID: ESX410-201101225-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates vmware-esx-pam-config library

ID: ESX410-201101226-SG  Impact: HostSecurity  Release date: 2011-02-10  Products: esx 4.1.0 Updates glibc packages

ID: ESX410-Update01  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 VMware ESX 4.1 Complete Update 1

ID: ESXi410-201101201-SG  Impact: HostSecurity  Release date: 2011-02-10  Products: embeddedEsx 4.1.0 Updates the ESXi 4.1 firmware

ID: ESXi410-201101202-UG  Impact: Critical  Release date: 2011-02-10  Products: embeddedEsx 4.1.0 Updates the ESXi  4.1 VMware Tools

ID: ESXi410-201101223-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: embeddedEsx 4.1.0

3w-9xxx: scsi driver for VMware ESXi

ID: ESXi410-201101224-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: embeddedEsx 4.1.0

vxge: net driver for VMware ESXi

ID: ESXi410-Update01  Impact: HostGeneral  Release date: 2011-02-10  Products: embeddedEsx 4.1.0 VMware ESXi 4.1 Complete Update 1

Performance charts fail after Daylight Savings changes are applied

November 5th, 2010

Daylight savings changes this weekend allow many folks to get an extra hour of sleep.  However, a VMware vSphere 4.1 bug has surfaced which may spoil the fun. 

VMware has published KB 1030305 (Performance charts fail after Daylight Savings changes are applied) which serves as a reminder that the pitfalls and treachery of mixing daylight savings changes and million dollar datacenters are not behind us yet.  Those who are on vSphere 4.1 and observe the weekend time change will run into problems come Sunday morning:

After Daylight Savings settings are applied:

  • Performance charts do not display data
  • Past week, month, and year performance overview charts are not displayed
  • Datastore performance/space data charts are not displayed
  • You receive the error: The chart could not be loaded
  • This occurs when clocks are set back 1 hour from Daylight Savings Time to Standard Time

VMware offers the following workaround:

Use Advanced Chart Options:

  1. Click Performance
  2. Click Advanced
  3. Click Chart Options and then choose the chart you want to review

Use a custom time range when viewing performance charts after clocks are set back:

  1. Click Performance
  2. Click the Time Range dropdown
  3. Choose Custom
  4. Specify From and To options that exclude the hours for when the time change occurred

For example:

If Standard Time settings were applied on November 7, at 01:00 AM, you could use these ranges:
Before the time change:
From 1/11/2010 12:00 AM To 7/11/2010 12:00 AM
After the time change:
From 7/11/2010 03:00 AM To 8/11/2010 15:00 PM

Have a great weekend!

vCenter MAC Address allocation and conflicts

November 3rd, 2010

Paul Davey of Xtravirt published a VMware networking article a few weeks ago called vCenter MAC Address allocation and conflicts.  The article describes the mechanism behind MAC address assignments in vCenter, and more specifically how conflicts are avoided:

When a vCenter server is installed a unique ID is generated. This ID is randomly generated and is in the range of 0 to 64. The ID gets used when generating MAC address and the UUIDS, or unique identifiers for virtual machines. You can see that if two vCenter servers had the same unique ID, a possibility exists that duplicate MAC addresses might get generated; cue packet loss, connectivity issues and your desk phone ringing a lot…

 SnagIt Capture

I receive email updates from Xtravirt at regular intervals – about one every week or two.  Each update contains new virtualization content as well as links to their most popular content.  I find the content very interesting and always look forward to opening new email from them.  I think this speaks volumes considering how much of a chore email can be at times.  If this sounds appealing to you, head to their site and look at the bottom of the page to sign up for the Xtravirt Newsletter.   No purchase or invitation necessary.

Updated 3/14/11: I thought this might also be helpful for this article.  VMware explains the automatic MAC address generation process as follows:

The VMware Universally Unique Identifier (UUID) generates MAC addresses that are checked for conflicts. The generated MAC addresses are created by using three parts: the VMware OUI, the SMBIOS UUID for the physical ESXi machine, and a hash based on the name of the entity that the MAC address is being generated for.

VMware DPM Issue

October 24th, 2010

I’ve been running into a DPM issue in the lab recently.  Allow me briefly describe the environment:

  • 3 vCenter Servers 4.1 in Linked Mode
  • 1 cluster with 2 hosts
    • ESX 4.1, 32GB RAM, ~15% CPU utilization, ~65% Memory utilization, host DPM set for disabled meaning the host should never be placed in standby by DPM.
    • ESXi 4.1, 24GB RAM, ~15% CPU utilization, ~65% Memory utilization, host DPM set for automatic meaning the host is always a candidate to be placed in standby by DPM.
  • Shared storage
  • DRS and DPM enabled for full automation (both configured at Priority 4, almost the most aggressive setting)

Up until recently, the ESX and ESXi hosts weren’t as loaded and DPM was working reliably.  Each host had 16GB RAM installed.  When aggregate load was light enough, all VMs were moved to the ESX host and the ESXi host was placed in standby mode by DPM.  Life was good.

There has been much activity in the lab recently.  The ESX and ESXi host memory was upgraded to 32GB and 24GB respectively.  Many VMs were added to the cluster and powered on for various projects.  The DPM configuration remained as is.  Now what I’m noticing is that with a fairly heavy memory load on both hosts in cluster, DPM moves all VMs to the ESX host and places the ESXi host in standby mode.  This places a tremendous amount of memory pressure and over commit on the solitary ESX host.  This extreme condition is observed by the cluster and nearly as quickly, the ESXi host is taken back out of standby mode to balance the load.  Then maybe about an hour later, the process repeats itself again.

I then configured DPM to manual mode so that I could examine the recommendations being made by the calculator.  The VMs were being evacuated for the purposes of DPM via a Priority 3 recommendation which is half way between Conservative and Aggressive recommendations.

SnagIt Capture

What is my conclusion?  I’m surprised at the perceived increase in aggressiveness of DPM.  In order to avoid the extreme memory over commit, I’ll need to configure DPM slide bar for Priority 2.  In addition, I’d like to get a better understanding of the calculation.  I have a difficult time believing the amount of memory over commit being deemed acceptable in a neutral configuration (Priority 3) which falls half way between conservative and aggressive.  In addition to that, I’m not a fan of a host continuously entering and exiting standby mode, along with the flurry of vMotion activity which results.  This tells me that the calculation isn’t accounting for the amount of memory pressure which is actually occurring once a host goes into standby mode, or coincidentally there are significant shifts in the workload patterns shortly after each DPM operation.

If you are a VMware DPM product manager, please see my next post Request for UI Consistency.