Posts Tagged ‘vCenter Server’

VMware vCenter as a vCloud Director vApp

February 27th, 2012

Snagit CaptureThe way things work out, I tend to build a lot of vCenter Servers in the lab.  Or at least it feels like I do.  I need to test this.  A customer I’m meeting with wants to specifically see that.  I need don’t want to taint or impact an existing vCenter Server which may already be dedicated to something else having more importance.  VMware Site Recovery Manager is a good example.  Each time I bring up an environment I need a pair of vCenter Servers which may or not be available.  Whatever the reason, I’ve reached the point where I don’t need to experience the build process repeatedly.

The Idea

A while ago, I had stood up a private cloud for the Technical Solutions/Technical Marketing group at Dell Compellent.  I saved some time by leveraging that cloud environment to quickly provision platforms I could install vCenter Server instances on.  vCenter Servers as vApps – fantastic use case.  However, the vCenter installation process is lengthy enough that I wanted something more in terms of automated cookie cutter deployment which I didn’t have to spend a lot of time on.  What if I took one of the Windows Server 2008 R2 vApps from the vCD Organization Catalog, deployed it as a vApp, bumped up the vCPU and memory count, installed the vSphere Client, vCenter Server, licenses, a local MS SQL Express database, and the Dell Compellent vSphere client plug-in (download|demo video), and then added that vApp back to the vCD Organization Catalog?  Perhaps not such a supported configuration by VMware or Microsoft, but could I then deploy that vApp as future vCenter instances?  Better yet, build a vApp consisting of a pair of vCenter Servers for the SRM use case?  It sounded feasible.  My biggest concerns were things like vCenter and SQL Express surviving the name and IP address change as part of the vCD customization.


Although I ran into some unrelated customization issues which seemed to have something to do with vCD, Windows Server 2008 R2, and VMXNET3 vNICs (error message: “could not find network adapters as specified by guest customization. Log file is at c:\windows\temp\customize-guest.log.” I’ll save that for a future blog post if I’m able to root cause the problem), the Proof of Concept test results thus far have been successful.  After vCD customization, I was able to add vSphere 5 hosts and continue with normal operations from there.

Initially, I did run into one minor issue and that was hosts would fall into a disconnected status approximately two minutes after being connected to the vCenter Server.  This turned out to be a Windows Firewall issue which was introduced during the customization process.  Also, there were some red areas under the vCenter Service Status which pointed to the old instance name (most fixes for that documented well by Rick Vanover here, plus the vCenter Inventory Service cleanup at VMware KB 2009934).

The Conclusion

To The Cloud!  You don’t normally hear that from me on a regular basis but in this case it fits.  A lengthy and increasingly cumbersome task was made more efficient with vCloud Director and vSphere 5.  Using the Linked Clone feature yields both of its native benefits: Fast Provisioning and Space Efficiency.  I’ll continue to leverage vCD for similar and new use cases where I can.  Lastly, this solution can also be implemented with VMware Lab Manager or simply as a vSphere template.  Caveats being Lab Manager retires in a little over a year and a vSphere Template won’t be as space efficient as a Linked Clone.

View Pool And Desktop Hung in Deleting State

February 7th, 2012

The VMware View 5.0 environment in the lab has been running well and has proven itself as an extremely reliable remote access replacement for the old Citrix Presentation Server 4.0 solution I had in the past.  However, in an effort to address a licensing issue related to the View App for iPad demo environment, I managed to force both a pool and a single desktop from within that pool into a perpetually stuck state of ‘deleting’.  In addition, the VM representing the desktop was gone, but I could see from within vCenter the parent replica for the pool still remained.  I spent some time poking at it from several angles including the View Connection Server, the vCenter Server, and the View Composer Server.  It became clear that the underlying issue was deeper, in a database perhaps, and couldn’t be resolved using the standard management tools VMware offers.

The issue isn’t an uncommon one and I quickly turned up familiar hits at VMware’s community forums spanning a few different versions of VMware View.  The root cause is explained in VMware KB Article 1008658 which applies to View versions 3.x through 5.2.x (this KB article also walks through the steps of manually deleting pae-VMs from the ADAM database as well as manually deleting dubious SVI_rows from various tables in the View Composer database as necessary).  Also see Manually deleting linked clones or stale virtual desktop entries from VMware View Manager 4.5 and later at VMware KB 2015112 (which uses the SviConfig.exe View Composer utility and is linked from KB 1008658):

This issue occurs if a table in the database has an incorrect data. It can also occur if the virtual machine name has been changed in the vCenter Server manually after the pool has been created, causing View Composer and vCenter Server to refer to the same virtual machine with different names.

The problem can largely be avoided by managing the View environment with the intended tool – the VMware View Administrator interface as opposed to making changes outside of View, such as using VMware vCenter.

SnagIt Capture SnagIt Capture


Resolving the issue is achieved by following the detailed in the KB articles above.  Follow the steps carefully and slowly in a production View environment and keep in mind that not all steps may be required for your particular situation.

VCA4-DT and VCP5 Exam Reviews

January 6th, 2012

With 2011 wrapped up and the holiday festivities over with, I decided to kick off 2012 by sitting a few new VMware certification exams.  Before I get into the details of the exam experience, I must extend my sincere appreciation to the new testing center I tried out – New Horizons Computer Learning Center in Eagan, MN.  It’s a new facility, friendly staff, state of the art equipment, AND THEY ALLOW COFFEE IN THE EXAM ROOM!  I’m locked on to this facility for all future exams.

Ok, VMware Certified Associate 4 – Desktop, otherwise known as VCA4-DT.  Thursday morning, 70 questions, multiple choice, 90 minutes if I remember right. Time isn’t much of a factor on this exam as it has been in past exams I’ve sat.  Unfortunately I failed by a narrow margin. 289/500 (300 is the passing mark).  Not passing was a bummer since I’ve only failed one other exam and that was 14 years ago.  The reality was that I hadn’t had enough View Administrator seat time to recall what was being tested.  I can’t go into specifics but I will say that having a photographic memory of the View Admin console will go a long way to get by this exam.  I’ve managed a tiny View 4.6 and now 5.0 environment in my lab but I haven’t spent countless hours in the console on a day to day basis which is what I think is really required.  That makes sense – after all it is an Administrator role based exam.  My hope was that brushing up on the blueprint objectives and reading Mike Laverick’s Administrating VMware View 4.5 book cover to cover the night before the exam would have been enough to get by.  It wasn’t.  No fault to Mike of course, his was a fine book.  I planned short on the preparation, rolled the dice, and.. well you know by now what happened.  It was a humbling experience but at the same time it’s an effective method to learn more.  After I get back from Dell Storage Forum London I’ll plan on hitting the lab and ultimately finishing the exam the proper way.  After that, I’ve got my sights set on VMware Certified Professional 4 – Desktop (VCP4-DT) which I may already be better prepared for.

On to the VMware Certified Professional 5 or VPC5.  Friday morning, 85 questions, multiple choice, 90 minutes.  I reached the end of the exam with 14 minutes left to review marked questions – I had quite a few.  I don’t know why – I rarely change my answer when reviewing questions.  I mark the questions with the intent that there may be a better answer which comes to me later on in the exam but it rarely happens and I believe statistics prove that on average, first instinct is going to be the better or correct answer.  I’ll be honest, dwelling on yesterday’s fail did a number to my confidence level but I had no choice but to push forward studying the blueprint for a solid 8 hours last night into the wee hours of the morning.  Granted, the VCP5 exam should be higher on the difficulty level, but the infrastructure content maps quite a bit better to my expertise that VMware View administration does.  I had seen some comments from others that the VCP5 exam didn’t contain much along the lines of Configuration Maximums type questions.  Based on that, I didn’t spend much time in the vSphere 5 Configuration Maximums document.  I brushed up on HA, DRS, and although I have little hands on working experience with the appliance based bolt ons like the vCenter appliance, vDR, VSA, or Auto Deploy, I tried to pick up as much as I could on those areas.  On exam difficulty, the content came easier to me based on familiarity.  For most of the exam I was pretty well within my comfort zone.  As a Technical Marketing Product Specialist at Dell Compellent, the storage related questions aren’t quite the level of difficulty they once were.  There was a pretty good blend of easy/medium/difficult questions, and also a few which I felt were worded poorly enough such that I knew the correct answer either way, but interpretation of the question is going to determine a right or wrong answer.  Results on this exam were better – 406/500 (300 passing).  There were plenty of questions on the other vSphere products I talked about earlier such as the vCenter appliance, vDR, VSA, and Auto Deploy. While I feel I did answer a few of those questions correctly, the remainder is likely what accounts for the majority of the points I missed on the exam.  By the way, if you’re not using vSphere Update Manager on a regular basis to assist in upgrading your environment, you should be, and you’ll want to know that product for this exam as well.

Have a great weekend and for those attending Dell Storage Forum London next week, I hope to meet up with you.

vCloud Director and vCenter Proxy Service Failure

December 16th, 2011

In the past couple of weeks I have spent some time with VMware vCloud Director 1.5.  The experience yielded three blog articles Collecting diagnostic information for VMware vCloud Director, Expanding vCloud Director Transfer Server Storage and now this one.

In this round, the vCD cell stopped working properly (single cell server environment).  I could log into the vCD provider and organization portals but the deployment of vApps would run for an abnormally long time and then fail after 20 minutes with one of the resulting failure messages being Failed to receive status for task.

Doing some digging in the environment I found a few problems.

Problem #1:  None of the cells have a vCenter proxy service running on the cell server.

Snagit Capture

Problem #2:  Performing a Reconnect on the vCenter Server object resulted in Error performing operation and Unable to find the cell running this listener.

Snagit Capture

Snagit Capture

I search the Community Forums, talked with Chris Colotti (read his blog) for a bit, and then opened an SR with VMware support.  VMware sent me a procedure along with a script to run on the Microsoft SQL Server:

  1. BACKUP the entire SQL Database.
  2. Stop all cells. (service vmware-vcd stop)
  3. Run the attached reset_qrtz_tables_sql_database.sql
    — shutdown all cells before executing
    delete from qrtz_scheduler_state
    delete from qrtz_fired_triggers
    delete from qrtz_paused_trigger_grps
    delete from qrtz_calendars
    delete from qrtz_trigger_listeners
    delete from qrtz_blob_triggers
    delete from qrtz_cron_triggers
    delete from qrtz_simple_triggers
    delete from qrtz_triggers
    delete from qrtz_job_listeners
    delete from qrtz_job_details
  4. Start one cell and verify if issue is resolved. (service vmware-vcd start)
  5. Start the remaining cells.

Before running the script I knew I had to make a few modifications to select the vCloud database first.

When running the script, it failed due to case sensitivity with respect to the table names.  Upon installation, vCD creates all tables with upper case names.  When the MS SQL Server database was first created by yours truly, case sensitivity, along with accent sensitivity, were enabled with COLLATE Latin1_General_CS_AS which comes straight from page 17 of the vCloud Director Installation and Configuration Guide.

After fixing the script, it looked like this:

USE [vcloud]

— shutdown all cells before executing
delete from QRTZ_CALENDARS
delete from QRTZ_TRIGGERS
delete from QRTZ_JOB_DETAILS

The script ran successfully wiping out all rows in each of the named tables.  A little sidebar discussion here.. I talked with @sqlchicken (Jorge Segarra, read his blog here) about the delete from statements in the script. It is sometimes a best practice to use the truncate table statement instead so that the transaction logs are bypassed instead of using the delete from statement which is more resource intensive due to the row by row deletion method and the rows being recorded in the transaction logs. Thank you for that insight Jorge! More on MS SQL Delete vs Truncate here. Jorge was also kind enough to provide a link on the subject matter but credentials will be required to view the content.

I was now able to restart the vCD cell and my problems were gone. Everything was working again. All errors have vanished.  I thanked the VMware support staff and then tried to gain a little bit more information about how the problem was resolved by deleting table rows and what exactly are the qrtz tables?  I had looked at the table rows myself before they were deleted and the information in there didn’t make a lot of sense to me (but that doesn’t necessarily classify it as transient data).  This is what he had to say:

These [vCenter Proxy Service] issues are usually caused by a disconnect from the database, causing the tables to become stale. vCD constantly needs the ability to write to the database and when it cannot, the cell ends up in a state that is similar to the one that you have seen.

The qrtz tables contain information that controls the coordinator service, and lets it know when the coordinator to be dropped and restarted, for cell to cell fail over to another cell in multi cell enviroment.

When the tables are purged it forces the cell on start up to recheck its status and start the coordinator service. In your situation the cell, due to corrupt records in the table was not allowing this to happen.

So by clearing them forced the cell to recheck and to restart the coordinator.

Good information to know going forward. I’m going to keep this in my back pocket. Or on my blog as it were.  Have a great weekend!

VMware vSphere 4.1 Update 2 Released

October 27th, 2011

As I sit here working on an SRM lab, VUM just sent an email to me reporting 28 new patches for ESX(i) 4.1 including the release of 4.1 Update 2.

What’s New

The VMware vCenter Server 4.1 Update 2 release offers the following improvements:

  • Support for new processors: vCenter Server 4.1 Update 2 supports hosts with processors on AMD Opteron 6200 series (Interlagos) and AMD Opteron 4200 series (Valencia).
    Note: For the AMD Opteron 6200 and 4200 series (Family 15h) processors, vCenter Server 4.1 Update 2 treats each core within a compute unit as an independent core, except while applying licenses. For the purpose of licensing, vCenter Server treats each compute unit as a core. For example, although a processor with 8 compute units can provide the processor equivalent of 16 cores on vCenter Server 4.1 Update 2, only 8 cores will be counted towards license usage calculation.
  • Additional vCenter Server Database Support: vCenter Server now supports the following databases.
    • Microsoft SQL Server 2008 Express (x32 and x64)
    • Microsoft SQL Server 2008 R2 Express (x32 and x64)
  • Resolved Issues: This release delivers a number of bug fixes that have been documented in the Resolved Issues section.

What’s New

The following information describes some of the enhancements available in this release of VMware ESXi:

  • Support for new processors – ESXi 4.1 Update 2 supports AMD Opteron 6200 series (Interlagos) and AMD Opteron 4200 series (Valencia).Note: For the AMD Opteron 6200 and 4200 series (Family 15h) processors, ESX/ESXi 4.1 Update 2 treats each core within a compute unit as an independent core, except while applying licenses. For the purpose of licensing, ESX/ESXi treats each compute unit as a core. For example, although a processor with 8 compute units can provide the processor equivalent of 16 cores on ESX/ESXi 4.1 Update 2, it only uses 8 licenses.
  • Support for additional guest operating system ESX 4.1 Update 2 adds support for Ubuntu 11.10 guest operating system. For a complete list of guest operating systems supported with this release, see the VMware Compatibility Guide.

Resolved Issues In addition, this release delivers a number of bug fixes that are documented in the Resolved Issues section.

ESXi 4.0 Update 2 hosts may PSOD after vCenter Server is upgraded to 5.0

October 11th, 2011

This important notice just came across my radar:

VMware has become aware of some ESXi 4.0 hosts experiencing a purple screen after vCenter Server is upgraded to 5.0

The Knowledgebase Team has prepared KB article: ESXi 4.0 Update 2 hosts may experience a purple screen after vCenter Server is upgraded to 5.0 (2007269) and an alert has been placed on the Support page to alert customers of this issue.

This Knowledge Base article will be updated if new information becomes available (you can subscribe to rss feeds on individual KB articles for this purpose). If you have been affected by this, please read the KB.

We apologize for any inconvenience this may have caused you. If you know how to spread the word to your friends and colleagues, please do so.


You may encounter an issue where:

  • You have recently upgraded your vCenter Server to version 5.0
  • You have ESXi 4.0 Update 2 hosts in the inventory of this vCenter Server
  • After the vpxa agents are upgraded, the ESXi 4.0 Update 2 hosts experience a purple screen that includes this error:NOT_IMPLEMENTED bora/vmkernel/filesystems/visorfs/visorfsObj.c:3391


This is caused by an issue in the handling of the vpxa agent upgrade.


This issue has been resolved in 4.0 Update 3. To avoid this issue, upgrade all ESXi 4.0 Update 2 hosts to at least version ESXi 4.0 Update 3 before upgrading vCenter Server to 5.0.
ESXi 4.0 Update 3 and later versions can be downloaded from the VMware Download Center.

Enabling vCenter Server 5.0 Database Monitoring

September 27th, 2011

I stumbled across this while rummaging through the vSphere 5.0 Installation and Setup document.  Page 183 contains a small section (new in vSphere 5.0) which describes a process to enable database monitoring for Microsoft SQL Server (surrounding pages discuss enabling the same for other supported database platforms).  The SQL script provided in the documentation contains an error on the first line but I was able to adjust that and run it on the SQL 2008 R2 server in the lab.  Following is the script I ran:

use master
grant VIEW SERVER STATE to vcenter

Once access has been granted, vCenter will collect certain SQL Server health statistics and store them in the rotating vCenter profile log located by default at C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd-profiler-xx.log.  These metrics were taken from my vCenter Server log file and serve as an example of what is being collected from the SQL Server by the vCenter Server:

–> <dbMonitoring>
–> DbMonitoring/Counter/Storage: Manually extensible data files/Unit/count/Range Type/range/RangeMin/0/RangeMax/0/Timestamp/2011-09-27T18:00:01.79Z/Value/0
–> DbMonitoring/Counter/Memory:Database pages/Unit/timesIncrease/Range Type/range/RangeMin/0/RangeMax/3/Timestamp/1970-01-01T00:00:00Z/Value/N/A
–> DbMonitoring/Counter/Storage: Peak data file storage utilization/Unit/percent/Range Type/range/RangeMin/60559224/RangeMax/90/Timestamp/2011-09-27T18:00:01.802999Z/Value/0
–> DbMonitoring/Counter/Memory:Availaable/Unit/kiloBytes/Range Type/range/RangeMin/5120/RangeMax/60559416/Timestamp/1970-01-01T00:00:00Z/Value/N/A
–> DbMonitoring/Counter/Memory:Page Life Expectancy/Unit/seconds/Range Type/range/RangeMin/300/RangeMax/60559416/Timestamp/1970-01-01T00:00:00Z/Value/N/A
–> DbMonitoring/Counter/IO:Log growths/Unit/timesIncrease/Range Type/range/RangeMin/0/RangeMax/3/Timestamp/1970-01-01T00:00:00Z/Value/N/A
–> DbMonitoring/Counter/CPU:Usage/Unit/percent/Range Type/range/RangeMin/0/RangeMax/80/Timestamp/2011-09-27T18:00:01.75Z/Value/44
–> DbMonitoring/Counter/Memory:Buffer cache hit ratio/Unit/percent/Range Type/range/RangeMin/90/RangeMax/100/Timestamp/1970-01-01T00:00:00Z/Value/N/A
–> DbMonitoring/Counter/General:User Connections/Unit/count/Range Type/range/RangeMin/255/RangeMax/60559416/Timestamp/1970-01-01T00:00:00Z/Value/N/A
–> </dbMonitoring>

Per VMware’s documentation:

vCenter Server Database Monitoring captures metrics that enable the administrator to assess the status and health of the database server. Enabling Database Monitoring helps the administrator prevent vCenter downtime because of a lack of resources for the database server. Database Monitoring for vCenter Server enables administrators to monitor the database server CPU, memory, I/O, data storage, and other environment factors for stress conditions. Statistics are stored in the vCenter Server Profile Logs. You can enable Database Monitoring for a user before or after you install vCenter Server. You can also perform this procedure while vCenter Server is running.

One thing that I noticed is that these metrics were being collected in the vCenter log files prior to running the enabling script.  I’m not sure if this is because vCenter already had the required permissions to the master database (I use SQL authentication and I didn’t explicitly grant this), or perhaps this is enabled by default in the vCenter installation routine when the database prepare script runs.

The instructions provide plenty of context but are are fairly brief and don’t identify next steps or how to harvest the collected metrics.  Perhaps the vCenter Service Health agent monitors the profile log and will alarm through vCenter.  If not, then I view this as a monitoring framework VMware provides which can tailored for specific environments.  Thresholds could be defined which trigger alerts proactively before dangers or an outage occurs.  Admittedly I’m not a DBA.  With what’s provided, I’m not sure if this provides much value above and beyond native monitoring and alerting provided by SQL Server and Perfmon.