Posts Tagged ‘Microsoft’

New Microsoft .NET Framework Update Breaks vSphere Client

June 10th, 2010

Just a quick heads up to bring attention to an issue which I caught on Twitter.  VMware published KB 1022611 today which describes a new issue that is introduced by a recent Microsoft .NET Framework 2.0 SP2 & 3.5 SP1 update.  Upon installing the update, the vSphere Client stops working.  According to the article, the issue impacts ESX(i)3.5, 4.0, and vCenter 4.0.  Contrary to the topic of this blog post, I am not placing blame on Microsoft.  It remains unclear to me which company’s development staff is responsible for the software incompatibility.  Microsoft obviously issued the udpate which revealed the problem, but VMware has some skin in this as well in that they need to make sure they are following Microsoft .NET Framework development standards and best practices for their enterprise hypervisor management.

Key details from the VMware KB article:

The vSphere Clients, prior to the Update 1 release, cannot be used to access the vCenter Server or ESX hosts. A Microsoft update that targets the .NET Framework, released on June 9th 2010 is causing this issue. The update causes the vSphere Client to stop working.    To correct the issue there are two options that can be performed:

  • Remove the MS update from your Windows operating system. The vSphere Client works after the update is removed.

Note: This affects Windows XP, Windows 2003, Windows 2008, Windows Vista, and Windows 7.

VMware Workstation Upgrade to 7.1

May 26th, 2010

Microsoft Windows 7 Professional 64-bit
VMware Workstation 7.0.1 build-227600

I had heard VMware Workstation 7.1 was released.  Unfortunately, the VMware Workstation “check for updates” feature doesn’t seem to be serving its intended purpose as it told me no updates were available.

I downloaded the installation package manually and performed the upgrade.  Two reboots were required:

  1. After the uninstall of my previous version of Workstation
  2. After the install of Workstation 7.1

I hope the usability experience is better than my upgrade experience.  I realize some of the reboot business is on the Microsoft Windows 7 operating system but come on, would someone please figure this out?  Is there no way to perform an in place upgrade of Workstation to minimize the reboots to one?

What’s New in VMware Workstation 7.1

•Support for 8 virtual processors (or 8 virtual cores) and 2 TB virtual disks.

•Support for OpenGL 2.1 for Windows Vista and Windows 7 guests.

•Greatly improved DirectX 9.0 graphics performance for Windows Vista and Windows 7 guests. Up to 2x faster than Workstation 7.

•Launch virtualized applications directly from the Windows 7 taskbar to create a seamless experience between applications in your virtual machines and the desktop.

•Optimized performance for Intel’s Core i3, i5, i7 processor family for faster virtual machine encryption and decryption.

•Support for more Host and Guest Operating Systems, including: Hosts: Windows 2008 R2, Ubuntu 10.04, RHEL 5.4, and more Guests: Fedora 12, Ubuntu 10.04, RHEL 5.4, SEL 11 SP1, and more.

•Now includes built in Automatic Updates feature to check, download, and install VMware Workstation updates.

•Ability to import and export Open Virtualization Format (OVF 1.0) packaged virtual machines and upload directly to VMware vSphere, the industry’s best platform for building cloud infrastructures.

Microsoft Exchange 2003 to Exchange 2010 Upgrade Notes

May 14th, 2010

Last weekend I successfully upgraded, ahem, migrated the lab infrastructure from Microsoft Exchange 2003 to Exchange 2010.  This upgrade has been on my agenda for quite some time but I had been delaying it mainly due to lack of time and thorough knowledge of the steps.  I had a purchased the Microsoft Exchange Server 2010 Administrator’s Pocket Consultant (ISBN: 978-0-7356-2712-3) in January and marked up a few pages with a highlighter.  However, the deeper I got in the book, the more daunting the task seemed to have become, even for a simple one-server environment like mine.  In my mind, Exchange has always been somewhat of a beast, with increasing levels of difficulty as new editions emerged.  The pocket consultant series of books are wonderfully technical, but they haven’t been able to fit in my pocket for about a decade. They contain so much content that it has become difficult to rely on them as a CliffsNotes guide for platform upgrades, especially when it comes to Exchange.

Then two things happened miraculously at the same time.  First, I was invited to a private beta test of a virtualization related iPad application.  As part of this test, I needed to be able to send email from my iPad.  I had been unsuccessful thus far in getting Microsoft Exchange ActiveSync to work with the iPad (even after following Stephen Foskett’s steps) and could only assume that it was due to several years of wear and tear on my Exchange 2003 Server.  I needed to get that upgrade to Exchange 2010 done quickly.  Second, the May 2010 issue of Windows IT Pro magazine showed up in my mailbox.  To my delight, it was chock full of Exchange 2010 goodness, including a cover story of “Exchange 2003 to Exchange 2010 Step-by-Step Exchange Migration”. I’m pretty sure this was divine intervention with the message being “Get it done this weekend, you can do this.”

The upgrade article by Michael B. Smith started on page 26 and was 100% in scope.  The focus was a single server Exchange environment upgrade from 2003 to 2010.  I read the seven page artile in its entirety, marking up key “to-do” steps with a highlighter.  Following are some things I learned along the way:

  1. Naturally the Exchange server is virtualized on VMware vSphere.
  2. My Exchange environment is built upon a foundation that dates back as far as Exchange 5.5 (pre-Active Directory).  There would be no in place upgrades.  Exchange hasn’t provided an upgrade since Exchange 2003.  That suited me just fine as the Exchange 2003 server has been through so much neglect, although it had gotten pretty slow, it’s a miracle it was still functional.  The Exchange migration will consist of bringing up a fresh OS with a new installation of Exchange, and then migrating the mailboxes and services, and then retiring the old Exchange Server.  Microsoft calls this a migration rather than an upgrade.
  3. Exchange must be running in Native mode.  Not a problem, I was already there.
  4. Pre-migration, there exists a hotfix from Microsoft which is recommended to be installed on the Exchange 2003 server.
  5. The Schema Master mast be running Windows Server 2003 SP1 or higher.
  6. There needs to be at least one Global Catalog server at Windows Server 2003 SP1 or higher in the Exchange site.
  7. The AD forest needs to be at Server 2003 Forest Functional Level or higher.
  8. The AD domain needs to be at Server 2003 Domain Functional Level or higher.
  9. For migration flexibility purposes, Exchange 2003 and Exchange 2010 both support DFL and FFL up to Server 2008 R2.
  10. Exchange 2010 requires 64-bit hardware.  No problem, that requirement was met with vSphere .
  11. Exchange 2010 can be installed on Windows Server 2008 or Windows Server 2008 R2.  I naturally opted for R2.  No sense in deploying a two-year old OS when a more current one exists and is supported.  Plus, I personally need more exposure to 2008 and R2… 2003 is getting long in the tooth.
  12. Copy the Exchange DVD to a data/utility drive on the server.  Reason being, you can drop the most recent rollup available into the \Updates\ folder and basically perform a slipstream installation of Exchange with the most recent rollup applied out of the gate.  As of this writing, the most current is Rollup 3.
  13. Here’s a big time saver, install the server roles and features Exchange 2010 requires using the provided script on the DVD:
    \scripts\ServerManagerCmd -ip Exchange-Typical.xml -restart
    Other sample pre-requisite installer scripts can be found here.
  14. The 2007 Office System Converter: Microsoft Filter Pack (x64) is required to be installed.  This is downloadable from Microsoft’s website.  A little strange, but I’ll play along.  It’s required for the Exchange full-text search engine to search Office format documents.
  15. Run the following commands for good measure. It may or may not be required depending on what’s been done to the server so far:
    sc config NetTcpPortSharing start= auto
    net start NetTcpPortSharing
  16. Setup logs for Exchange are found in C:\ExchangeSetupLogs\  The main one is ExchangeSetup.log.  Hopefully you won’t have to rely on these logs and you are blessed with a trouble-free installation.
  17. There are the usual Active Directory preparatory steps to expand the Schema which seem to have increased in quantity but I could be hallucinating:
    1. /PrepareLegacyExchangePermissions
    2. /PrepareSchema
    3. /PrepareAD
    4. /PrepareAllDomains
  18. Installation can be invoked by CLI with /mode:install /roles:ca,ht,mb however, I chose a GUI installation which was more intuitive for me.
  19. The article stated the installation would take at least 20 minutes on fast hardware.  My installation took less than 15 minutes on a VM hosted by four year old servers attached to fibre channel EMC Celerra storage – bitchin.
  20. A Send connector is required before Exchange 2010 will route mailto the internet.
  21. Exchange 2010 ships with two Receive connectors but they must be configured before they will accept anonymous email from the internet.
  22. Exchange 2010 is managed by the Exchange Management Console which is called the EMC for short.  That will be easy to remember.
  23. Exchange 2010 is also managed by PowerShell scripts (also called an Exchange Management Shell, or EMS for short).  There are some configuration tasks which can only be made via PowerShell script and not via the EMC.
  24. Lend your end users and Helpdesk staff a hand by creating a meta-refresh document in C:\inetpub\wwwroot\ which points to https://<mail_server_fqdn>/owa effectively teleporting them into Outlook Web App (did you catch the name change? no more Outlook Web Access)
  25. Mailboxes are no longer moved online due to their potential size and problems which may occur if a mailbox is accessed during migration.  Mailbox migrations are now handled via EMC by way of a Move Request (either local [same org] or remote[different org]).  When a move request is submitted, the process begins immediately but may take some time to complete obviously based on the size of the mailbox as well as the quantity of mailboxes multiple selected for the move request.  Tony Redmond wrote a decent article on how this is done.  Scheduled move requests can be instantiated via PowerShell script.
  26. One of the final steps of a successful migration is properly decommissioning the old Exchange 2003 environment.  This is where things got a little hairy, and I half wasn’t surprised.  Upon attempting to uninstall Exchange 2003 to properly remove its tentacles from Active Directory and the Exchange organization, I was greeted by two errors in the following message:
    5-9-2010 9-16-31 PM
    In the legacy Exchange 2003 System Manager, there are two Recipient Update policies which exist.  Going from memory, one was for the domain which I was able to remove easily, and one was an Enterprise policy which cannot be removed via the System Manager.  Follow the instructions near the end of this article for the procedure to modify Active Directory with adsiedit.
    The second error message deals with removal of the legacy Routing Group Connector.  There were actually two which needed to be removed.  The only way to remove the Routing Group Connector is via PowerShell and it is also described towards the end of this article.
  27. After addressing the issues above, the uninstaller ran briefly and then failed for an unknown reason.  Upon attempting to re-run the uninstall, I noticed the ability uninstall Exchange 2003 via Add/Remove Programs in the Control Panel had disappeared, as if it was successfully uninstalled. Clearly it was not as the Exchange services still existed, were running, and I could launch System Manager and manage the organization.
  28. ActiveSync doesn’t work out of the box on privileged administrator level accounts due to security reasons.  If you accept the risk, this behavior can be changed by enabling the inheritance checkbox on the user account security property sheet.

I’m pretty happy with the results.  The process took took quite a few steps but I am nonetheless pleased.  Careful work following a very nicely outlined procedure by Michael B. Smith has yielded both a snappy-fast Exchange 2010 server on Windows Server 2008 R2 as well as ActiveSync integration with my iPad.  Exchange 2010 is a beast.  I can’t imagine tackling an Exchange project for anything larger than the smallest of environments.  I’m not sure how I can have so many years experience managing my own small Exchange environment yet still lack the confidence in the technology.  I guess it mostly runs itself and as I said earlier, it’s quite resilient meaning it doesn’t require much care and feeding from me.  And thank God for that.

Windows 2008 R2 and Windows 7 on vSphere

March 28th, 2010

If you run Windows Server 2008 R2 or Windows 7 as a guest VM on vSphere, you may be aware that it was advised in VMware KB Article 1011709 that the SVGA driver should not be installed during VMware Tools installation.  If I recall correctly, this was due to a stability issue which was seen in specific, but not all, scenarios:

If you plan to use Windows 7 or Windows 2008 R2 as a guest operating system on ESX 4.0, do not use the SVGA drivers included with VMware Tools. Use the standard SVGA driver instead.

Since the SVGA driver is installed by default in a typical installation, it was necessary to perform a custom installation (or scripted perhaps) to exclude the SVGA driver for these guest OS types.  Alternatively, perform a typical VMware Tools installation and remove the SVGA driver from the Device Manager afterwards.  What you ended up with, of course, is a VM using the Microsoft Windows supplied SVGA driver and not the VMware Tools version shown in the first screenshot.  The Microsoft Windows supplied SVGA driver worked and provided stability as well, however one side effect was that mouse movement via VMware Remote Console felt a bit sluggish.

Beginning with ESX(i) 4.0 Update 1 (released 11/19/09), VMware changed the behavior and revised the above KB article in February, letting us know that they now package a new version of the SVGA driver in VMware Tools in which the bits are populated during a typical installation but not actually enabled:

The most effective solution is to update to ESX 4.0 Update 1, which provides a new WDDM driver that is installed with VMware Tools and is fully supported. After VMware Tools upgrade you can find it in C:\Program Files\Common Files\VMware\Drivers\wddm_video.

After a typical VMware Tools installation, you’ll still see a standard SVGA driver installed.  Following the KB article, head to Windows Device Manager and update the driver to the bits located in C:\Program Files\Common Files\VMware\Drivers\wddm_video:


The result is the new wddm driver, which ships with the newer version of VMware Tools, is installed: 

After a reboot, the crisp and precise mouse movement I’ve become accustomed to over the years with VMware has returned.  The bummer here is that while the appropriate VMware SVGA drivers get installed in previous versions of Windows guest operating systems, Windows Server 2008 R2 and Windows 7 require manual installation steps, much like VMware Tools installation on Linux guest VMs.  Add to this the fact that the automated installation/upgrade of VMware Tools via VMware Update Manager (VUM) does not enable the wddm driver.  In short, getting the appropriate wddm driver installed for many VMs will require manual intervention or scripting.  One thing you can do is to get the wddm driver installed in your Windows Server 2008 R2 and Windows 7 VM templates.  This will ensure VMs deployed from the templates have the wddm driver installed and enabled.

The wddm driver install method from VMware is helpful for the short term, however, it’s not the scalable and robust long term solution.  We need an automated solution from VMware to get the wddm driver installed.  It needs to be integrated with VUM.  I’m interested in finding out what happens with the next VMware Tools upgrade – will the wddm driver persist, or will the VMware Tools upgrade replace the wddm version with the standard version?  Stay tuned.

Update 11/6/10:  While working in the lab tonight, I noticed that with vSphere 4.1, the correct wddm video driver is installed as part of a standard VMware Tools installation on Windows 7 Ultimate x64 – no need to manually replace the Microsoft video driver with VMware’s wddm version as this is done automatically now.

Update 12/10/10: As a follow up to these tests, I wanted to see what happens when the wddm driver is installed under ESX(i) 4.0 Update 1 and its corresponding VMware Tools, and then the VM is moved to an ESX(i) 4.1 cluster and the VMware Tools are upgraded.  Does the wddm driver remain in tact, or will the 4.1 tools upgrade somehow change the driver?  During this test, I opted to use Windows 7 Ultimate 32-bit as the guest VM guinea pig.  A few discoveries were made, one of which was a surprise:

1.  Performing a standard installation of VMware Tools from ESXi 4.0 Update 1 on Windows 7 32-bit will automatically install the wddm driver, version as shown below.  No manual steps were required to install this driver forcing a 2nd reboot.  I wasn’t counting on this.  I expected the Standard VGA Graphics Adapter driver to be installed as seen previously.  This is good.

SnagIt Capture

After moving the VM to a 4.1 cluster and performing the VMware Tools upgrade, the wddm driver was left in tact, however, its version was upgraded to  This is also good in that the tools ugprade doesn’t negatively impact the desired results of leveraging the wddm driver for best graphics performance.

SnagIt Capture

More conclusive testing should be done with Windows 7 and Windows Server 2008 R2 64-bit to see if the results are the same.  I’ll save this for a future lab maybe.

Create a 32-bit vCenter DSN on a 64-bit Operating System

November 21st, 2009

As I had pointed out in this blog post, VMware hints that 64-bit may be the future for vCenter Server. I decided that for my upgrade to vCenter 4.0 Update 1 this weekend, I would take the opportunity to rebuild my vCenter server from Windows Server 2003 32-bit to Windows Server 2008 64-bit.

Once the 64-bit base operating system build was complete, I installed the 64-bit Microsoft SQL Server Native Client drivers (downloadable here) since my back end database is Microsoft SQL Server 2005 on a remote server. A key thing to remember about this installation is that it installs both 64-bit and 32-bit DSN drivers.

The next step is to create the vCenter ODBC DSNs. Although vCenter Server runs on 64-bit operating systems, it currently requires a 32-bit ODBC DSN. This is important to remember because the Windows Start Menu launches the 64-bit ODBC DSN tool, not the 32-bit version I needed.  The vCenter Server (and Update Manager) installation will not complete without a 32-bit DSN.

To create a 32-bit DSN on a 64-bit operating system, run the following executable:


Once the utility opens, you’ll be greeted by all the legacy 32-bit ODBC DSNs you’ve likely seen for years working with tiered Windows platforms. If using Microsoft SQL Server 2005 like me, be sure to select the SQL Native Client driver towards the bottom of the list, and not Driver da Microsoft para arquivos texto highlighted below:

Proceed with the creation of the vCenter Server and Update Manager ODBC DSNs and complete the vCenter Server and Update Manager installations.

This information and much more can be found in the ESX and vCenter Server Installation Guide, page 73.

VMware ESX Guest OS I/O Timeout Settings (for NetApp Storage Systems)

October 29th, 2009

You may already be aware that installing VMware Tools in a Windows VM configures a registry value which controls the I/O timeout for all Windows disk in the event of a short storage outage. This is to help the guest operating system survive high latency or temporary outage conditions such as SAN path failover or maybe a network failure in Ethernet based storage.  VMware Tools changes the Windows default value of 10 seconds for non-cluster nodes, 20 seconds for cluster nodes, to 60 seconds (or x03c hex).

Did you know that disk I/O timeout is a configurable parameter in other guest operating systems as well? And why not, it makes sense that we would want every guest OS to be able to outlast a storage deficiency.

NetApp offers a document titled VMware ESX Guest OS I/O Timeout Settings for NetApp Storage Systems. It’s published as kb41511 and you’ll need a free NetApp NOW account to access the document. This white paper serves a few useful purposes:

  • Defines recommended disk I/O timeout settings for various guest operating systems on NetApp storage systems
  • Defines benchmark disk I/O timeout settings for various guest operating systems which could be used on any storage system, including local SCSI
  • In some cases provides scripts to make the necessary changes
  • Explains the methods to make the disk I/O timeout changes on the following guest operating systems:
    • RHEL4
    • RHEL5
    • SLES9
    • SLES10
    • Solaris 10
    • Windows

Now on the subject disk I/O timeouts, understand the above is to be used as chance for extending the uptime of a VM during adverse storage conditions. As in life, there are no guarantees. A guest OS with high disk I/O activity may not be able to tolerate sustained read and/or write requests for the duration of the timeout value. Windows guests may freeze or BSOD. Linux guests may go read-only on their root volumes which requires a reboot. Which brings me to the next point…

A larger timeout value isn’t necessarily better. In extending disk I/O timeout values, we’re applying virtual duct tape to an underlying storage issue which needs further looking into. Given the complex and wide variety of shared storage systems available to the datacenter today, storage issues can be caused by many variables including but not limited to disks (spindles), target controllers, fabric components such as fibre cables, SFP/GBICs, HBAs, fabric switches, zoning, network components such as copper cabling, NICs, network switches, routers, and firewalls. Also keep in mind that while the OS may survive the disk I/O interruption, application(s) running on the OS platform may not.  Applications themselves implement response timeout values which are likely going to be hard coded and non-configurable by a platform or virtualization administrator in the application itself.

Lastly, try to remember that if you go through the effort of increasing your disk I/O timeout values on Windows guests beyond 60 seconds, future installation of VMware Tools or other applications/updates may reset the disk I/O timeout back to 60 seconds.  What this means is that in medium to large environments, you’re going to need an automated method to deploy custom disk I/O timeout values at least for Windows guests.  For those with NetApp storage, NetApp pushes these standards firmly, along with other VMware best practices which I’ll save for a future blog article.

Update 4/28/10:  VMware Tools for vSphere installation doesn’t change the disk timeout setting if a custom value was previously set (ie. 190 seconds)

Update 9/12/11:  See also VMware KB article 1009465 Increasing the disk timeout values for a Linux 2.6 virtual machine

SQL 2005 SP2 End of Support to Force Rapid vSphere Upgrade?

October 1st, 2009

The way I read it, the Microsoft Support Lifecycle for SQL Server 2005 tells me that SQL Server 2005 SP2 support ends on 12/15/2009. That’s about 10 weeks from today.

Why should you care? If you’re utilizing VMware vCenter Server 2.5 in your production datacenter, you’ve got about 10 weeks to upgrade to vSphere to stay within a VMware supported configuration. The VMware Virtual Infrastructure Compatibility Matrixes reveal on page 10 that vCenter 2.5 is only compatible with SQL Server 2005 up to Service Pack 2. SP3 is not supported.

To make the jump to SQL Server 2005 SP3 or SQL Server 2008 requires upgrading to vSphere to stay within a VMware supported configuration.

I would venture to guess that a lot of VI customers are not ready for the jump to vSphere, especially those who wish to take advantage of the new features and the design considerations which must be evaluated and planned before deployment. Not to mention the licensing considerations which are tied to the new features. While we’re on the subject of licensing, keep in mind Enterprise licensing is retired mid December 2009. To keep existing Enterprise features in the virtual infrastructure will require Enterprise Plus licensing after the mid December Enterprise license retirement date.

With the SQL 2005 SP2 retirement date approaching, I’ll be looking for VMware modify their support stance to support SQL Server 2005 SP3. A lot of customers are going to need this to keep within support.

Speaking of SQL Server 2008, beware a caveat that Orchestrator 4.0 is not supported on SQL 2008 (yet).