Archive for October, 2009

« Older Entries

VMware ESX Guest OS I/O Timeout Settings (for NetApp Storage Systems)

October 29th, 2009

You may already be aware that installing VMware Tools in a Windows VM configures a registry value which controls the I/O timeout for all Windows disk in the event of a short storage outage. This is to help the guest operating system survive high latency or temporary outage conditions such as SAN path failover or maybe a network failure in Ethernet based storage.  VMware Tools changes the Windows default value of 10 seconds for non-cluster nodes, 20 seconds for cluster nodes, to 60 seconds (or x03c hex).

Did you know that disk I/O timeout is a configurable parameter in other guest operating systems as well? And why not, it makes sense that we would want every guest OS to be able to outlast a storage deficiency.

NetApp offers a document titled VMware ESX Guest OS I/O Timeout Settings for NetApp Storage Systems. It’s published as kb41511 and you’ll need a free NetApp NOW account to access the document. This white paper serves a few useful purposes:

  • Defines recommended disk I/O timeout settings for various guest operating systems on NetApp storage systems
  • Defines benchmark disk I/O timeout settings for various guest operating systems which could be used on any storage system, including local SCSI
  • In some cases provides scripts to make the necessary changes
  • Explains the methods to make the disk I/O timeout changes on the following guest operating systems:
    • RHEL4
    • RHEL5
    • SLES9
    • SLES10
    • Solaris 10
    • Windows

Now on the subject disk I/O timeouts, understand the above is to be used as chance for extending the uptime of a VM during adverse storage conditions. As in life, there are no guarantees. A guest OS with high disk I/O activity may not be able to tolerate sustained read and/or write requests for the duration of the timeout value. Windows guests may freeze or BSOD. Linux guests may go read-only on their root volumes which requires a reboot. Which brings me to the next point…

A larger timeout value isn’t necessarily better. In extending disk I/O timeout values, we’re applying virtual duct tape to an underlying storage issue which needs further looking into. Given the complex and wide variety of shared storage systems available to the datacenter today, storage issues can be caused by many variables including but not limited to disks (spindles), target controllers, fabric components such as fibre cables, SFP/GBICs, HBAs, fabric switches, zoning, network components such as copper cabling, NICs, network switches, routers, and firewalls. Also keep in mind that while the OS may survive the disk I/O interruption, application(s) running on the OS platform may not.  Applications themselves implement response timeout values which are likely going to be hard coded and non-configurable by a platform or virtualization administrator in the application itself.

Lastly, try to remember that if you go through the effort of increasing your disk I/O timeout values on Windows guests beyond 60 seconds, future installation of VMware Tools or other applications/updates may reset the disk I/O timeout back to 60 seconds.  What this means is that in medium to large environments, you’re going to need an automated method to deploy custom disk I/O timeout values at least for Windows guests.  For those with NetApp storage, NetApp pushes these standards firmly, along with other VMware best practices which I’ll save for a future blog article.

Update 4/28/10:  VMware Tools for vSphere installation doesn’t change the disk timeout setting if a custom value was previously set (ie. 190 seconds)

Update 9/12/11:  See also VMware KB article 1009465 Increasing the disk timeout values for a Linux 2.6 virtual machine

8 comments »

Posted in Virtualization

Tags:

New Blog Theme

October 28th, 2009

Over the course of the past year, I’ve received some feedback that my dark blog theme, while nifty, was hurting readers’ eyes.  In fact, there are some readers who only read this blog through an RSS reader so that their eyes are not strained.  I’m in agreement and have been for quite some time.  The only reason I hadn’t changed it was because I didn’t want to be known as someone who changes themes often and for the heck of it.

I’ve chosen this new theme called Green Park 2. The green colored bar across the top gives it a “green feeling” which is quite appropriate for the blog’s subject of virtualization in the datacenter (and beyond).

Anyway, I hope you enjoy it.  Barring any problems, I intend to keep it around indefinitely.  Perhaps it will encourage a few of the RSS lurkers to come out of the woodwork.

11 comments »

Posted in Virtualization

Tags:

TrainSignal vSphere Training DVD 1 Completed

October 23rd, 2009

This evening I finished viewing the first of three TrainSignal vSphere Training DVDs authored by VCP and CCIE David Davis. Having viewed TrainSignal’s last VMware Virtual Infrastructure training on VI3, I knew I was in for some good stuff.

DVD 1 starts off with introductions to the video’s instructor as well as a hypothetical company which is used as a focus and discussion point throughout the video series. Practical application of technologies to a role played scenario, the Wired Brain Coffee Company in this case, serves as positive reinforcement to the lessons being taught and is an effective method for knowledge retention, especially if the student is following along and working hands on in their own lab through the examples.

The video then sets a beginner’s pace as it covers VMware certification, virtualization basics. Moving on, it compares and contrasts VMware, Microsoft, and Citrix hypervisors. Beyond this comparison, the focus from here on out is on VMware products where a closer look is taken at the different components and tiers of vSphere.

Half way through the DVD, we’re finally to the point where we’re installing and configuring the vSphere products. One valuable offering from the video is a lesson describing the steps needed to install ESX and ESXi in VMware Workstation. This is what is called a nested hypervisor – an ESX(i) type 1 (bare metal) hypervisor running on top of a VMware Workstation type 2 (hosted) hypervisor. Nested hypervisors are not supported in production environments but they are quite helpful in lab, test, and portable environments.

Towards the end, lesson 17 provides a nice demonstration of a VMware Tools installation in a Linux guest operating system which isn’t nearly as straight forward as a VMware Tools on Windows installation. The last two lessons begin touching on some of the new advanced features that vSphere offers: Hot Add/Hot Plug virtual hardware and Host Profiles.

Thus far my feeling is this training is geared towards the beginner to intermediate level. I’m looking forward to DVD 2 where the instructor dives into more of the advanced design, configuration, and operational topics of VMware vSphere. I’ve attended VMware’s vSphere What’s New (2 day) and VMware’s vSphere Quick Start (5 day) classes. With approximately 150 new features making their debut in vSphere, I’ve yet to see anyone cover them all – that would be a tall order.

DVD 1 Lessons:

  1. Meet Your Instructor
  2. Our Scenario with the Wired Brain Coffee Company
  3. VMware Certification – Preparing for the VCP and VCDX
  4. Introduction to Virtualization
  5. Virtualization Products Compared
  6. VMware ESXi4 Free Edition for the SMB
  7. VMware vSphere 4 and ESX Essentials
  8. vSphere Management Options
  9. Installing the VMware vSphere Client
  10. Navigating vSphere Using the vSphere Client
  11. Running VMware ESX 4 in Workstation
  12. Installing VMware ESX 4
  13. Installing VMware ESXi Version 4
  14. Installing VMware vCenter 4
  15. vCenter4 – Configuring Your New Virtual Infrastructure
  16. Creating & Modifying Virtual Guest Machines
  17. Installing and Configuring VMware Tools
  18. Adding Virtual Machine Hardware with vSphere Hot Plug
  19. Using vSphere Host Profiles

6 comments »

Posted in Virtualization

Tags:

Happy Birthday Blog

October 20th, 2009

This blog turned one year old on Sunday. The inaugural entry, My First Blog and How To Install WordPress, was posted on October 18th, 2008.

Happy Birthday Blog!

For some reason, it feels like I’ve been at this a lot longer than one year.

Some statistics related to this blog during the past 365 days:

  • 224 posts
    • 195 Virtualization related
    • 30 Non-virtualization related (typically other technologies and a few personal)
    • Average of one post per every 1 1/2 days
    • Using a very conservative estimate of 2 hours spent writing each post, total time spent writing:
      • 448 hours
      • 19 days
      • Almost 3 weeks
      • Doesn’t include any lab time
  • 179,389 Unique visitors
    • Generating 5,399,277 Hits
    • Consuming 71.36GB Send (upload) traffic
    • Costing $1,199.88 in bandwidth
    • Plus another $1,800 estimated in rack electrical/cooling
  • 743 legit comments (approved)
    • Average of 2 legit comments per day
  • 7,873 spam comments (blocked and IP banned)
    • Average of 22 spam comments per day
  • 90 Tags
  • 14 Active WordPress plugins
  • 7 Inactive WordPress plugins
  • 4 Sponsors
  • 1 Theme change
  • 3 VMware exams passed
    • VCDX Enterprise Administration exam
    • VCP4 exam
    • VCDX Design exam
  • 2 VMworlds attended
    • VMworld Europe 2009
    • VMworld 2009
  • 7th most popular virtualization blog as rated by the VMware community
  • 1 vCalendar idea
  • 1 vExpert award
  • 0 VMware NFR licenses received
    • Out of a dozen or so requests
    • Over the course of 2+ years
  • Established about a million virtualization industry
    • Contacts
    • Friends
    • Acquaintances
  • Through
    • Blog
    • Twitter
    • VMworld
    • LinkedIn

Thank you for reading. I look forward to another great year with more VMware virtualization information to share! A special thank you also goes out to other bloggers and VMware virtualization community members for sharing your time and knowledge and continuing to inspire me to do the same.

11 comments »

Posted in General

Tags:

8 New ESX 3.5.0 Patches Released; 3 Critical

October 16th, 2009

Eight new patches have been released for ESX 3.5.0. Other versions of ESX, including vSphere and ESXi, are not impacted.

3 of the 8 patches are rated critical and should be evaluated quickly for application in your virtual infrastructure.

ID: ESX350-200910401-SG Impact: HostSecurity Release date: 2009-10-16 Products: esx 3.5.0 Updates VMkernel, Tools, hostd

This patch contains the following fixes and enhancements:

This patch updates the service console kernel version to kernel-2.4.21-58.EL. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the names CVE-2008-4210, CVE-2008-3275, CVE-2008-0598, CVE-2008-2136, CVE-2008-2812, CVE-2007-6063, and CVE-2008-3525 to the security issues fixed in kernel-2.4.21-58.EL.

This patch reduces the boot time of ESX hosts and should be applied when multiple ESX hosts detect LUNs used for Microsoft Cluster Service (MSCS).

Symptom: Error messages similar to the following might be logged in the /var/log/vmkernel log file of the service console:

Jul 24 14:34:24 VMEX3EQCH1100003 vmkernel: 165:15:48:57.500 cpu0:1033)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Jul 24 14:34:24 VMEX3EQCH1100003 vmkernel: 165:15:48:57.500 cpu0:1033)WARNING: SCSI: 5615: status SCSI reservation conflict, rstatus 0xc0de01 for vmhba1:0:9. residual R 919, CR 0, ER 3

Jul 24 14:34:24 VMEX3EQCH1100003 vmkernel: 165:15:48:57.500 cpu0:1033)SCSI: 6608: Partition table read from device vmhba1:0:9 failed: SCSI reservation conflict (0xbad0022)

Any additional lines or customizations added by a user in the /etc/fstab file are deleted when VMware Tools is reinstalled or reconfigured. This issue occurs because when uninstalling, VMware Tools restores the files which were backed up during installation.

After applying this patch, any request for connection with ESX 3.5 using cipher suite of 56-bit encryption will be dropped. As a result, browsers that exclusively use cipher suites with 40-bit and 56-bit encryption cannot connect to ESX 3.5. Microsoft has made the Internet Explorer High Encryption Pack available for Internet Explorer 5.01 and earlier. Internet Explorer 5.5 and higher versions already use 128-bit encryption.

This patch contains a fix for a security vulnerability in the ISC third-party DHCP client. This vulnerability allows for code execution in the client by a remote DHCP server through a specially crafted subnet-mask option. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2009-0692 to this issue.

ID: ESX350-200910402-BG Impact: Critical Release date: 2009-10-16 Products: esx 3.5.0 Updates ESX Scripts

This patch is required to be installed with ESX350-200910401-SG (KB 1013124) to resolve a boot-time-related issue. The patch reduces the boot time of ESX hosts and should be applied when multiple ESX hosts detect LUNs used for Microsoft Cluster Service (MSCS).

ID: ESX350-200910403-SG Impact: HostSecurity Release date: 2009-10-16 Products: esx 3.5.0 Updates Web Access

This patch updates the following:

WebAccess component Tomcat server to 5.5.27. This update addresses multiple security issues that exist in the earlier releases of the Tomcat server.

The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the names CVE-2008-1232, CVE-2008-1947, and CVE-2008-2370 to the issues addressed by Tomcat 5.5.27. For more information on these security vulnerabilities, refer to the Apache Tomcat 5.x Vulnerabilities page at http://tomcat.apache.org/security-5.html.

WebAccess component JRE to 1.5.0_18. This update addresses multiple security issues that existed in the previous versions of JRE.

The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the following names to the security issues fixed in JRE 1.5.0_17:

CVE-2008-2086, CVE-2008-5347, CVE-2008-5348, CVE-2008-5349, CVE-2008-5350, CVE-2008-5351, CVE-2008-5352, CVE-2008-5353, CVE-2008-5354, CVE-2008-5356, CVE-2008-5357, CVE-2008-5358, CVE-2008-5359, CVE-2008-5360, CVE-2008-5339, CVE-2008-5342, CVE-2008-5344, CVE-2008-5345, CVE-2008-5346, CVE-2008-5340, CVE-2008-5341, CVE-2008-5343, and CVE-2008-5355.

The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the following names to the security issues fixed in JRE 1.5.0_18:

CVE-2009-1093, CVE-2009-1094, CVE-2009-1095, CVE-2009-1096, CVE-2009-1097, CVE-2009-1098, CVE-2009-1099, CVE-2009-1100, CVE-2009-1101, CVE-2009-1102, CVE-2009-1103, CVE-2009-1104, CVE-2009-1105, CVE-2009-1106, and CVE-2009-1107.

ID: ESX350-200910404-SG Impact: HostSecurity Release date: 2009-10-16 Products: esx 3.5.0 Updates cim

After applying this patch, any request for connection to CIM port 5989 on ESX 3.5 using cipher suite of 56-bit encryption will be dropped.

ID: ESX350-200910405-SG Impact: HostSecurity Release date: 2009-10-16 Products: esx 3.5.0 Updates mptscsi drivers

This patch updates the mptscsi driver to a version that is compatible with the service console version kernel-2.4.21-58.EL.

ID: ESX350-200910406-SG Impact: HostSecurity Release date: 2009-10-16 Products: esx 3.5.0 Updates Service Console DHCP Client

The service console package dhclient has been updated to version dhclient-3.0.1-10.2. This fixes a stack buffer overflow flaw in the ISC DHCP client and a flaw in the way the DHCP daemon init script handles temporary files. The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the names CVE-2009-0692 and CVE-2009-1893 to these issues.

ID: ESX350-200910408-BG Impact: Critical Release date: 2009-10-16 Products: esx 3.5.0 Updates VMkernel iSCSI driver

When ESX 3.5 hosts are connected to Adaptec Snap Server series or Dell NX series of NAS appliances through the ESX software iSCSI initiator, sometimes the iSCSI LUNs are not detected by the ESX 3.5 hosts. The issue is caused due to the way the software iSCSI driver detects an overflow condition. This patch fixes the issue.

ID: ESX350-200910409-BG Impact: Critical Release date: 2009-10-16 Products: esx 3.5.0 Updates Emulex FC driver

ESX 3.5 Update 4 hosts with Emulex HBAs might stop responding when accessed through vCenter Server. This Emulex driver patch fixes the issue.

Symptom: On ESX hosts, any application making an ioctl call in to the Emulex driver might fail.

No comments »

Posted in Virtualization

Tags:

Virtualizing vCenter With vDS Catch-22

October 9th, 2009

I’ve typically been a fan of virtualizing the vCenter management server in most situations. VMware vCenter and Update Manager both make fine virtualization candidates as long as the underlying infrastructure for vCenter stays up. Loss of vCenter in a blackout situation can make things a bit of a hassle, but one can work through it with the right combination of patience and knowledge.

A few nights ago I had decided to migrate my vCenter VM to my vSphere virtual infrastructure. Because my vCenter VM was on a standalone VMware Server 2.0 box, I had to shut down the vCenter VM in order to cold migrate it to one of the ESX4 hosts directly, transfer the files to the SAN, upgrade virtual hardware, etc. Once the files were migrated to the vSphere infrastructure, it was time to configure the VM for the correct network and power it up. This is where I ran into the problem.

vCenter was shut down and unavailable, therefore, I had connected my vSphere client directly to the ESX4 host in which I transferred the VM to. When trying to configure the vCenter VM to use the vNetwork Distributed Switch (vDS) port group I had set up for all VM traffic, it was unavailable in the dropdown list of networks. The vCenter server was powered down and thus the vDS Control Plane was unavailable, eliminating my view of vDS networks.

This is a dilemma. Without a network connection, the vCenter server will not be able to communicate with the back end SQL database on a different box running SQL. This will cause the vCenter server services to not start and thus I’ll never have visibility to the vDS. Fortunately I have a fairly flat network in the lab with just a few subnets. I was able to create a temporary vSwitch and port group locally on the ESX4 host which would grant the vCenter VM the network connectivity it needed so I could then modify the network, changing from a local to a vDS port group on the fly.

Once the vCenter server was back up, I further realized that vDS port groups are still unable to be seen when the vSphere client is connected directly to an ESX4 host. The ability configure a VM to utilize vDS networking requires both that the vCenter server be functional, as well as a vSphere client connected to said vCenter server and not a managed host.

The situation I explained above is the catch-22 – the temporary inability to configure VMs for vDS networking while the vCenter server is unavailable. One might call my situation a convergence of circumstances, but with an existing virtualized vCenter server that you’re looking to migrate to a vDS integrated vSphere infrastructure, the scenario is very real. I’d like to note all VMs that had been running on a vDS port continued to run without a network outage as the vDS Data Plane is maintained on each host and remained in tact.

33 comments »

Posted in Virtualization

Tags:

SQL 2005 SP2 End of Support to Force Rapid vSphere Upgrade?

October 1st, 2009

The way I read it, the Microsoft Support Lifecycle for SQL Server 2005 tells me that SQL Server 2005 SP2 support ends on 12/15/2009. That’s about 10 weeks from today.

Why should you care? If you’re utilizing VMware vCenter Server 2.5 in your production datacenter, you’ve got about 10 weeks to upgrade to vSphere to stay within a VMware supported configuration. The VMware Virtual Infrastructure Compatibility Matrixes reveal on page 10 that vCenter 2.5 is only compatible with SQL Server 2005 up to Service Pack 2. SP3 is not supported.

To make the jump to SQL Server 2005 SP3 or SQL Server 2008 requires upgrading to vSphere to stay within a VMware supported configuration.

I would venture to guess that a lot of VI customers are not ready for the jump to vSphere, especially those who wish to take advantage of the new features and the design considerations which must be evaluated and planned before deployment. Not to mention the licensing considerations which are tied to the new features. While we’re on the subject of licensing, keep in mind Enterprise licensing is retired mid December 2009. To keep existing Enterprise features in the virtual infrastructure will require Enterprise Plus licensing after the mid December Enterprise license retirement date.

With the SQL 2005 SP2 retirement date approaching, I’ll be looking for VMware modify their support stance to support SQL Server 2005 SP3. A lot of customers are going to need this to keep within support.

Speaking of SQL Server 2008, beware a caveat that Orchestrator 4.0 is not supported on SQL 2008 (yet).

9 comments »

Posted in Virtualization

Tags: