Archive for October, 2008

Connect a fibre attached tape device to a VM on ESX

October 27th, 2008

Have you ever considered virtualizing your tape backup server? Maybe you’ve thought about it in the past but reasoning produced drawbacks that were too compelling to go forth and virtualize. For instance, pinning a VM to a clustered ESX host which is connected with a SCSI cable attached tape device hanging off of it. Pinning a VM to a clustered host means you lose the benefit of a VM portability, you lose the flexibility of host maintenance during production cycles, and you lose the use of valuable dollars spent on VMotion, DRS, HA, shared storage, and FT (future).

What if you had the hardware to make it possible? Would you do it? If I had to purchase hardware to specifically make this happen, cost effectiveness would need to be researched. Everything else being equal, if I had the hardware infrastructure in place already, yes I would. I had access to the hardware, so I headed into my lab to give it a shot.

What’s required?

  • Hardware
    • One or more ESX hosts
    • At least one fibre HBA in each ESX host that supportes fibre tape devices (enabled in the HBA BIOS typically at POST)
    • A fibre attached tape device (the fibre HBA in a tape device is called an NSR or Network Storage Router)
    • At least one fibre SAN switch
      • If using more than one in a single fabric, be sure they are ISL’d together
      • If multipathing in separate fabrics, at least two HBAs per host will be required and at least two NSRs in the tape device (although this is really going overboard and would be quite expensive)
    • Fibre cable between ESX host and SAN switch
    • Fibre cable between NSR and SAN switch
    • Optional components: shared storage
  • Software
    • VMware ESX or ESXi
    • Virtual Infrastructure Client
    • Latest firmware on HBA(s), NSR(s), and SAN switch(es)
    • Appropriate zoning created and enabled on SAN switch for all ESX host HBAs and NSR
    • Optional components: VirtualCenter, VMotion, DRS, HA

The steps to setting this up aren’t incredibly difficult.

  1. Attach fibre cables between HBAs and SAN switch
  2. Attach fibre cable between NSR and SAN switch
  3. On the fibre SAN switch, zone the NSR to all HBAs in all ESX hosts that will participate. Be sure to enable the active zone configuration. On Brocade SAN switches this is a two step process.
  4. Perform a scan on the fibre HBA cards (on all ESX hosts) to discover the fibre tape device. In this case, I’ve got an HP MSL5026 autoloader containing a robotic library and two tape drives:
  5. Once each ESX host can “see” the tape device, add the tape device to the VM as a SCSI passthru device. In the drop down selection box, the two tape drives are seen as “tape” and the robotic library is seen as “media”. Take a look at the .vmx file and see how the SCSI passthru device maps back to VMHBA1:1:2 and ultimately the tape drive as a symbolic link:
    fctape2 fctape3 fctape9 fctape10 fctape11
  6. The VM can now see the tape device. Notice it is SCSI and not fibre. At this time, VMs only see SCSI devices. Fibre is not virtualized within the VMware virtual machines to the extent that a VM can see virtual fibre or a virtual HBA. The current implementation of NPIV support in VMware is something different and will be explored in an upcoming blog:

Good news! The fibre attached tape drive works perfectly using Windows ntbackup.exe. Effective throughput rate of many smaller files to tape is 389MB/minute or 6.5MB/second. As expected, running a second backup job with less files but larger sizes, I saw an increased throughput rate of 590MB/minute or nearly 10MB/second. These speeds are not bad:
fctape5 fctape6 fctape8

Now for the bad news. When trying to migrate the VM while it was powered on (VMotion) or powered off (cold migration), I ran into a snag. VMware sees the fibre tape device as a raw disk with an LSI Logic SCSI controller which is not supported for migration (I tried changing the LSI Logic bus to use Physical bus sharing, but that did not work):
fctape7 fctape9

The VM migration component of my test was a failure, but the fibre connectivity was a success. Perhaps we’ll have SCSI passthru migration ability in a future version of VMware Virtual Infrastructure. Maybe v-SCSI passthru is the answer (v-* seems to be the next generation answer to many datacenter needs). What this experiment all boils down to is that I can’t do much more with a fibre attached tape device than I can with a SCSI attached tape device. In addition, a VM with an attached SCSI passthru device remains pinned to an ESX host and therefore doesn’t belong on a clustered host.  However, I can think of a few potential advantages of a fibre attached tape device which may still be of interest:

  1. Fibre cabling offers better throughput speed and more bandwidth than SCSI.
  2. Fibre cabling offers much longer cable run distances in the datacenter.
  3. A failed SCSI card on the host often means a motherboard replacement. A failed HBA on the host means replacing an HBA.
  4. Fibre cabling allows multipathing while SCSI cabling along with the required bus termination does not.
  5. Fibre cabling leverages a SAN fabric infrastructure which can be used to gather detailed reports using native and robust SAN fabric tools such as SAN switch performance monitor, SANsurfer, HBAnywhere, etc.
  6. VMs with fibre attached tape can still be migrated to other zoned hosts by simply removing the tape device in virtual hardware, performing the migration, then re-adding the tape device, all without leaving my chair. A SCSI attached tape device would actually need to be re-cabled behind the rack.

Microsoft Windows Add or Remove Programs terminology clarified

October 26th, 2008

A look at “Add or Remove Programs” on a Microsoft Windows machine reveals a list of installed software and Microsoft Windows Updates.  To the right of each program are details on installation size, use frequency, and date last used.  I rarely use the information on the right hand side because I’ve found it to be unreliable.  Take a peek at the example below.  Adobe Acrobat, a program I use often for reading and creating .PDF files is listed as being used “frequently”, yet I apparently haven’t used Adobe Acrobat since 12/23/2005, which was around the time this machine was built.


Well what exactly does the term “frequently” mean then?  Below are the defintions from Microsoft. 


Things are so much clearer to me now.  Well, not really.  Arbitrary definitions from one Microsoft developer are just that, arbitrary and potentially meaningless to the next person.  The reality of it is this is a broken feature that I’ll venture guess has behaved this way since Windows 2000 (I recall the screens being similar or identical).  Added shame is this misinformation comes from a Windows Server.  One would think this type of information would be easily gathered and reliably reported on a server class operating system.

Blog infrastructure updates

October 26th, 2008

I finally got around to removing the default logo on the banners used in the Xplosive Reloaded 1.0 theme I’m using. Anyone using Xplosive Reloaded can feel free to “borrow” my updated version of the banners. I’m not a graphics art guru but I was able to use SnagIt (a screen capturing tool I regularly use) to get the job done. All four of the banner images (.PNG format) are identical in pixel dimensions. The only difference between the four banners are the colors used. Once saved, the aqua (blue) and prairie (green) banners are 4KB, the default (red) banner is 8KB, and the autumn (orange) banner is 12KB. Can someone can please explain that to me? First person with the correct answer wins a new MacBook w/o firewire (just kidding).

Added a few plugins:
Akismet (spam filter for comments)
Flickr Manager (using my own bandwidth for hosting images wasn’t going to be feasible)
Feedburner (again, a bandwidth saving measure, but also a feed statistics tool as well)
Related Posts (appends other potentially related posts to new posts)
TinyMCE Advanced (more wysiwyg editor goodness for writing)
Twitter Tools (displays Twitter updates in a sidebar widget)

ESX configuration script – display full path in COS

October 25th, 2008

MS-DOS old timers should appreciate this. I’ve been using this snippet for a few years in my ESX host deployment scripts.

ESX script to display the full path in COS


In your ESX console, instead of seeing this:

[root@host vmware]#

You’ll see this:

[root@host /etc/vmware]#

#Paste the following into PuTTY on your ESX host:

mv /etc/bashrc /etc/bashrc.old

sed -e “s/\\h \\\W/\\h \\\w/g” /etc/bashrc.old > /etc/bashrc

Close your PuTTY session and re-open to see the change take effect.

Tip for virtualizing Citrix servers involving user profiles

October 25th, 2008

I virtualize Citrix servers and have had great success since VI3 was released. One of the things I learned along the way was a conflict that was created when introducing VMware Tools to a Citrix server.

My Citrix users receive mandatory profiles when their first session is established with the Citrix server. Although the user is assigned a mandatory read only profile which lives in an isolated directory on each Citrix server, a profile bearing the user’s account name is still created under \Documents and Settings\<username>\. This is normal Windows Terminal Services behavior. Now, what’s supposed to happen is when the user logs off their Citrix session, the automatically created profile is supposed to be automatically deleted. However, the installation of VMware Tools will prevent the clean up and deletion of the profile. The next time that user logs on, a new profile folder is created with a .001 extension. Then .002.  Then .003.  And so on.  On a larger scale with many users logging on and logging off, many profile folders are created and then orphaned. Left undiscovered, several hundred orphan folders will be discovered within just a day or two depending on how many sessions the Citrix server handles.

The root cause is that a file named \Documents and Settings\<username>\Application Data\VMware\hgfs.dat cannot be deleted by Windows and thus the folder structure must remain in place. The VMware Tools installation is partly responsible for the conflict. When VMware Tools is installed, it appends a value in the Windows registry to








The value of hgfs is appended.

The fix is simple. Right-click ProviderOrder and choose Modify. In the Edit String Value dialog box, edit the value data string and remove the characters ,hgfs (including the leading comma). For example, if the data string contains LanmanWorkstation,hgfs then change it to LanmanWorkstation. If the value data string contains only hgfs, then erase it and leave the value data string empty.

Problem solved. Unfortunately only for the time being. The next time you upgrade VMware Tools on the Citrix VM, hgfs will be appended back in the registry and once again an accumulation of folders under \Documents and Settings\ will begin.

ESX –> ESXi Separation Anxiety

October 25th, 2008

For me, this topic could easily be part two in a series of many (Part one would have been my recent blog on ESXi partitioning). I’m an ESX guy. I grew up on the Service Console. But I must get familiar with ESXi. I firmly believe ESXi is the future and ESX will be put out to pasture. If you pay attention to what VMware has been saying lately, you’ll pick up the subtle hints. Maybe you have heard some of them:

  • Smaller hypervisors mean faster deployment times
  • Smaller hypervisors mean less code vulnerability, less risk to the environment, and less time spent patching the hypervisor
  • Smaller hypervisors can be embedded in server hardware
  • ESXi is free, lending itself well to rapid and wide spread implementation
  • Development efforts for scripting and automation are getting away from the Service Console (PowerShell, Host Profiles, Distributed Virtual Switch, etc.)

So my latest adventure in ESXi once again involved the “Tech Support Mode” console. I should really just stay out of there for my own good. Each time I go in there, I come out battered and bruised with more questions than I had going in. This time I was trying to troubleshoot an NPIV issue (future blog post). /var/log/vmkernel is what I wanted to take a look at but it wasn’t there. I searched the ESXi forums and found several hits referencing the vmkernel log and path validating my assumption that it did indeed exist in the spot where I was looking for it. Well, it doesn’t exist and the various posts in the ESXi forum referencing the existence /var/log/vmkernel are inaccurate, not to mention misleading. A quick conversation with helpful VMware employee dilpreet on the VMTN forums summed it up nicely:

All logging previously logged in vmkernel, vmkwarning is logged in /var/log/messages. There is no cron or dmesg log since they are not needed.

On that note, the supported method for viewing the Messages log is via the ESXi host console. Hit <F2>, enter the necessary credentials, choose the View System Logs menu option, and from the menu on the right, choose <1>. You get a forward and backward scrollable consolidated log file. For those who pride themselves on their grep or vi skills, your search capability has been reduced to simply hitting the </> key in the log viewer for RegExp Search.

I’ll get there eventually with ESXi. I’m just a little behind on the adoption and need to catch up with what others have already learned months ago. Admittedly, some of this is a pride issue.

Looking for VMware Tools?

October 25th, 2008

Are you looking for VMware Tools? Maybe just curious where they come from?

Look no further than your ESX host under /vmimages/tools-isoimages. There you’ll find the .iso files that mount as images into the virtual CD-ROM tray when the “Install/Upgrade VMware Tools” command is passed to the ESX host for a particular VM or group of VMs.

What else is in /vmimages? Browse to the floppies folder and you’ll find vmscsi- which is the optimized BusLogic SCSI driver for Windows guests. Although this driver will work for all Windows guest types, it’s best used with Windows NT and Windows 2000. Windows XP, Windows 2003, and beyond achieve better performance using the LSI Logic SCSI driver (which by the way can be downloaded from the LSI Logic website by following this link)