Archive for December, 2008

VMware Update Manager plugin failures

December 8th, 2008

Roger Lund posted several links on his blog which I was personally interested in because I have dealt with them in one way shape or form. One of them was a potential resolution to the issue where the VIC loses connectivity to VMware Update Manager and the VUM plugin unloads. The error message is “Your session with the VMware Update Manager Server is no longer valid. The VMware Update Manager Client plugin will be unloaded from the VI Client”

12-7-2008 9-19-21 PM

This is an issue that I wouldn’t say I’m plagued with, however, it does pop up every few days and the easy fix is to simply re-enable the VUM plugin. It’s an inconvenience that I wanted to get to the bottom of some day when I had time, but thus far it hasn’t been a high priority. I had checked the VUM logs but was not able to determine anything conclusive.

At any rate, I was excited to see the link on Roger Lund’s blog pointing to VMware KB article 1007099 “Update Manager Client is randomly disabled”. The link discusses a potential solution of disabling anti-virus scanning of the VUM repository (where all the code and metadata is downloaded to). I performed this over the weekend by neutering Symantec Antivirus Corporate Edition and kept my fingers crossed.

Things were looking good until Sunday night when the VUM error popped up again. Oh well, back to the drawing board. If anyone has any other ideas, I’m all ears.

Additional spam filter and comment subscriptions installed

December 6th, 2008

I’ve been dealing with an increasing amount of comment spam since mid last week and I decided to do something proactive about it.  I’ve installed a new spam filter plugin called Simple Spam Filter for WordPress by tan tan noodles.  It leverages both Akismet and reCAPTCHA technologies but in a slightly non-traditional way which I prefer.

When a user submits a blog post comment, they are not automatically required to decode a reCAPTCHA banner image.

If their comment passes the Akismet filter and it passes filters integrated in the Simple Spam Filter, the comment will be posted as is.

If the comment fails either of the previously mentioned filter checks, then the comment submission will be challenged by a reCAPTCHA banner image.  If the reCAPTCHA banner image is decoded successfully, the comment will post.  If the reCAPTHA banner image is not decoded successfully, the comment is immediately rejected and discarded (it won’t wait for me in a spam queue for approval).

Since I haven’t used it before, I can’t guarantee this plugin is 100% bug free. If you are having any issues submitting comments, please let me know via email so I can look into it.  The last thing I want to do is hinder meaningful discussions or the functionality of this blog.

I also installed the Subscribe To Comments and WP-Mail-SMTP plugins earlier this evening which allow readers to subscribe to new comments as well as manage their existing comment subscriptions.  Notification is delivered via Email (hence the need for the Mail plugin).  I’d like to thank and recognize Rich Brambley over at for leading me to the comment subscription plugin.

Virtualization rematch corrections and clarifications

December 4th, 2008

In the December 2008 issue of Windows IT Pro magazine, Michael Otey publishes part two of his VMware ESX/Microsoft Hyper-V comparison. From the VMware side of the world, I felt the article was well written, fair, and mostly accurate, however, there are a few things that I wanted to point out to minimize the confusion for those still trying to decide which hypervisor and feature set is best for them.

  • For most of the article, ESX is referred to as ESX Server. VMware dropped the word “Server” from their bare metal hypervisor a while back. ESX Server is merely ESX or ESXi and should not be confused with VMware’s free hosted product VMware Server.
  • Page 22 mentions the most noticeable missing feature from the Virtual Infrastructure Client is a native Windows Explorer-like file manager and direct connection from host to host. This is incorrect. From the VIC, you can either double click or right click on any datastore seen by the host and choose “Browse Datastore”. From the Datastore Browser, files and folders of the VMFS volume structure can be copied, cut, moved, renamed, created, deleted, and downloaded to the desktop. To address the host to host communication piece, the scp command can be used in the service console (COS) of an ESX host to copy files to or from another ESX host (ok, it’s not pretty, but it’s an option that does exist and I’ve used it many times).
    12-4-2008 9-43-23 PM 12-4-2008 9-52-44 PM
  • Michael goes on to say the VIC doesn’t provide the ability to copy or clone VMs. I grant Michael that in this example the VIC is not as intuitive as the other VMware hosted product management consoles which provide the user menu driven options to clone VMs, however, as I explained in the bullet above, the Datastore Browser will copy VMs which accomplishes one part of a manual cloning process. Another cloning step I will talk about in the next bullet.
  • Lastly, Michael explains he doesn’t get a graphical option to register VMs. Once again, using the Datastore Browser, we can right click on the VM’s *.vmx configuration file and choose “Add to Inventory” which registers the VM on the host (this is equivalent to the vmware-cmd -s register <config_file_path> command in the service console).
    12-4-2008 9-57-51 PM

This appears to be a simple case of Mr. Otey missing the Datastore Browser feature in the VIC which I’ve shown does exist and provides great utility and improvement over the MUI based file manager we had in the legacy ESX 2.x days. The Datastore Browser is a direct result of VMware listening to and responding to end user feedback stating we weren’t satisfied with using out of band 3rd party utilities like WinSCP for ESX host file management. Michael’s conclusion was a five-diamond rating for ESX over Hyper-V. He goes on to recommend ESX to midsized-to-large businesses looking for performance and manageability. With ESXi offered for free from VMware, I think he is missing the value of performance and manageability for small businesses as well.

Speaking of Windows IT Pro magazine, little known fact, I was the winner of the Reader Challenge in the April 2000 issue. Back then, the magazine was called Windows 2000 Magazine (and before that Windows NT Magazine). Back in the Windows NT days, the magazine was as thick as a 20,000 family telephone book. These days, the magazine still has some good articles like those written by Michael Otey, but sadly it has dwindled to 65 pages, the majority of which seem to be filled with ads and they still boast a $54.95 annual subscription cost. I’m not sure how they sleep at night.

VMware product name changes

December 3rd, 2008

Quick update on a news item you may have already heard about. Remember those VMware product/component decoder rings you might have started working on after the VMworld 2008 announcements? It’s time for an update. VMware announced a handful of product name changes on Monday:

  1. VMware VirtualCenter is now VMware vCenter Server
  2. VMware vCenter is the family name for all management products
  3. VMware Lab Manager is now VMware vCenter Lab Manager (since it is in the management products family)
  4. The VMware vCenter prefix applies to the other products in the management products family as well
  5. VMware View is the family name for all VDI/VDM products
  6. VMware VDI is now VMware View
  7. VMware VDM is now VMware View Manager

I’m not real fond of name changes unless there is a good reason behind it. I’ll give VMware the benefit of the doubt that there was good reason to make these changes, although not knowing myself 100% what is up VMware’s sleeve, the timing is somewhat debatable. Couldn’t they have waited until the next generation of Virtual Infrastructure to align the products and components? Citrix did this with Presentation Server when they instantly re-branded it to XenApp. It confused a lot of people, especially the newcomers. I hope confusion among VMware customers is minimized. It’s going to take a little while for these new names to become second nature for me.

What do you think of the name changes? Feedback is always welcomed here.

LAN party!

December 3rd, 2008

Once or twice a year, I like to pack up the truck with computer equipment and head to a LAN party. Last weekend, a friend of mine at work hosted his annual “After Thanksgiving LAN Party”. We got together Saturday morning and set up. We played games and ate comfort food throughout the day, into the evening, and past midnight (this is the usual schedule). Games played were Team Fortress 2, Unreal Tournament 2004, and Battlefield II Special Forces. The Team Fortress 2 dedicated server was virtualized with VMware Workstation 6.5 – it ran flawlessly. Thank you Brad for putting on a great time as usual!


DSC00007 DSC00002 DSC00004 DSC00005 DSC00006

Know thy open snapshots

December 2nd, 2008

VMware snapshotting is a wonderful and powerful technology that affords IT and Developer staff great flexibility and recovery options with virtual machines (VMs) that weren’t so flexible with physical machines or flat out did not exist. With this technology comes the responsibility of using it properly and knowing its limitations. Snapshots have a shelf life that varies somewhere between the moment the snapshot was created and infinity. Boy, that was real helpful wasn’t it?

Let me see if I can explain a little better. When a snapshot is created, a delta file is created on the VMFS volume and in the folder where the VM resides. The initial size of the delta file is 16MB. The purpose of the delta file is to maintain the delta changes to virtual disks since the snapshot was taken. This would be any disk write I/O activity inside the guest VM OS.

12-2-2008 9-56-54 PM

Disk write I/O inside a guest VM may be seldom or it may be very active. It depends on the role of the VM and more specifically the software and features installed inside the VM. When the initial 16MB delta file fills to capacity with the delta changes it maintains, it dynamically increases its size by another 16MB. Once again, if and when the delta file fills to capacity with delta changes, it grows by another 16MB. For those who excel in math, our delta file is now 48MB in size. Do you see the pattern? The delta file will continue to grow in 16MB increments to a maximum size of the parent file (and in some cases very rapidly!) unless one of a few conditions is met:

  1. Someone closes the snapshot
  2. Someone creates an additional child snapshot (perpetuating a potential problem)
  3. The snapshot file somehow becomes corrupted before or during closing of the snapshot (bad news)
  4. The VMFS volume where the VM and delta file are stored runs out of available storage space (update your resume. All other VMs on the same VMFS volume, snapshotted or not, as well as VMKernel swap and VM logs are now also out of write space)

Let’s connect the dots. The amount of time a snapshot should be left open is going to vary because of factors identified above. The amount of available VMFS storage, the rate at which the delta file is growing since the VM was snapped, number of VMDKs snapped, decaying VM disk performance as the delta file becomes fragmented across non-contiguous spots on disk, time to recovery if the snapshot is lost and the VM has to be restored, your personal comfort level, etc. To compound the anxiety, there are likely other VI administrators in your shop or automated backups creating and leaving snapshots open that you are unaware of on a regular basis. The urgency to have all open snapshots on your radar has increased.

Unfortunately in the current builds, VMware doesn’t give us real good (or automated) visibility of open snapshots. I liken it to handing a loaded gun to a child – it’s only a matter of time before an accident happens. That analogy is quite extreme but it gets my point across on the importance of preventing such an accident from happening. What we have right now from the Virtual Infrastructure Client console (as well as a few of the hosted product consoles) is called the Snapshot Manager. Snapshot Manager displays open snapshots and their hierarchy – but only when we open Snapshot Manager and that’s on a VM by VM basis. Very tedious.

12-2-2008 9-33-35 PM

So how do we gain better visibility of snapshots that’s not going to tie up a bunch of our valuable time? Fortunately there are some good 3rd party solutions available for free to help us out. A few that I like are Xtravirt’s Snaphunter, RVTools, and Hyper9.

Snaphunter is a simple piece of code that you install on an ESX host and schedule scanning and emailed reports via CRON. I get two Snaphunter reports emailed to me daily at noon (1 for PROD storage, 1 for DEV storage):

12-2-2008 11-00-44 PM

RVTools is a .NET Windows application that you can run from your desktop and get visibility of all VMs managed by a VirtualCenter instance. In addition to snapshots, RVTools shows a bunch of other cool stuff. This utility is worth checking out:

12-2-2008 11-06-34 PM

Hyper9 is an up and coming enterprise architected product (currently in beta, GA to release in early 2009) which will report on open snapshots as well as a many other facets of the virtual infrastructure:

12-2-2008 11-23-01 PM

Snapshots are so easy to create and there in and of itself lies its Achilles heel – snapshot sprawl and lack of native tools from VMware to keep them under control and keep us safe from danger.

Go forth and virtualize – but let’s be safe out there.

What I’m reading

December 1st, 2008

VMware ESX Essentials in the Virtual Data Center by David Marshall, Stephen S. Beaver, and Jason W. McCarty

ISBN: 978-1420070279

I picked up this book at VMworld 2008.  Great book so far.  I’m credited on page 70 for building the lengthy ks.cfg file on display when actually I only contributed script snippets.  I didn’t write that whole thing.  My hunch is Steve Beaver wrote the majority of it.  Thanks anyways guys 🙂