Tip for virtualizing Citrix servers involving user profiles

October 25th, 2008 by jason 11 comments »

I virtualize Citrix servers and have had great success since VI3 was released. One of the things I learned along the way was a conflict that was created when introducing VMware Tools to a Citrix server.

My Citrix users receive mandatory profiles when their first session is established with the Citrix server. Although the user is assigned a mandatory read only profile which lives in an isolated directory on each Citrix server, a profile bearing the user’s account name is still created under \Documents and Settings\<username>\. This is normal Windows Terminal Services behavior. Now, what’s supposed to happen is when the user logs off their Citrix session, the automatically created profile is supposed to be automatically deleted. However, the installation of VMware Tools will prevent the clean up and deletion of the profile. The next time that user logs on, a new profile folder is created with a .001 extension. Then .002.  Then .003.  And so on.  On a larger scale with many users logging on and logging off, many profile folders are created and then orphaned. Left undiscovered, several hundred orphan folders will be discovered within just a day or two depending on how many sessions the Citrix server handles.

The root cause is that a file named \Documents and Settings\<username>\Application Data\VMware\hgfs.dat cannot be deleted by Windows and thus the folder structure must remain in place. The VMware Tools installation is partly responsible for the conflict. When VMware Tools is installed, it appends a value in the Windows registry to

HKEY_LOCAL_MACHINE\

SYSTEM\

CurrentControlSet\

Control\

NetworkProvider\

Order\

ProviderOrder

The value of hgfs is appended.

The fix is simple. Right-click ProviderOrder and choose Modify. In the Edit String Value dialog box, edit the value data string and remove the characters ,hgfs (including the leading comma). For example, if the data string contains LanmanWorkstation,hgfs then change it to LanmanWorkstation. If the value data string contains only hgfs, then erase it and leave the value data string empty.

Problem solved. Unfortunately only for the time being. The next time you upgrade VMware Tools on the Citrix VM, hgfs will be appended back in the registry and once again an accumulation of folders under \Documents and Settings\ will begin.

ESX –> ESXi Separation Anxiety

October 25th, 2008 by jason 1 comment »

For me, this topic could easily be part two in a series of many (Part one would have been my recent blog on ESXi partitioning). I’m an ESX guy. I grew up on the Service Console. But I must get familiar with ESXi. I firmly believe ESXi is the future and ESX will be put out to pasture. If you pay attention to what VMware has been saying lately, you’ll pick up the subtle hints. Maybe you have heard some of them:

  • Smaller hypervisors mean faster deployment times
  • Smaller hypervisors mean less code vulnerability, less risk to the environment, and less time spent patching the hypervisor
  • Smaller hypervisors can be embedded in server hardware
  • ESXi is free, lending itself well to rapid and wide spread implementation
  • Development efforts for scripting and automation are getting away from the Service Console (PowerShell, Host Profiles, Distributed Virtual Switch, etc.)

So my latest adventure in ESXi once again involved the “Tech Support Mode” console. I should really just stay out of there for my own good. Each time I go in there, I come out battered and bruised with more questions than I had going in. This time I was trying to troubleshoot an NPIV issue (future blog post). /var/log/vmkernel is what I wanted to take a look at but it wasn’t there. I searched the ESXi forums and found several hits referencing the vmkernel log and path validating my assumption that it did indeed exist in the spot where I was looking for it. Well, it doesn’t exist and the various posts in the ESXi forum referencing the existence /var/log/vmkernel are inaccurate, not to mention misleading. A quick conversation with helpful VMware employee dilpreet on the VMTN forums summed it up nicely:

All logging previously logged in vmkernel, vmkwarning is logged in /var/log/messages. There is no cron or dmesg log since they are not needed.

On that note, the supported method for viewing the Messages log is via the ESXi host console. Hit <F2>, enter the necessary credentials, choose the View System Logs menu option, and from the menu on the right, choose <1>. You get a forward and backward scrollable consolidated log file. For those who pride themselves on their grep or vi skills, your search capability has been reduced to simply hitting the </> key in the log viewer for RegExp Search.

I’ll get there eventually with ESXi. I’m just a little behind on the adoption and need to catch up with what others have already learned months ago. Admittedly, some of this is a pride issue.

Looking for VMware Tools?

October 25th, 2008 by jason No comments »

Are you looking for VMware Tools? Maybe just curious where they come from?

Look no further than your ESX host under /vmimages/tools-isoimages. There you’ll find the .iso files that mount as images into the virtual CD-ROM tray when the “Install/Upgrade VMware Tools” command is passed to the ESX host for a particular VM or group of VMs.

What else is in /vmimages? Browse to the floppies folder and you’ll find vmscsi-1.2.0.2.flp which is the optimized BusLogic SCSI driver for Windows guests. Although this driver will work for all Windows guest types, it’s best used with Windows NT and Windows 2000. Windows XP, Windows 2003, and beyond achieve better performance using the LSI Logic SCSI driver (which by the way can be downloaded from the LSI Logic website by following this link)

ESX partitioning a lost art form in ESXi

October 24th, 2008 by jason 17 comments »

On the heels of Duncan Epping’s blog article about ESX partitioning best practices comes my first look at the automatic partitioning in ESXi.

Before I dive into ESXi, allow me to provide a brief primer on the history of ESX partitioning as I see it. Once upon a time, long before the coming of ESXi and when there was only ESX, there existed great debates in books, whitepapers, knowledgebase articles, datacenters, forums, and blogs about how best to properly partition ESX. Careful planning up front meant better optimization, stability, and uptime of the ESX host, saving the company money from poor host performance, reduced downtime, and unplanned outages. The ESX administrator also slept better at night. What is the big debate? ESX administrators evolve from varying backgrounds where they dealt with a range of operating systems. Each administrator brings their best ideas, experiences, and nightmares the he or she would probably like to forget, to the table. With the ESX Service Console (Console Operating System or COS for short) based on a version of Red Hat, Linux and Unix administrators were natively the best equipped to carry on an intelligent conversation of Linux partitioning “Do’s and Don’ts”. However, ESX did add a few twists in how it used the COS and the file system. Taking into account the native behavior of Red Hat in addition to the ESX specific characteristics, partitioning best practices evolved. While not every administrator will agree on the exact size a given partition should be, a pattern in how ESX is properly partitioned is fairly evident, plus or minus the partition size variance that fits the personal taste of the administrator or perhaps company baseline policies or standards. ESX partitioning strategy was an art form; maybe something to brag about when getting your geek on in a circle of peers. I’ll go out on a limb here and say that no self respecting ESX administrator used the automatic partitioning ESX offered even though it might have saved a little time and in fact was and is still today the default partitioning choice during an ESX installation. No offense intended, but the automatic partitioning in ESX was less than stellar. In fact, automatic partitioning sizes were not consistent with best practices taught in the Virtual Infrastructure classroom training. Right there is a good a reason as any to partition manually.

Recently, I began evaluating ESXi. Having been a seasoned ESX administrator for quite a while, one of the things I noticed about the ESXi installation (in addition to its incredibly fast installation time) was the partitioning configuration. Or lack thereof. Had I blown through the screens so quickly that I didn’t notice and accidentally performed automatic partitioning? It was no accident. A second installation revealed that ESXi makes the partitioning choices for us, and provides us no opportunity to size our own partitions short of deleting and recreating after the fact using fdisk in the unsupported “Tech Support Mode” console. That’s nothing but trouble. The only partitions you should be creating after the ESX/ESXi installation are VMFS volumes.

So I’m stuck with automatic partitioning in ESXi. It’s a paradigm change I could get used to if it’s technically sound. But what does it look like and what are the partitions in ESXi used for? An unsupported trek into the “Tech Support Mode” console allowed me to run some df -h, fdisk -l, and ls commands to find out.

On an 8GB disk, an installation of ESXi 3.5.0 Update 2 creates the following partitions automatically:

Vmhba1:0:0:1 Extended

Vmhba1:0:0:2 FAT16 4,095MB Looks like /. contains directories: downloads, uwswap, var

Vmhba1:0:0:3 VMFS 3,347MB datastore for VMs, ISOs, VMKernel swap

Vmhba1:0:0:4 FAT16 <32M 4MB Unknown

Vmhba1:0:0:5 FAT16 48MB contains miscellaneous files (boot.cfg, *.tgz, etc.)

Vmhba1:0:0:6 FAT16 48MB contains miscellaneous files (boot.cfg, *.tgz, etc.)

Vmhba1:0:0:7 VMKcore 110MB allocated for VMkernel core dump

Vmhba1:0:0:8 FAT16 540MB contains directories: core (for COS dumps?), opt

Ok, some of the partition sizes above were quite small. Understandably, I started out with a small 8GB disk, not giving ESXi much to work with. My next question was “would the auto created partition sizes be larger if I used a larger disk?”. So I reinstalled using a 16GB disk to find out.

On a 16GB disk, an installation of ESXi 3.5.0 Update 2 creates the following partitions automatically:

Vmhba1:0:0:1 Extended

Vmhba1:0:0:2 FAT16 4,095MB Looks like /. contains directories: downloads, uwswap, var

Vmhba1:0:0:3 VMFS 11,539MB datastore for VMs, ISOs, VMKernel swap

Vmhba1:0:0:4 FAT16 <32M 4MB Unknown

Vmhba1:0:0:5 FAT16 48MB contains miscellaneous files (boot.cfg, *.tgz, etc.)

Vmhba1:0:0:6 FAT16 48MB contains miscellaneous files (boot.cfg, *.tgz, etc.)

Vmhba1:0:0:7 VMKcore 110MB allocated for VMKernel core dump

Vmhba1:0:0:8 FAT16 540MB contains directories: core (for COS dumps?), opt

Between the 8GB install and the 16GB install, not a single partition size change with the exception of VMFS. It simply grew the VMFS partition to consume the additional 8GB of disk.

With this automatic partitioning scheme, how heavily are each of the partitions utilized out of the box?

Vmhba1:0:0:2 FAT16 4,095MB 25% used

Vmhba1:0:0:3 VMFS 11,539MB n/a

Vmhba1:0:0:4 FAT16 <32M 4MB Unknown

Vmhba1:0:0:5 FAT16 48MB 0% used

Vmhba1:0:0:6 FAT16 48MB 76% used

Vmhba1:0:0:7 VMKcore 110MB 0% used (and let’s hope for the sake of uptime it stays this way)

Vmhba1:0:0:8 FAT16 540MB 32% used

Bottom line: Am I comfortable with this partitioning? Time will tell if it’s adequate. I’ll keep a watchful eye on the experiences of others on the VMTN forums. I’m not using ESXi in production or development right now. Further information gathering on ESXi may reveal there are other deployment methods for ESXi that allow more granular control of its installation parameters. For now, I’m in the evaluation stage to see if ESXi is the right fit. It certainly carries with it some attractive attributes but there are also things I must learn to let go of such as the Service Console and all of the utility it provides.

The Lab – Then and Now

October 19th, 2008 by jason No comments »

I’ve completed a new blog page:

The Lab – Then and Now
A gallery documenting its evolution using VMware

To visit, just click the “Lab” menu item at the top of the page.

My First Blog and How To Install WordPress

October 18th, 2008 by jason 9 comments »

Well how special am I?  I had two paragraphs of my first blog post written and after changing to full screen mode, I lost everything I had typed!  And I was just about to comment how impressed I am with WordPress.  I still am but I just learned my first valuable lesson with this web based application.  I am actually blown away by the flexibility and number of versatile configuration screens the admin console has.  I am proof that with WordPress, anyone can have a professional looking blog, irregardless of content.  Now I just need to find a decent theme so my blog isn’t mistaken for RTFM Education.

So actually this isn’t quite my first ever blog post.  My first blog post was written several months ago on a different platform when I had five minutes to kill.  I also write family news once in a while on my boche.net site to keep our relatives up to date on current events around our house.

Before installing WordPress, I had a few thoughts on what my first blog post should be.  Helpful and insightful to the reader.  As I struggled for a few hours getting WordPress to work, I knew I had a topic in the making:  How To Install WordPress.  Well, not so much how to install and configure WordPress, but rather how to install and configure the prerequisite platform components so that WordPress works correctly.  Hopefully someone is able to follow the steps I did without wasting hours troubleshooting PHP, IIS, and directory permissions.  By the way, this is on Windows Server 2003.  The error message I was struggling when trying to use WordPress was “Your PHP installation appears to be missing the MySQL extension which is required by WordPress”.  Here are the steps I followed to get WordPress up and running successfully:

  1. You need a web server.  Install Windows Server 2003 and Internet Information Services 6 (IIS).  Windows Server 2008 and IIS 7 may work also, but that platform introduces some significant differences which I don’t have the tested steps for.  Create your new website in IIS Manager, or use the default website.
  2. Install MySQL 5.0.67.  I used the MSI installer (file name is mysql-essential-5.0.67-win32.msi).  Don’t use the .zip file.  Choose an installation drive letter that’s going to provide enough disk space for your WordPress database and any other MySQL databases you might want down the road.
  3. Create your WordPress database by using the MySQL Command Line Client from the Start Menu.  The command is CREATE DATABASE wordpress;   Don’t forget the trailing semi-colon.
  4. Install the MySQL Administrator which you can download from the MySQL website.  File name is mysql-gui-tools-5.0-r13-win32.msi.  Use MySQL Administrator to create a WordPress user account that will be used by WordPress to manage the WordPress database in MySQL.  When creating the user account, grant all MySQL roles for the user on the WordPress database.
  5. Install PHP 5.2.6 using the MSI installer (not the .zip file).  File name is php-5.2.6-win32-installer.msi.�
    1. Perform a custom install. 
    2. Web Server Setup:  IIS ISAPI module
    3. Important:  Choose two extensions to install:  MySQL and MySQLi
  6. The installer will perform several key configurations for you:
    1. Adds your PHP installation directory to your PATH statement allowing your server to find the file libmysql.dll
    2. Places the two extension .DLLs in the \ext\ directory which PHP based WordPress needs
    3. Modifies the php.ini file to activate the two extension .DLLs
    4. Creates a PHP Web Service Extension in IIS and sets to Allow
    5. Adds the .PHP application extension and appropriate verbs to IIS
    6. Creates the environment variable PHPRC=(path to your PHP directory)
  7. Next step, set the correct NTFS permissions on your PHP directory.  Allow subfolders and files to inherit.
    1. Internet Guest Account (IUSR_servername) = Read & Execute
    2. NETWORK SERVICE = Read & Execute
  8. Install WordPress 2.6.2 by unzipping the folder contents into the appropriate IIS directory structure for the website defined in step 1 above.
  9. Set the correct NTFS permissions on your WordPress installation directory.  Allow subfolders and files to inherit.
    1. Internet Guest Account (IUSR_servername) = Modify
    2. NETWORK SERVICE = Modify
  10. In IIS Manager, modify the properties of your website.  Documents tab.  Enable default content page:  Add:  index.php
  11. Restart the World Wide Web Publishing service
  12. You’re ready to install and configure WordPress.  Follow the instructions at http://codex.wordpress.org/Installing_WordPress

Using the PHP MSI installer with the appropriate options is the best use of your time.  Many Google internet searches on the error message above will have you performing steps that may or may not resolve your issues if you choose to install PHP manually.  None of the home brew solutions worked for me and resulted in a waste of time:

  1. Modifying the php.ini, changing the path for extension_dir=
  2. Modifying the php.ini, tweaking values for extension=
  3. Copying the file libmysql.dll to %systemroot%\system32\
  4. Adding and modifying environment variables
  5. Rebooting your web server

Special thanks to Michael Sharman and his website who ultimately pointed me in the direction of re-looking at the PHP MSI installer where I had missed the steps of adding the two MySQL extensions.

By the way, after struggling for a while on my production web server and not wanting to muck it up while spinning my wheels, I called on the assistance of a vanilla VMware virtual machine running Windows Server 2003 to successfully test my new installation steps (taking snapshots along the way).  VMware virtualization helps save the day (again).