Archive for November, 2013

Storage Center 5.6 Released

November 25th, 2013

I don’t have the latest and greatest Dell Compellent SC8000 controllers or SC220 2.5″ drive enclosures in my home lab although I dream nightly about Santa unloading some on me this Christmas.  What I do have is an older Series 20 and I am thankful for that.  But having an older storage array doesn’t mean I cannot leverage some of the latest and greatest features and operating systems available for datacenters.

Storage Center 5.6 was released just a short time ago and it ushers in some feature and platform support currently built into Storage Center 6.x as well as a large number of bug fixes.  This is a big win for me and anyone with 32-bit system (Series 30 or below) needing these features because SCOS 6.x is 64-bit only for Series 40 and newer which today includes the SC8000.

So what are these new features in 5.6 and why am I so excited?  I’m glad you asked.  For this guy, and on top of the list, it’s full support of all VAAI primitives.  Storage Center 5.5 and older boasted support of the block zeroing primitive.  Space Reclamation was there as well although that primitve alone did not satisfy the other component of the thin provisioning primitive which was STUN.

Shown below a Storage Center 5.5 datastore where I lack Atomic Test and Set (aka Hardware Assisted Locking) and XCOPY.  I have block zeroing and Space Reclamation using the Free Space Recovery agent for vSphere guest VMs using physical RDMs. VAAI support status can be obtained in full using esxcli:

Snagit Capture

Or in part using the vSphere Client GUI:

Snagit Capture

After the Storage Center 5.6 upgrade, I’ve got additional VAAI primitive support where Clone in most cases is going to be the biggest one in terms of fabric and host efficiency and performance. Not shown is support for Thin Provisioning Stun but that has been added as well:

Snagit Capture

The vSphere Client GUI now reflects full VAAI support after the 5.6 upgrade:

Snagit Capture

What else? Added support for vSphere 5.5 as an operating system type:

Snagit Capture

Last but not least, added support for Windows 2012 and some of its features including Offloaded Data Transfer, Thin Provisioning, Space Reclamation, and Server Objects:

Snagit Capture

Storage Center 5.6 also adds new storage features which are storage host agnostic such as Background Media Scans (BMS) as well as improved disk and HBA management for server objects.  And the bug fixes I mentioned earlier – refer to the SCOS 5.6 Release Notes for details.

To wrap this up, if you’ve got an older Storage Center model and you want support for these new features while avoiding a forklift upgrade, Storage Center Operating System 5.6 is the way to go.

Microsoft Sysprep Change in vCloud Director 5.5

November 18th, 2013

If you’re like me, you still support legacy Windows operating systems from time to time.  Let’s face it, Windows Server 2003 was a great server operating system and will probably remain in some environments for quite a while.  I won’t at all be surprised if the Windows Server 2003 legacy outlasts that of Windows XP.  To that point, even the VCAP5-DCA  exam I sat a few weeks ago used Windows Server 2003 guests in the lab.

All of that being said in what is almost the year 2014, hopefully you are not still deploying Windows Server 2003 as a platform to deliver applications and services in a production environment.  However, if you are and you’re using VMware vCloud Director 5.5, you should be aware of subtle changes which I noticed while reading through the documentation.  Page 31 of the vCloud Director 5.5 Installation and Upgrade Guide to be exact.

In previous versions of vCloud Director including 5.1, Microsoft Sysprep files were placed in a temporary directory within operating system specific folders on the first cloud cell server in the cluster.  The next step was to invoke the /opt/vmware/vcloud-director/deploymentPackageCreator/ script which bundled all of the Sysprep files into a /opt/vmware/vcloud-director/guestcustomization/ file.  At this point, Sysprep was installed and configured on the first cell server.  It could then optionally be distributed by way of copying the .cab file and the file to the guestcustomization directory of the other cell servers in the cluster.  I call this step optional because not all vCloud deployments will have multiple cell servers.  If multiple cell servers did exist, you’d likely want all of them to be able to perform guest customization duties for legacy Windows operating systems in the catalog and thus this optional step would be required.

So a few things have changed now in 5.5.  First, the Windows operating system specific folder names have changed to match the folder names which vCenter Server has always used historically (see VMware KB 1005593) and on this note, Windows 2000 Server support has been put out to pasture in vCD 5.5.

Version pre-vCD 5.5 vCD 5.5
Windows 2000 /win2000 unsupported
Windows Server 2003 (32-bit) /win2k3 /svr2003
Windows Server 2003 (64-bit) /win2k3_64 /svr2003-64
Windows XP (32-bit) /winxp /xp
Windows XP (64-bit) /winxp_64 /xp-64

Next, the method to create the Sysprep package and distribute it to the other cell servers has changed.  The script no longer exists and as a result, a bundled .cab file is not created.  Instead, the Sysprep files are copied in their entirety to their new directory names within the directory /opt/vmware/vcloud-director/guestcustomization/default/windows/sysprep.  So what you need to do here is create the directory structure under $VCLOUD_HOME and SCP the Sysprep files to each of the cell servers.  I’ve provided the directory creation commands below:

mkdir -p /opt/vmware/vcloud-director/guestcustomization/default/windows/sysprep/svr2003

mkdir -p /opt/vmware/vcloud-director/guestcustomization/default/windows/sysprep/sv42003-64

mkdir -p /opt/vmware/vcloud-director/guestcustomization/default/windows/sysprep/xp

mkdir -p /opt/vmware/vcloud-director/guestcustomization/default/windows/sysprep/xp-64

As the documentation reminds us, the Sysprep files must be readable by the user vcloud.vcloud (this user is created on each cell server during the initial vCloud Director installation) and that can be ensured by running the following command:

chown -R vcloud.vcloud $VCLOUD_HOME/guestcustomization

These installation changes are important to note if you’re deploying a net new vCloud Director 5.5 environment and there is a need for legacy Windows OS vAPP guest customization.  A vCloud Director 5.5 upgrade from previous versions will perform the necessary Sysprep migration steps automatically.  Lastly, Sysprep won’t be needed in vCloud environments where guest customization isn’t required or legacy versions of Windows aren’t being deployed and customized (Beginning with Windows Vista and Windows Server 2008, Sysprep is bundled within the operating system).

Single Sign-On Warning 25000

November 12th, 2013

Up to this point, I’ve deployed several net new instances of vCenter Server 5.5 and of course its essential components including Single Sign-On, Inventory Service, next generation Web Client, and the legacy vSphere Client.  Most of these deployments leveraged the vCenter appliance.  Using the appliance is a very easy to deploy vCenter because all of the essential components are pre-installed in the appliance and only need to be configured.

One area I hadn’t tackled much yet are upgrades of existing Windows-based vCenter environments to vSphere 5.5.  Having recently completed an inline upgrade of vCloud Director 5.1.2 to 5.5, it was now time to upgrade said vCloud’s underlying vSphere 5.1 (Update 1a I believe) virtual infrastructure.   Prior to starting the upgrade, I took the necessary precautions of getting a point in time snapshot of the vCenter Server, the vCloud Director Cells, and the Microsoft SQL Server databases for each (three total: SSO, vCenter, and vCD).  I accomplished this using array based snapshots – in this case Dell Compellent Storage Center Replays.

I launched autorun from the vCenter 5.5 installation media.  I opted for the custom installation and started with the Single Sign-On (SSO) upgrade from 5.1 to 5.5.  During the installation, I was met with

Warning 25000.  Please verify that the SSL certificate for your vCenter Single Sign-On 5.1 SSL is not expired.  If it did expire, please replace it with a valid certificate before upgrading to vCenter Single Sign-On 5.5.

Snagit Capture

In this particular environment, self-signed certificates from VMware were in use.  I know that this environment was deployed new less than two years ago and a verification of the SSL certificates in use proved that none were expired.  But because SSO and vCenter are such integral components to vCloud Director, I didn’t want to proceed without further vetting this out.


Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511) – This KB article describes a situation in which Warning 25000 results when a registry value on the existing Windows-based SSO 5.1 server does not match a field on the SSL certificate.  The resolution involves simply changing the registry value to match that which is on the SSL certificate.  I won’t repeat the details because you can read the KB article yourself.  Furthermore it didn’t resolve the problem in this case because the field on my SSL certificate and the registry key were an identical match.

Upgrading to VMware vCenter Single Sign-On 5.5 displays the error: Warning 25000 (2061478) – This KB article describes a problem for which there is no resolution. However, there is a workaround and that involves changing service_id and files.  More detail is available in the KB article and again the symptoms in the log files weren’t a close match.

The Trouble With SSL Certificates and Upgrading to VMware SSO 5.5 – Then I took a look at Michael Webster’s blog article on precisely the same error message.  Michael briefly discusses the two SSL certificate deployment models and then digs into VMware KB 2060511 mentioned above.  While the information in Michael’s blog article reassured me I was not alone in my journey, KB 2060511 didn’t solve my problem either.  But sometimes the value of blog articles is not only in the original author’s content, but also in the follow up comments from the readers.  Such was the case here.  A number of Michael’s readers responded by saying they were essentially in the same boat I’m in – it sounds like KB 2060511, but in the end this article doesn’t have the solution because there was nothing wrong with their SSO registry values.  The readers found no choice but to push onward beyond Warning 25000 with fingers crossed.  As it turned out in my as well as with some others, Warning 25000 was benign in nature and the installation completed successfully with no rollback.

In summary, this blog post does not represent global authority to ignore Warning 25000.  Rather it is meant to highlight one particular scenario where Warning 25000 may present itself and the actions that were taken to work through the problem.  I can’t stress enough the importance of the SSO component of vCenter going forward.  If any conclusion can be drawn here, it is that a backup of the infrastructure components should be secured before committing to the upgrade steps.  In this case, snapshots are the quickest and easiest method to provide data protection and recovery.  Although vSphere snapshots would work in some deployment architectures, recovering an environment when the environment being upgraded is managing the snapshots could be a challenge.  That is why I chose an out of band array based snapshot in this instance.

I would also like to point out in closing that vSphere 5.5 is still relatively new and VMware appears to still be chasing down all possible causes, resolutions, and workarounds to Warning 25000.  New information as well as VMware KB articles may develop subsequent to this writing so it may be worth continuing your own Google searching beyond this point.

Have a great week!

VMware VCAP5-DCA Exam Experience

November 7th, 2013

Snagit Capture

For quite some time I’ve had it on my agenda to sit the VMware Certified Advanced Professional Datacenter Administrator (VCAP5-DCA) exam.  It was starting to bother me and since I hadn’t sat a VCAP5 exam since June of last year (the DCD) and because I didn’t want to let the upgrade path lapse, it was time.  So a month ago I scheduled the exam looking for the next available slot across three different Pearson VUE testing centers in the Twin Cities.  First available was 8am Wednesday November 6th.  So by now you’ll know what this blog post is about.

VCAP exams aren’t impossible but they aren’t easy either.  There’s a time investment in the preparation plus half a day spent in the exam room.  Then there is the price tag of the VCAP exam which stings but the sting is far worse if you have to pay the same again fee for a 2nd attempt.  I’ve taken four of them in the past and on each one, I’ve been challenged by the time management component as are many others.  I had spoken to a few others over the past six months and each of them were consistent in painting the same picture of their own experience in that they did not complete the exam or came nowhere close to completing before running out of time.  Now while it is still possible to pass the exam without actually finishing it, obviously points are left on the table and I didn’t want that to be a deciding factor on my own success or failure.

So I decided to once again up my strategy for a VCAP exam.  Instead of simply being conscious of time on the exam – in other words knowing when I need to move on to the next question, I wanted to improve my pace by increasing the tempo at which I work without sacrificing accuracy and hopefully without advancing to the next question before all tasks on the previous question were completed.  In the past I might have gone into the exam room without preparing as much as I really should have (I’ve been known to do that with VCAP beta exam attempts).  Instead I just relied on the skills I had built up to that point.  While that strategy mostly worked, I also spent a lot of precious time in deep thought over questions here and there because I hadn’t prepared enough for the breadth and depth of skills that were being measured.  That leads to running out of time before completing all of the questions on the exam and I didn’t want to go that route this time.

To accomplish this I needed to dig deep into the blueprint and really attack the weak areas, as unsexy as they may be.  Taking a look at the VCAP5-DCA blueprint, it wasn’t quite that bad but clearly there were a few areas I needed sharpen up on especially since I’ve been out of day to day management of large vSphere infrastructures for a few years which naturally kept me sharp enough at the time.  Fortunately I came across a fantastic study guide by Jason Langer and Josh Coen and sponsored by Veeam who has been nothing but great to the vCommunity.  I would summarize this 237 page guide as being balls-on dead-accurate as far as what you need to know for the exam (that said, the exam itself mapped very well to the blueprint – no surprises whatsoever).  These guys really did a great job in compiling all of the blueprint subject matter in one spot.  Often times there are multiple tools or methods to complete a task and this is true in the VCAP exam room.  Team Langer and Coen demonstrate the multiple methods available.

Another good resource I looked at is Rick Scherer’s VMware vSphere 5.0 Auto Deploy video on YouTube.  I’ll be honest – I haven’t been overly impressed with Auto Deploy and as such I never invested much time in all the PowerShell memorization required to build the depots, images, and rules for stateless ESXi deployment.  Bottom line here is it’s clearly on the exam blueprint and if you want to score some points on Auto Deploy, you must learn how to build, configure, and manage it.  Rick’s 30 minute video is no-nonsense and moves at a brisk pace making it look pretty easy actually.  Did I get it all memorized? Not quite but I knew how to grab at least a few points by setting up DHCP/GPXE, building the depot, adding custom VIBs, and cloning profiles and exporting either images or repositories.

Next is Canadian eh? Mike Preston who is building a great series called 8 weeks of VCAP.  Over there you’ll find some good content on Host Profiles, Auto Deploy, as well as other blueprint content.  Mike tells it like it is and has fun with it.

Last but not least, I took a look at Michael Webster’s blog post on his VCAP5-DCA experience.  It’s a good writing and buried within you’ll uncover several great tips for this particular exam that either he came up with or he learned from others.  I used those tips in the exam room this morning.  For instance, the tip about only using the vSphere Client from the first/main RDP session where the toolbar exists (as opposed to an RDP session into the vCenter Server and then launching the vSphere Client and subsequent Remote Consoles from there – it starts getting nasty with sessions within sessions within sessions).  Of course his bit on time management is dead-on accurate.  He also mentions the potential danger in skipping too many tasks.  While not all labs and tasks build on each other, a fair amount of them do.  If you do skip a task and need to go back, you can go backwards in the exam which is good but would chew up more valuable time.  Based on my exam experience, I would surmise that if one were to completely skip all tasks which had interrelated dependencies, it would be enough to fail.  That said, of course I don’t know VMware’s scoring rubric – it’s just a guess.  He also mentions the environment isn’t perfect.  That’s true and will likely vary from kit to kit.  I discovered a few items which impacted my results and I would guess weren’t designed to be part of the lab.  Unfortunately I was offered no areas to submit exam/lab feedback so I left the lab scoring proctor a nice long letter in the ‘notes’ field of one of the virtual machines hoping he/she would see it when scoring was performed on that particular VM.

All of the above is very helpful.  I would add that if you’re already good at most of the day to day garden variety vSphere administration with a mouse, explore some territory not often visited.  For example, step away from the familiar GUI tools and spend a day or two immersed in vCLI and/or PowerShell.  On the vCLI side of the house, one command you’ll want to get to know well is esxcli, particularly the pluggable storage architecture (PSA) components and management.  While most people probably use the vSphere Client or a vendor plug-in to manage their shared storage, you’ll need to understand the moving parts and how to accomplish the same tasks and more via CLI.  It might sound crazy but there is a level of satisfaction and appreciation making a datastore disappear and reappear with some MASK_PATH command line work.  There is also a good chunk of storage device management that can be performed using CLI where there is no GUI equivalent short of a storage vendor plug-in and I haven’t come across one yet that goes into that level of detail.  If it’s not completely obvious by now – you have to have a lab environment to work in.  As one of the resources above points out, you don’t need an enterprise lab, but you do need a pair of vSphere hosts that can run at least five VMs plus some flavor of shared storage, preferably block if choice must be made between block or file but best case scenario you have both available as the exam does have some NFS coverage.


Snagit Capture


Exam Format:

  • 26 live labs, each with multiple tasks varying in quantity and difficulty
  • 220 minutes for native English speakers
  • A fair amount of reading and comprehension but not overload
  • No coffee in the exam room
  • $400US (but I had a discount voucher for this one)

I wouldn’t say I studied long but I studied hard on the focus areas that needed attention.  I worked hard in the lab, did some whiteboarding to test memory retention, and it paid dividends.  I was able to move through the VCAP5-DCA exam swiftly and I was pretty shocked to have reached the end with a few minutes to spare.  I’ll go so far as to say I was actually enjoying myself for the first time in a long time in an exam room.  It certainly helps knowing the content well which in turn builds confidence through each of the completed labs.  Conversely, not knowing the material makes for an increasingly dismal situation as the exam progresses.  Did I make some mistakes?  Yes I made a few which I realized later in the day as I spent time reflecting.  I’d like a 2nd shot at those but it’s water under the bridge now.  While saying I reached the end with some time to spare is accurate, I need to qualify that with the fact that I skipped two labs which were going to be time intensive and potentially error prone.  I made the call to skip them in the interest of seeing more of the exam before running out of time.  Discounting those two labs, I did complete more of the exam than I actually expected and for that I am quite pleased.  I would say my approach of time and tempo management this round was a success.

So now I wait 15 business days for my results from VMware.  I feel confident but one never really knows with these exams.

Update 11-20-13: I received a passing grade from VMware this morning but I haven’t seen a score yet. That may come later.

Announcing the Mastering vSphere 5.5 Book Winners

November 1st, 2013

Snagit CaptureUpon returning from VMworld 2013 San Francisco, I kicked off a contest with an award pool containing five copies of the upcoming Mastering vSphere 5.5 book.  Amazon is reflecting an in-stock date of November 3rd and I’ve seen at least one mention on Twitter from an author that he has received his hard copy.  My pre-orders should be arriving soon and I should announce the winners of the contest.

A few details:

  1. My name is written in green approximately in cell AD4 along with the inscription “VMware is my life and passion”
  2. The contest entries came in very quickly and there were five winners inside of two hours or less.
  3. By design, I kicked off the contest late at night to give folks across the pond and on the other side of the world a chance.  From what I can tell, most of the winners are not from this continent so I’ll be sending books overseas.  Please be patient for their arrival.

And now the moment everyone has been waiting for – the winners are:
Nick Carbone, appears to be from LA
David Wilde, I’m guessing UK
Seb, somewhere in the UK
Fernando Martinez, Mannheim, Germany
Conor Buckley, Cork, Ireland

Thank you to all who participated and my congratulations go out to the winners.  I’ll be contacting each of you via email for your shipping address.