Posts Tagged ‘vCenter Server’

vCenter Server 6 Appliance fsck failed

April 4th, 2016

A vCenter Server Appliance (vSphere 6.0 Update 1b) belonging to me was bounced and for some reason was unbootable. The trouble during the boot process begins with /dev/sda3 contains a file system with errors, check forced. At approximately 27% of the way through, the process terminates with fsck failed. Please repair manually and reboot.

Unable to access a bash# prompt from the current state of the appliance, I followed VMware KB 2069041 VMware vCenter Server Appliance 5.5 and 6.0 root account locked out after password expiration, particularly the latter portion of it which provides the steps to modify a kernel option in the GRUB bootloader to obtain a root shell (and subsequently run the e2fsck -y /dev/sda3 repair command.

The steps are outlined in VMware KB 2069041 and are simple to follow.

  1. Reboot the VCSA
  2. Be quick about highlighting the VMware vCenter Server appliance menu option (the KB article recommends hitting the space bar to stop the default countdown)
  3. p (to enter a root password and continue with additional commands the next step)
  4. e (to edit the boot command)
  5. Append init=/bin/bash (followed by Enter to return to the GRUB menu
  6. b (to start the boot process)

This is where e2fsck -y /dev/sda3 is executed to repair file system errors on /dev/sda3 and allow the VCSA to boot successfully.

When the process above completes, reboot the VCSA and that should be all there is to it.

Update 10/9/17: I ran into a similar issue with VCSA 6.5 Update 1 where the appliance wouldn’t boot and I was left at an emergency mode prompt. In this situation, following the steps above isn’t so straight forward in part due to the Photon OS splash screen and no visibility to the GRUB bootloader (following VMware KB 2081464). In this situation, I executed fsck /dev/sda3 at the emergency mode prompt answering yes to all prompts. After reboot, I found this did not resolve all of the issues. I was able to log in by providing the root password twice. The journalctl command revealed a problem with /dev/mapper/log_vg-log. Next I ran fsck /dev/mapper/log_vg-log again answering yes to all prompts to repair. When that was finished, the appliance was rebooted and came up operational.

VMware vCenter Cookbook

July 27th, 2015

Back in June, I was extended an offer from PACKT Publishing to review a new VMware book. I’ve got a lot on my plate at the moment but it sounded like an easier read and I appreciated the offer as well as the accommodation of my request for paperback in lieu of electronic copy so I accepted. I finished reading it this past weekend.

The book’s title is VMware vCenter Cookbook and it is PACKT’s latest addition to an already extensive Cookbook series (Interested in Docker, DevOps, or Data Science? There’s Cookbooks for that). Although it was first published in May 2015, the content isn’t quite so new as its coverage includes vSphere 5, and vSphere 5 only with specific focus on vSphere management via vCenter Server as the title of the book indicates. The author is Konstantin Kuminsky and as I mentioned earlier the book is made available in both Kindle and paperback formats.

Admittedly I’m not familiar with PACKT’s other Cookbooks but the formula for this one is much the same as the others I imagine: “Over 65 hands-on recipes to help you efficiently manage your vSphere environment with VMware vCenter”. Each of the recipes ties to a management task that an Administrator of a vSphere environment might need to carry out day to day, weekly, monthly, or perhaps annually. Some of the recipes can also be associated with and aid in design, architecture, and planning although I would not say these are not the main areas of focus. The majority of the text is operational in nature.

The recipes are organized by chapter and while going from one to the next, there may be a correlation, but often there is not. It should be clear at this point it reads like a cookbook, and not a mystery novel (although for review purposes I did read it cover to cover). Find the vCenter how-to recipe you need via the Table of Contents or the index and follow it. Expect no more and no less.

Speaking of the Table of Contents…

  • Chapter 1: vCenter Basic Tasks and Features
  • Chapter 2: Increasing Environment Availability
  • Chapter 3: Increasing Environment Scalability
  • Chapter 4: Improving Environment Efficiency
  • Chapter 5: Optimizing Resource Usage
  • Chapter 6: Basic Administrative Tasks
  • Chapter 7: Improving Environment Manageability

It’s a desktop reference (or handheld I suppose depending on your preferred consumption model) which walks you through vSphere packaging and licensing on one page, and NUMA architecture on the next. The focus is vCenter Server and perhaps more accurately vSphere management. Fortunately that means there is quite a bit of ESXi coverage as well with management inroads from vCenter, PowerShell, and esxcli. Both Windows and appliance vCenter Server editions are included as well as equally fair coverage of both vSphere legacy client and vSphere web client.

Bottom line: It’s a good book but it would have been better had it been released at least a year or two earlier. Without vSphere 6 coverage, there’s not a lot of mileage left on the odometer. In fairness I will state that many of the recipes will translate identically or closely to vSphere 6, but not all of them. To provide a few examples, VM templates and their best operational practices haven’t changed that much. On the other hand, there are significant differences between FT capabilities and limitations between vSphere 5 and vSphere 6. From a technical perspective, I found it pretty spot on which means the author and/or the reviewers did a fine job.

Thank you PACKT Publishing for the book and the opportunity.

Legacy vSphere Client Plug-in 1.7 Released for Storage Center

July 23rd, 2014

Dell Compellent Storage Center customers who use the legacy vSphere Client plug-in to manage their storage may have noticed that the upgrade to PowerCLI 5.5 R2 which released with vSphere 5.5 Update 1 essentially “broke” the plug-in. This forced customers to make the decision to stay on PowerCLI 5.5 in order to use the legacy vSphere Client plug-in, or reap the benefits of the PowerCLI 5.5 R2 upgrade with the downside being they had to abandon use of the legacy vSphere Client plug-in.

For those that are unaware, there is a 3rd option and that is to leverage vSphere’s next generation web client along with the web client plug-in released by Dell Compellent last year (I talked about it at VMworld 2013 which you can take a quick look at below).

Although VMware strongly encourages customers to migrate to the next generation web client long term, I’m here to tell you that in the interim Dell has revd the legacy client plug-in to version 1.7 which is now compatible with PowerCLI 5.5 R2.  Both the legacy and web client plug-ins are free and quite beneficial from an operations standpoint so I encourage customers to get familiar with the tools and use them.

Other bug fixes in this 1.7 release include:

  • Datastore name validation not handled properly
  • Create Datastore, map existing volume – Server Mapping will be removed from SC whether or not it was created by VSP
  • Add Raw Device wizard is not allowing to uncheck a host once selected
  • Remove Raw Device wizard shows wrong volume size
  • Update to use new code signing certificate
  • Prevent Datastores & RDMs with underlying Live Volumes from being expanded or deleted
  • Add support for additional Flash Optimized Storage Profiles that were added in SC 6.4.2
  • Block size not offered when creating VMFS-3 Datastore from Datacenter menu item
  • Add Raw Device wizard is not allowing a host within the same cluster as the select host to be unchecked once it has been selected
  • Add RDM wizard – properties screen showing wrong or missing values
  • Expire Replay wizard – no error reported if no replays selected
  • Storage Consumption stats are wrong if a Disk folder has more than one Storage Type

Failed to connect to VMware Lookup Service

March 14th, 2014

Judging by the search results returned by Google, it looks like my blog is among the few virtualization blogs remaining which does not have a writeup on this topic.  It’s Friday so… why not.

Scenario:  vSphere 5.5 Update 1 VMware vSphere Web Client fails to log into vCenter Server (appliance version) with the following error returned:

Failed to connect to VMware Lookup Service

https://fqdn:7444/lookupservice/sdk –

SSL certificate verification failed.

Snagit Capture

Contributing factors in my case which may have played a role in this once working environment:

  1. Recently upgraded vCenter 5.5.0 Server appliance to Update 1 (unlikely as other similar environments were not impacted after upgrade)
  2. This particular vCenter appliance was deployed as a vApp from a vCloud Director catalog (likely  but unknown at this time if a customization was possible or attempted during deployment)
  3. The hostname of the appliance may have been changed recently (very likely)

The solution is quite simple.

  1. Log into the vCenter Server appliance management interface (https://fqdn:5480/)
  2. Navigate to the Admin tab
  3. Certificate regeneration enabled: choose Yes
  4. Click the Submit button
  5. Navigate to the System tab
  6. Reboot the appliance

After the appliance reboots

  1. Log into the vCenter Server appliance management interface (https://fqdn:5480/)
  2. Navigate to the Admin tab
  3. Certificate regeneration enabled: choose No
  4. Click the Submit button
  5. Log out of the vCenter Server appliance management interface
  6. Log into the VMware vSphere Web Client normally

Admittedly I recalled the Certificate regeneration feature first by logging into the vCenter Server appliance management interface, but then verified with a search to ensure the purpose of the Certificate regeneration feature.  The search results turned up Failed to connect to VMware Lookup Service – SSL Certificate Verification Failed (among many other blog posts as mentioned earlier) in addition to VMware KB 20333338 Troubleshooting the vCenter Server Appliance with Single Sign-On login.  Both more or less highlight a discrepancy between the appliance hostname and the SSL certificate resulting in the need to regenerate the certificate to match the currently assigned hostname.

I ran across another issue this week during the Update 1 upgrade to the vCenter appliance which I may or may not get to writing about today.

At any rate, have wonderful and Software Defined weekend!

vCenter Server Appliance 5.5 root account locked out after password expiration

January 10th, 2014

Thanks to Chris Colotti, I learned of a new VMware KB article today which could potentially have wide spread impact, particularly in lab, development, or proof of concept environments.  The VMware KB article number is 2069041 and it is titled The vCenter Server Appliance 5.5 root account locked out after password expiration.

In summary, the root account of the vCenter Server Appliance version 5.5 becomes locked out 90 days after deployment or root account password change.  This behavior is by design which follows a security best practice of password rotation.  In this case, the required password rotation interval is 90 days after which the account will be forcefully locked out if not changed.

The KB article describes processes to prevent a forced lockout as well as unlocking a locked out root account.

Approximately 90 days have elapsed since the release of vSphere 5.5 and I imagine this issue will quickly begin surfacing in large numbers where the vCenter Server Appliance 5.5 has been deployed using system defaults.

Update 6/16/16: For more information on vCenter Server Appliance password policies, including the local root account, check out vCSA 6.0 tricks: shell access, password expiration and certificate warnings.

Single Sign-On Warning 25000

November 12th, 2013

Up to this point, I’ve deployed several net new instances of vCenter Server 5.5 and of course its essential components including Single Sign-On, Inventory Service, next generation Web Client, and the legacy vSphere Client.  Most of these deployments leveraged the vCenter appliance.  Using the appliance is a very easy to deploy vCenter because all of the essential components are pre-installed in the appliance and only need to be configured.

One area I hadn’t tackled much yet are upgrades of existing Windows-based vCenter environments to vSphere 5.5.  Having recently completed an inline upgrade of vCloud Director 5.1.2 to 5.5, it was now time to upgrade said vCloud’s underlying vSphere 5.1 (Update 1a I believe) virtual infrastructure.   Prior to starting the upgrade, I took the necessary precautions of getting a point in time snapshot of the vCenter Server, the vCloud Director Cells, and the Microsoft SQL Server databases for each (three total: SSO, vCenter, and vCD).  I accomplished this using array based snapshots – in this case Dell Compellent Storage Center Replays.

I launched autorun from the vCenter 5.5 installation media.  I opted for the custom installation and started with the Single Sign-On (SSO) upgrade from 5.1 to 5.5.  During the installation, I was met with

Warning 25000.  Please verify that the SSL certificate for your vCenter Single Sign-On 5.1 SSL is not expired.  If it did expire, please replace it with a valid certificate before upgrading to vCenter Single Sign-On 5.5.

Snagit Capture

In this particular environment, self-signed certificates from VMware were in use.  I know that this environment was deployed new less than two years ago and a verification of the SSL certificates in use proved that none were expired.  But because SSO and vCenter are such integral components to vCloud Director, I didn’t want to proceed without further vetting this out.

Google.

Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511) – This KB article describes a situation in which Warning 25000 results when a registry value on the existing Windows-based SSO 5.1 server does not match a field on the SSL certificate.  The resolution involves simply changing the registry value to match that which is on the SSL certificate.  I won’t repeat the details because you can read the KB article yourself.  Furthermore it didn’t resolve the problem in this case because the field on my SSL certificate and the registry key were an identical match.

Upgrading to VMware vCenter Single Sign-On 5.5 displays the error: Warning 25000 (2061478) – This KB article describes a problem for which there is no resolution. However, there is a workaround and that involves changing service_id and service.properties files.  More detail is available in the KB article and again the symptoms in the log files weren’t a close match.

The Trouble With SSL Certificates and Upgrading to VMware SSO 5.5 – Then I took a look at Michael Webster’s blog article on precisely the same error message.  Michael briefly discusses the two SSL certificate deployment models and then digs into VMware KB 2060511 mentioned above.  While the information in Michael’s blog article reassured me I was not alone in my journey, KB 2060511 didn’t solve my problem either.  But sometimes the value of blog articles is not only in the original author’s content, but also in the follow up comments from the readers.  Such was the case here.  A number of Michael’s readers responded by saying they were essentially in the same boat I’m in – it sounds like KB 2060511, but in the end this article doesn’t have the solution because there was nothing wrong with their SSO registry values.  The readers found no choice but to push onward beyond Warning 25000 with fingers crossed.  As it turned out in my as well as with some others, Warning 25000 was benign in nature and the installation completed successfully with no rollback.

In summary, this blog post does not represent global authority to ignore Warning 25000.  Rather it is meant to highlight one particular scenario where Warning 25000 may present itself and the actions that were taken to work through the problem.  I can’t stress enough the importance of the SSO component of vCenter going forward.  If any conclusion can be drawn here, it is that a backup of the infrastructure components should be secured before committing to the upgrade steps.  In this case, snapshots are the quickest and easiest method to provide data protection and recovery.  Although vSphere snapshots would work in some deployment architectures, recovering an environment when the environment being upgraded is managing the snapshots could be a challenge.  That is why I chose an out of band array based snapshot in this instance.

I would also like to point out in closing that vSphere 5.5 is still relatively new and VMware appears to still be chasing down all possible causes, resolutions, and workarounds to Warning 25000.  New information as well as VMware KB articles may develop subsequent to this writing so it may be worth continuing your own Google searching beyond this point.

Have a great week!

A Look At vCenter 5.5 SSO RC Installation

August 30th, 2013

This week at VMworld 2013, I attended a few sessions directly related to vCenter 5.5 as well as its components, one of which is vCenter Single Sign On (SSO):

  • VSVC5234 – Extreme Performance Series: vCenter of the Universe
  • VSVC4830 – vCenter Deep Dive

First of all, both sessions were excellent and I highly recommend viewing them if you have access to the post conference recordings. 

If you followed my session tweets or if perhaps you’ve read half a dozen or more already available blog posts on the subject, you know that several improvements have been made to vCenter SSO for the vSphere 5.5 release.  For instance:

  • Completely re-written from the ground
  • Multi-master architecture
  • Native replication mechanism
  • SSO now has site awareness (think of the possibilities for HA stretched clusters)
  • MMC based diagnostic suite available as a separately maintained download
  • The external database and its preparation dependency has been removed
  • Database patitioning to improve both scalability and performance (this was actually added in 5.1 but I wanted to call it out)
  • Revamped multi-site deployment architecture
  • Full Mac OS X web client support including remote console
  • Improved certificate management
  • Multi-tenant capabilities
  • Drag ‘n’ Drop in the 5.5 web client

With some of the new features now identified and VMware’s blessing, have a look at the installation screens and see if you can spot the differences as compared to a vCenter 5.1 SSO installation.  These stem from a manual installation of SSO, not an automated installation of all vCenter components (by the way, the next gen web client is now installed as part of an automated vCenter 5.5 installation whereas it was not in 5.1).  Keep in mind these were pulled from a release candidate version and may change when vCenter 5.5 GAs at a future date.

I noticed one subtle change here – clicking on the Microsoft .NET 3.5 SP1 link in Windows 2008R2 actually installs the feature rather than just throwing up a dialogue box asking you to install the feature yourself.

Snagit Capture

As this is a manual installation, we have the option to use the default or specify the installation location.  Best practice is to install all vCenter components together so that they can communicate at server bus speed and won’t be impacted by network latency.  However, for larger scale environments, SSO should be isolated on a separate server with five or more vCenter Servers in the environment.  On a somewhat related note, the Inventory Service may benefit from an installation on SSD, again in large infrastructures.

Snagit Capture

We won’t likely see this in the GA version.

Snagit Capture

We’re going through the process of installing vCenter version 5.5 but in terms of the SSO component, again this is a complete re-write and bears the respective version of 2.0.

Snagit Capture

We always read the EULA in full and agree to the license terms and conditions.

Snagit Capture

Snagit Capture

Big changes here.  Note the differences in the deployment models compared to the previous 5.1 version – previous deployment models are honored through an upgrade to 5.5.  Again, this is where the VMworld sessions noted above really go into detail. 

Snagit Capture

the System-Domain namespace has been replaced with vsphere.local.

Snagit Capture

The new site awareness begins here.

Snagit Capture

Snagit Capture

Snagit Capture

Snagit Capture

I hope you agree that SSO installation in vCenter 5.5 has been simplified while many new features have been added at the same time.

As always, thank you for reading and it was a pleasure to meet and see everyone again this year at VMworld.