VMware vCenter Operations Manager Essentials

July 10th, 2014 by jason No comments »

A new vSphere book has just arrived and has been added to my library. The book’s title is VMware vCenter Operations Manager Essentials which was authored by Technical Virtualization Architect and vExpert Lauren Malhoit (@malhoit) with reviews from Michael Poore, Mike Preston, and Chris Wahl.

I ordered this book while attending Dell User Forum a few weeks ago where I did some breakout session speaking on vC Ops and the new Dell Storage adapters for vC Ops.

“This book is written for administrators, engineers, and architects of VMware vSphere as well as those who have or are interested in purchasing the vCenter Operations Manager Suite. It will particularly help administrators who are hoping to use vCenter Operations Manager to optimize their VMware environments as well as quickly troubleshoot both long-term and short-term issues.”

Skimming through the chapter list covering 236 pages, it looks like it’s going to be a pretty good read.

Chapter 1: Introduction to vCenter Operations Manager

Chapter 2: Installing vCenter Operations Manager

Chapter 3: Dashboards and Badges (badges?…. had to be said)

Chapter 4: Troubleshooting Our Virtual Environment with vCenter Operations Manager

Chapter 5: Capacity Planning with vCenter Operations Manager

Chapter 6: Reports

Chapter 7: vCenter Configuration Manager

Chapter 8: Log Insight

Chapter 9: VMware Horizon View Integration with vCenter Operations Manager

Chapter 10: vCenter Infrastructure Navigator

Chapter 11: EMC Storage Analytics

Why did I pick up this book? vC Ops is extremely powerful and it has a bit of a learning curve to it. This is what resonated with me the most when I first began using the product. Over time, vCenter has become an integral component in VMware vSphere virtualized datacenters and it will continue to do so as more and more applications and services are integrated with and become dependent on it. vC Ops ties together many datacenter infrastructure pieces and allows virtualization, IaaS, cloud computing, and VDI to be delivered more intelligently. I would like to learn more about vC Ops and hopefully pick up some helpful tips on building custom dashboards with stock and add-on adapters/collectors as well as custom widgets

Drive-through Automation with PowerGUI

July 9th, 2014 by jason 2 comments »

One of the interesting aspects of shared infrastructure is stumbling across configuration changes made by others who share responsibility in managing the shared environment. This is often the case in the lab but I’ve also seen it in every production environment I’ve supported to date as well. I’m not pointing any fingers – my back yard is by no means immaculate. Moreover, this bit is regarding automation, not placing blame (Note the former is productive while the latter is not).

Case in point this evening when I was attempting to perform a simple remediation of a vSphere 5.1 four-host cluster via Update Manager. I verified the patches and cluster configuration, hit the remediate button in VUM, and left the office.  VUM, DRS, and vMotion does the heavy lifting. I’ve done it a thousand times or more in the past in environments 100x this size.

I wrap up my 5pm appointment on the way home from the office, have dinner with the family, and VPN into the network to verify all the work was done. Except nothing had been accomplished. Remediation on the cluster was a failure.  Looking at the VUM logs reveals 75% of the hosts being remediated contain virtual machines with attached devices preventing VUM, DRS, and vMotion from carrying out the remediation.

Obviously I know how to solve this problem but to manually check and strip every VM of it’s offending device is going to take way too long. I know what I’m supposed to do here. I can hear the voices in my head of PowerShell gurus Alan, Luc, etc. saying over and over the well known automation battle cry “anything repeated more than once should be scripted!

That’s all well and good, I completely get it, but I’m in that all too familiar place of:

  1. Carrying out the manual tasks will take 30 minutes.
  2. Authoring, finding, testing a suitable PowerShell/PowerCLI script to automate will also take 30 minutes, probably more.
  3. FML, I didn’t budget time for either of the above.

There is a middle ground. I view it as drive-through efficiency automation. It’s call PowerGUI and it has been around almost forever. In fact, it comes from Quest which my employer now owns. And I’ve already got it along with the PowerPacks and Plug-ins installed on my new Dell Precision M4800 laptop. Establishing a PowerGUI session and authenticating with my current infrastructure couldn’t be easier. From the legacy vSphere Client, choose the Plug-ins pull down, PowerGUI Administrative Console.

The VMware vSphere Management PowerPack ships stock with not only the VM query to find all VMs with offensive devices attached, but also a method to highlight all the VMs and Disconnect.

Depending on the type of device connect to the virtual machines, VUM may also be able to handle the issue as it has the native ability to Disable any removable media devices connect to the virtual machines on the host. In this case, the problem is solved with automation (I won’t get beat up on Twitter) and free community (now Dell) automation tools. Remediation completed.

RVTools (current version 3.6) also has identical functionality to quickly locate and disconnect various devices across a virtual datacenter.  Click on the image below to read more about RVTools.

Click on the image below to read more about PowerGUI.

VMware vSphere Hardening Guides

June 7th, 2014 by jason No comments »

Quick security related resource pointer on a Saturday morning. Over the years I’ve been collecting the various vSphere hardening guide documents as they are released.  These guides can be used to lock down your own (or your customer’s) environment to prevent or isolate security related breaches and to satisfy internal or external IT audits. Thanks to Mike Foley, I noticed the vSphere Hardening Guide 5.5 Update 1 was released yesterday. You’ll find adds/moves/changes in the following categories:

  • General (VCM, etc.)
  • SSO
  • ESXi
  • Virtual Machines
  • vCenter Server and VCSA
  • VUM (Update Manager)
  • vSphere Web Client

If you haven’t yet, grab the guide, take a look at it, and upgrade to vSphere 5.5 Update 1, hopefully in that order.

In the past I recall these guides were spread out on somewhat sparsely on VMware’s site. What I hadn’t noticed until this morning is that VMware has now compiled all available vSphere hardening guide links onto a single landing page in addition to providing change tracking between each of the vSphere 5.x guides which I think is quite helpful.

VMware Trademark Guide

May 19th, 2014 by jason 2 comments »

Are you a technical writer? Blogger? Presenter? If so, this could be a handy resource for you.  It’s the VMware Trademark Guide.  Probably more important for VMware employees and their partners, with varying or less importance to bloggers.

…intended to provide guidance regarding the VMware brand names that tend to draw the greatest interest

I’ve seen a lot of citations with much justified debate around the spelling, capitalization, and acronymization of VMware products.  I believe this document to be the official source that should clear up any confusion.

The information is laid out in two columns: Brand Name and Approved Short Name/Acronym.

For example, VMware vSphere® Distributed Resource Scheduler™ has an approved short name/acronym of vSphere DRS.  To most people who have been around the products for a while, this may seem obvious.  However, with historical origins of DRS, HA (remember DAS?), FT, and vEverything, it has become commonplace to use and abuse the VMware brand with VMware-unofficial acronyms.  For instance, the guide goes on to say:

Use only approved short names. Most importantly, do not use abbreviations such as VCOPS, VCHS, VCNS, VSOM, ITBM and SRM to signify VMware products or services. Some of the abbreviations are being used informally, but should not be used in public-facing communications.

Wait… no VCOPS? No SRM?  Apparently it’s true (at least for public-facing communications and perhaps that’s the line that has been grossly forgotten and crossed) and I’m just as guilty as the next person for perpetuating wrongness in the vCommunity (Can I say that? To my knowledge, VMware doesn’t own that term on paper and has no jurisdiction).

Anyway, I don’t think the point is that people are going to get hauled off to jail for showing decks reflecting SRM and I’m quite sure this shorthand is still acceptable in social circles (with the added benefit of not being able to verbally screw up camel case).  The idea behind the document first and formost is to recognize each of the VMware registered trademarks and their proper use.  If nothing else, please identify the proper case and spelling of VMware.  If you’re a technical writer with a professional affiliation with VMware, it’s equally important to understand VMware’s requested use of short names and acronyms presumably so that we can maintain some consistency throughout the industry, minimize the confusion, and hopefully not slaughter VMware’s brand.

VMworld 2014 Justification Email

May 14th, 2014 by jason No comments »

I couldn’t find a 2014 version on the vmworld.com website so I resurrected a 2010 copy I had saved on my network (I’m not claiming to be the original author – I think this came from Troyer or someone in VMware or maybe the vCommunity) and I made a few updates.  I’m sharing the .PDF and .DOC version with friends and readers.  This is a two page request template designed to be sent via email (replace the highlighted sections with your own own name, name drop Duncan name to guarantee attendee assurance).

If you’re a prima donna, you can probably send a tweet sized justification.

If you’re a VCDX, why are you even here.  Why am I here?

See you at VMworld 2014.

http://boche.net/dropbox/vmworld-2014-justification-email.pdf

http://boche.net/dropbox/vmworld-2014-justification-email.doc

Registered Storage Providers Missing After vCenter 5.5 Update 1 Upgrade

March 17th, 2014 by jason No comments »

Taking a look at my VM Storage Policies compliance in the vSphere Client, I was alerted to a situation that none of my configured virtual machines were compliant with their assigned VM Storage Policy named “Five Nines Compellent Storage”.  Oddly enough, the virtual machine home directories and virtual disks were in fact on the correct datastores and showed as compliant a few days earlier. None had been migrated via Storage vMotion or SDRS.

Snagit Capture

Now you see it, now you don’t

I then verified my VASA configuration by looking at the status of my registered storage provider.  The issue was not so much that the provider was malfunctioning, but rather it was missing completely from the registered storage providers list.  This indeed explains the resulting Not Compliant status of my virtual machines.

Snagit Capture

I checked another upgraded environment where I know I had a registered VASA storage provider.  It reflected the same symptom and confirmed my suspicion that the recent process of upgrading to vCenter Server 5.5 appliance to Update 1 (via the web repository method) may have unregistered the storage provider once the reboot of the appliance was complete.

I had one more similar environment remaining which I had not upgraded yet. I verified the storage provider was registered and functioning prior to the Update 1 upgrade. I proceeded with the upgrade and after the reboot completed the storage provider was no longer registered.

What remains a mystery at this point is the root cause of the unregistered storage provider.  I was unable to find any VMware KB articles related to this issue.

Not the end of the world

The workaround is straightforward: re-register each of the missing storage providers.  For Dell Compellent customers, the storage provider points to the CITV (Compellent Integration Tools for VMware) appliance and the URL is follows the format:

https://fqdn:8443/vasa/services/vasaService

Snagit Capture

Dell Compellent customers should also keep the following in mind for VASA integration:

  • the integration requires the CITV appliance and Enterprise Manager 6.1 and above.
  • the out of box Windows Server Firewall configuration which Enterprise Manager sits on will block the initial VASA configuration in the CITV appliance. TCP 3033 incoming must be allowed or alternatively disable the Windows Firewall (not highly recommended).

Once the applicable storage provider(s) are added back, no additional VM Storage Policy reconfiguration is required other than to check for compliance.  All VMs should fall back into compliance.

Snagit Capture

Once again, I am unsure at this point as to why applying vCenter 5.5 Update 1 to the appliance caused the registered storage providers to go missing or what that connection is.  I will also add that I deployed additional vCenter 5.5 appliances under vCloud Director with a default configuration, no registered vSphere hosts, registered a VASA storage provider, upgraded to Update 1, rebooted, and the storage provider remained. I’m not sure what element in these subsequent tests caused the outcome to change but the problem itself now presents itself as inconsistent.  If I do see it again and find a root cause, as per usual I will be sure to update this article. To reiterate, Update 1 was applied in this case via the web repository method.  There are a few other methods available to apply Update 1 to the vCenter Server appliance and of course there is also the Windows version of vCenter Server – it is unknown by me if these other methods and versions are impacted the same way.

Looks like someone has a case of the Mondays

On a somewhat related note, during lab testing I did find that VM Storage Profiles configured via the legacy vSphere Client do not show up as configured VM Storage Policies in the next gen vSphere Web Client.  Likewise, VM Storage Policies created in the next gen vSphere Web Client are missing in the legacy vSphere Client.  However, registered storage providers themselves carry over from one client to the other – no issue there.  I guess the lesson here is to stick with a consistent method of creating, applying, and monitoring Profile-Driven Storage in your vSphere environment from a vSphere Client perspective.  As of the release of vSphere 5.5 going forward, that should be the next gen vSphere Web Client.  However, this client still seems to lack the ability to identify VASA provided storage capabilities on any given datastore although the entire list of possible capability strings is available by diving into VM Storage Policy configuration.

Last but not least, VMware KB 2004098 vSphere Storage APIs – Storage Awareness FAQ provides useful bits of information about the VASA side of vSphere storage APIs.  One item in that FAQ that I’ve always felt was worded a bit ambiguously in the context of vSphere consolidation is:

The Vendor Provider cannot run on the same host as the vCenter Server.

In most cases, the vCenter Server as well as the VASA integration component(s) will run as virtual machines.  Worded above as is, it would seem the vCenter Server (whether that be Windows or appliance based) cannot reside on the same vSphere host as the VASA integration VM(s).  That’s not at all what that statement implies and moreover it wouldn’t make much sense.  What it’s talking about is the use case of a Windows based vCenter Server.  In this case, Windows based VASA integration components must not be installed on the same Windows server being used to host vCenter Server.  For Dell Compellent customers, the VASA integration comes by way of the CITV appliance which runs atop a Linux platform. However, the CITV appliances does communicate with the Windows based Enterprise Manager Data Collector for VASA integration.  Technically, EM isn’t the provider, the CITV appliance is.  Personally I’d keep the EM and vCenter Server installations separate.  Both appreciate larger amounts of CPU and memory in larger environments and for the sake of performance, we don’t want these two fighting for resources during times of contention.

Failed to connect to VMware Lookup Service

March 14th, 2014 by jason No comments »

Judging by the search results returned by Google, it looks like my blog is among the few virtualization blogs remaining which does not have a writeup on this topic.  It’s Friday so… why not.

Scenario:  vSphere 5.5 Update 1 VMware vSphere Web Client fails to log into vCenter Server (appliance version) with the following error returned:

Failed to connect to VMware Lookup Service

https://fqdn:7444/lookupservice/sdk –

SSL certificate verification failed.

Snagit Capture

Contributing factors in my case which may have played a role in this once working environment:

  1. Recently upgraded vCenter 5.5.0 Server appliance to Update 1 (unlikely as other similar environments were not impacted after upgrade)
  2. This particular vCenter appliance was deployed as a vApp from a vCloud Director catalog (likely  but unknown at this time if a customization was possible or attempted during deployment)
  3. The hostname of the appliance may have been changed recently (very likely)

The solution is quite simple.

  1. Log into the vCenter Server appliance management interface (https://fqdn:5480/)
  2. Navigate to the Admin tab
  3. Certificate regeneration enabled: choose Yes
  4. Click the Submit button
  5. Navigate to the System tab
  6. Reboot the appliance

After the appliance reboots

  1. Log into the vCenter Server appliance management interface (https://fqdn:5480/)
  2. Navigate to the Admin tab
  3. Certificate regeneration enabled: choose No
  4. Click the Submit button
  5. Log out of the vCenter Server appliance management interface
  6. Log into the VMware vSphere Web Client normally

Admittedly I recalled the Certificate regeneration feature first by logging into the vCenter Server appliance management interface, but then verified with a search to ensure the purpose of the Certificate regeneration feature.  The search results turned up Failed to connect to VMware Lookup Service – SSL Certificate Verification Failed (among many other blog posts as mentioned earlier) in addition to VMware KB 20333338 Troubleshooting the vCenter Server Appliance with Single Sign-On login.  Both more or less highlight a discrepancy between the appliance hostname and the SSL certificate resulting in the need to regenerate the certificate to match the currently assigned hostname.

I ran across another issue this week during the Update 1 upgrade to the vCenter appliance which I may or may not get to writing about today.

At any rate, have wonderful and Software Defined weekend!