Registered Storage Providers Missing After vCenter 5.5 Update 1 Upgrade

March 17th, 2014 by jason No comments »

Taking a look at my VM Storage Policies compliance in the vSphere Client, I was alerted to a situation that none of my configured virtual machines were compliant with their assigned VM Storage Policy named “Five Nines Compellent Storage”.  Oddly enough, the virtual machine home directories and virtual disks were in fact on the correct datastores and showed as compliant a few days earlier. None had been migrated via Storage vMotion or SDRS.

Snagit Capture

Now you see it, now you don’t

I then verified my VASA configuration by looking at the status of my registered storage provider.  The issue was not so much that the provider was malfunctioning, but rather it was missing completely from the registered storage providers list.  This indeed explains the resulting Not Compliant status of my virtual machines.

Snagit Capture

I checked another upgraded environment where I know I had a registered VASA storage provider.  It reflected the same symptom and confirmed my suspicion that the recent process of upgrading to vCenter Server 5.5 appliance to Update 1 (via the web repository method) may have unregistered the storage provider once the reboot of the appliance was complete.

I had one more similar environment remaining which I had not upgraded yet. I verified the storage provider was registered and functioning prior to the Update 1 upgrade. I proceeded with the upgrade and after the reboot completed the storage provider was no longer registered.

What remains a mystery at this point is the root cause of the unregistered storage provider.  I was unable to find any VMware KB articles related to this issue.

Not the end of the world

The workaround is straightforward: re-register each of the missing storage providers.  For Dell Compellent customers, the storage provider points to the CITV (Compellent Integration Tools for VMware) appliance and the URL is follows the format:

https://fqdn:8443/vasa/services/vasaService

Snagit Capture

Dell Compellent customers should also keep the following in mind for VASA integration:

  • the integration requires the CITV appliance and Enterprise Manager 6.1 and above.
  • the out of box Windows Server Firewall configuration which Enterprise Manager sits on will block the initial VASA configuration in the CITV appliance. TCP 3033 incoming must be allowed or alternatively disable the Windows Firewall (not highly recommended).

Once the applicable storage provider(s) are added back, no additional VM Storage Policy reconfiguration is required other than to check for compliance.  All VMs should fall back into compliance.

Snagit Capture

Once again, I am unsure at this point as to why applying vCenter 5.5 Update 1 to the appliance caused the registered storage providers to go missing or what that connection is.  I will also add that I deployed additional vCenter 5.5 appliances under vCloud Director with a default configuration, no registered vSphere hosts, registered a VASA storage provider, upgraded to Update 1, rebooted, and the storage provider remained. I’m not sure what element in these subsequent tests caused the outcome to change but the problem itself now presents itself as inconsistent.  If I do see it again and find a root cause, as per usual I will be sure to update this article. To reiterate, Update 1 was applied in this case via the web repository method.  There are a few other methods available to apply Update 1 to the vCenter Server appliance and of course there is also the Windows version of vCenter Server – it is unknown by me if these other methods and versions are impacted the same way.

Looks like someone has a case of the Mondays

On a somewhat related note, during lab testing I did find that VM Storage Profiles configured via the legacy vSphere Client do not show up as configured VM Storage Policies in the next gen vSphere Web Client.  Likewise, VM Storage Policies created in the next gen vSphere Web Client are missing in the legacy vSphere Client.  However, registered storage providers themselves carry over from one client to the other – no issue there.  I guess the lesson here is to stick with a consistent method of creating, applying, and monitoring Profile-Driven Storage in your vSphere environment from a vSphere Client perspective.  As of the release of vSphere 5.5 going forward, that should be the next gen vSphere Web Client.  However, this client still seems to lack the ability to identify VASA provided storage capabilities on any given datastore although the entire list of possible capability strings is available by diving into VM Storage Policy configuration.

Last but not least, VMware KB 2004098 vSphere Storage APIs – Storage Awareness FAQ provides useful bits of information about the VASA side of vSphere storage APIs.  One item in that FAQ that I’ve always felt was worded a bit ambiguously in the context of vSphere consolidation is:

The Vendor Provider cannot run on the same host as the vCenter Server.

In most cases, the vCenter Server as well as the VASA integration component(s) will run as virtual machines.  Worded above as is, it would seem the vCenter Server (whether that be Windows or appliance based) cannot reside on the same vSphere host as the VASA integration VM(s).  That’s not at all what that statement implies and moreover it wouldn’t make much sense.  What it’s talking about is the use case of a Windows based vCenter Server.  In this case, Windows based VASA integration components must not be installed on the same Windows server being used to host vCenter Server.  For Dell Compellent customers, the VASA integration comes by way of the CITV appliance which runs atop a Linux platform. However, the CITV appliances does communicate with the Windows based Enterprise Manager Data Collector for VASA integration.  Technically, EM isn’t the provider, the CITV appliance is.  Personally I’d keep the EM and vCenter Server installations separate.  Both appreciate larger amounts of CPU and memory in larger environments and for the sake of performance, we don’t want these two fighting for resources during times of contention.

Failed to connect to VMware Lookup Service

March 14th, 2014 by jason No comments »

Judging by the search results returned by Google, it looks like my blog is among the few virtualization blogs remaining which does not have a writeup on this topic.  It’s Friday so… why not.

Scenario:  vSphere 5.5 Update 1 VMware vSphere Web Client fails to log into vCenter Server (appliance version) with the following error returned:

Failed to connect to VMware Lookup Service

https://fqdn:7444/lookupservice/sdk -

SSL certificate verification failed.

Snagit Capture

Contributing factors in my case which may have played a role in this once working environment:

  1. Recently upgraded vCenter 5.5.0 Server appliance to Update 1 (unlikely as other similar environments were not impacted after upgrade)
  2. This particular vCenter appliance was deployed as a vApp from a vCloud Director catalog (likely  but unknown at this time if a customization was possible or attempted during deployment)
  3. The hostname of the appliance may have been changed recently (very likely)

The solution is quite simple.

  1. Log into the vCenter Server appliance management interface (https://fqdn:5480/)
  2. Navigate to the Admin tab
  3. Certificate regeneration enabled: choose Yes
  4. Click the Submit button
  5. Navigate to the System tab
  6. Reboot the appliance

After the appliance reboots

  1. Log into the vCenter Server appliance management interface (https://fqdn:5480/)
  2. Navigate to the Admin tab
  3. Certificate regeneration enabled: choose No
  4. Click the Submit button
  5. Log out of the vCenter Server appliance management interface
  6. Log into the VMware vSphere Web Client normally

Admittedly I recalled the Certificate regeneration feature first by logging into the vCenter Server appliance management interface, but then verified with a search to ensure the purpose of the Certificate regeneration feature.  The search results turned up Failed to connect to VMware Lookup Service – SSL Certificate Verification Failed (among many other blog posts as mentioned earlier) in addition to VMware KB 20333338 Troubleshooting the vCenter Server Appliance with Single Sign-On login.  Both more or less highlight a discrepancy between the appliance hostname and the SSL certificate resulting in the need to regenerate the certificate to match the currently assigned hostname.

I ran across another issue this week during the Update 1 upgrade to the vCenter appliance which I may or may not get to writing about today.

At any rate, have wonderful and Software Defined weekend!

VMware Releases vSphere PowerCLI 5.5 R2

March 12th, 2014 by jason No comments »

I stumbled across some interesting news shared by Alan Renouf on Facebook this morning – an R2 release of vSphere PowerCLI 5.5 (Build 1649237).  New in R2 per the release notes:

  • Access to the vCenter Server SRM public API (Connect-SRMServer and Disconnect-SRMServer cmdlets) – an exciting addition for sure
  • Support for adding and removing tags and tag categories found in the next generation vSphere web client
  • Configuration and reporting of EVC mode for vSphere clusters
  • Management of security policies for the vSS and its portgroups
  • New support for MS Windows PowerShell 4.0
  • Support for vSphere hosts configured for IPv6
  • Added migration priority support for vMotion (VMotionPriority parameter in conjunctionw ith the Move-VM cmdlet)
  • Get-Datastore cmdlet
    • RelatedObject paremeter extended to accept the Harddisk object
    • now allows filtering by cluster
  • Enhanced Get-Stat and Get-StatType cmdlets
  • Support added for e1000e vNICs
  • All values for DiskStorageFormat can be specified during VM cloning operations
  • 64-bit mode support for New-OSCustomizationSpec and Set-OSCustomizationSpec cmdlets
  • ToolsVersion property added to VMGuest which returns a string
  • Get-VirtualSwitch and Get-DVSwitch cmdlets support virtual port groups as a RelatedObject
  • Get-VM cmdlet enhanced to retrieve a list of VMs by virtual switch
  • Miscellaneous bug fixes

VMware vSphere PowerCLI 5.5 R2 supports vSphere 4.1 through vSphere 5.5 as well as Microsoft Windows PowerShell versions 2.0, 3.0, and new in R2 4.0.

Thank you Alan and thank you VMware!

VMTurbo’s Disruptive Software-Driven Control Expands Across Storage and Fabric To Realize Full Value of Virtualization

February 9th, 2014 by jason No comments »

Press Release

VMTurbo’s Disruptive Software-Driven Control Expands Across Storage and Fabric To Realize Full Value of Virtualization

VMTurbo Operations Manager Enables Customers to Realize 30%Improvement in Utilization while assuring application workload performance

BOSTON, MA – January 27, 2014 – VMTurbo, provider of the only Software-Driven Control for virtualized environments, today announced a new version of its flagship product, VMTurbo Operations Manager, enhanced with control modules for storage and fabric to drive virtualized environments to their desired state and maintain control in that state across the data center and IT stack.  These new solutions enable 30%  improvement in utilization while providing greater control over all aspects of the environment the application workload touches – from compute and storage to fabric and cloud. 

One of the major advancements in this release is management of the Converged Fabric layer with Cisco (CSCO) UCS support. Not only does VMTurbo provide unprecedented visibility into UCS from the fabric interconnect down to individual blades, it also enables control of UCS to manage real demand for UCS ports to maximize port utilization and avoid unnecessary port licensing costs.

“We’ve made a significant investment in UCS and are happy with it but it’s a challenge to manage,” said Jonathan Brown, Desktop Administrator at Beaufort Memorial Hospital (www.bmhsc.org). “VMTurbo is the only solution we’ve found on the market that helps us understand the inner workings of UCS so we can better manage it. We love VMTurbo, and are excited for all the new features to help us manage the future growth of our environment.”

VMTurbo is also disrupting enterprise software with its model of “easy to try, buy, deploy, and use”. Customers download VMTurbo Operations Manager and realize value instantly – unlike traditional management software which takes several months to install and perform after significant integration costs.  In fact, VMTurbo offers customers a free health check assessment of their virtual environments.  With VMTurbo, customers can break free from expensive, monitoring solutions that fail to eliminate reactive and labor-intensive IT fire fighting..  

“I learned more about my data center in 15 minutes with VMTurbo than I did in the last five years,” said Chuck Green, CIO of AlphaMaxx Healthcare, Inc., the premiere NCQA-accredited perinatal population health management firm.  “It’s truly a paradigm shift.”

90% of customers that have implemented VMTurbo’s Software-Driven control system to manage their virtualized data centers and cloud infrastructures report a return-on-investment of less than three months from purchase – an unparalleled breakthrough disrupting traditional enterprise management software (Source: TechValidate).

VMTurbo was recently recognized last week by Forbes as one of 2014 America’s Most Promising Companies.  Earlier in the year VMTurbo received the JP Morgan Hall of Innovation award, being named one of the most innovative technologies in the data center.

“VMTurbo’s technology is helping JPMorgan Chase optimize the utilization of virtual environments and thereby supporting a move from reactive to predictive workload management,” said George Sherman, Head of Compute Services at JPMorgan Chase. “Automation will enable our support teams to focus on higher value activity by preventing incidents and dynamically optimizing virtual environments.” 

VMTurbo Operations Manager

VMTurbo Operations Manager is the only product on the market understanding application workload performance, resource utilization and constraints in virtualized datacenter and cloud deployments to drive an organization’s environment to its desired state – that state of perpetual health where application performance is assured while maximizing efficiency – while providing control over all aspects of the environment the application workload touches, from compute and storage to fabric and cloud. While competitive solutions focus on viewing – monitoring systems to send alerts requiring operational staff to troubleshoot and remedy issues – VMTurbo Operations Manager ties the viewing with the doing, so IT Operations staff can elevate their focus from reactive to strategic. To try VMTurbo Operations Manager in your own environment, visit www.vmturbo.com/download or for a free health check assessment, call 1.877.978.8818 .

For more detailed information on VMTurbo Operations Manager, visit vmturbo.com/operations-manager.

VMTurbo Storage Control Module

VMTurbo’s Storage Control Module ensures applications get the storage performance they require to operate reliably while enabling efficient use of storage infrastructure – thus preventing unnecessary over provisioning.  This module helps users solve their pressing storage performance and cost challenges, maximize their existing storage investments and embrace the adoption of advanced features and packaging such as NetApp Clustered Data ONTAP (cluster mode) and FlexPod. For more detailed information on VMTurbo Storage Control Module, visit www.vmturbo.com/storage-resource-management.

VMTurbo Fabric Control Module

Modern compute platforms and blade servers have morphed to fabrics unifying compute, network, virtualization and storage access into a single integrated architecture.  Furthermore, fabrics like Cisco (CSCO) UCS form the foundation of a programmable infrastructure for today’s private clouds and virtualized datacenters, the backbone of converged infrastructure offerings from VCE vBlock and NetApp FlexPod. 

With the addition of this Fabric Control Module, VMTurbo’s software-driven control system ensures workloads get the compute and network resources they need to perform reliably while maximizing the utilization of underlying blades and ports. For more detailed information on VMTurbo Fabric Control Module, visit  www.vmturbo.com/ucs-management 

About VMTurbo

VMTurbo’s Software-Driven Control platform enables organizations to manage cloud and enterprise virtualization environments to maximize infrastructure investments while assuring application performance. VMTurbo’s patent-pending Economic Scheduling Engine dynamically adjusts configuration, resource allocation and workload placement to meet service levels and business goals, and is the only technology capable of closing the loop in IT operation by automating the decision-making process to maintain an environment in its desired state. The VMTurbo platform first launched in August 2010 and since that time more than 10,000 cloud service providers and enterprises worldwide have deployed the platform, including JP Morgan Chase, Colgate-Palmolive and Ingram Micro. Using VMTurbo, our customers ensure that applications get the resources they need to operate reliably, while utilizing their most valuable infrastructure and human resources most efficiently. For more information, visit www.vmturbo.com.

vCenter Server Appliance 5.5 root account locked out after password expiration

January 10th, 2014 by jason No comments »

Thanks to Chris Colotti, I learned of a new VMware KB article today which could potentially have wide spread impact, particularly in lab, development, or proof of concept environments.  The VMware KB article number is 2069041 and it is titled The vCenter Server Appliance 5.5 root account locked out after password expiration.

In summary, the root account of the vCenter Server Appliance version 5.5 becomes locked out 90 days after deployment or root account password change.  This behavior is by design which follows a security best practice of password rotation.  In this case, the required password rotation interval is 90 days after which the account will be forcefully locked out if not changed.

The KB article describes processes to prevent a forced lockout as well as unlocking a locked out root account.

Approximately 90 days have elapsed since the release of vSphere 5.5 and I imagine this issue will quickly begin surfacing in large numbers where the vCenter Server Appliance 5.5 has been deployed using system defaults.

vSphere Consulting Opportunity in Twin Cities

December 14th, 2013 by jason No comments »

If you know me well, you know the area I call home.  If you’re a local friend, acquaintance, or member any of the three Minnesota VMware User Groups, then I have an opportunity that has crossed my desk which you or someone you know may be interested in.

A local business here in the Twin Cities has purchased vSphere and EMC VNXe storage infrastructure and is looking for a Consulting Engineer to deploy the infrastructure per an existing design.

Details:

  • Install and configure VMware vSphere 5.1 on two hosts
  • Install and configure VMware vCenter
  • Install and configure VMware Update Manager
  • Configure vSphere networking
  • Configure EMC VNXe storage per final design.

It’s a great opportunity to help a locally owned business deploy a vSphere infrastructure and I would think this would be in the wheelhouse of 2,000+ people I’ve met while running the Minneapolis VMware User Group.  As much as I’d love to knock this out myself, I’m a Dell Storage employee and as such I’m removing myself as a candidate for the role.  The best way I can help is to get the word out into the community.

If you’re interested, email me with your contact information and I’ll get you connected to the Director.

Happy Holidays!

Storage Center 5.6 Released

November 25th, 2013 by jason No comments »

I don’t have the latest and greatest Dell Compellent SC8000 controllers or SC220 2.5″ drive enclosures in my home lab although I dream nightly about Santa unloading some on me this Christmas.  What I do have is an older Series 20 and I am thankful for that.  But having an older storage array doesn’t mean I cannot leverage some of the latest and greatest features and operating systems available for datacenters.

Storage Center 5.6 was released just a short time ago and it ushers in some feature and platform support currently built into Storage Center 6.x as well as a large number of bug fixes.  This is a big win for me and anyone with 32-bit system (Series 30 or below) needing these features because SCOS 6.x is 64-bit only for Series 40 and newer which today includes the SC8000.

So what are these new features in 5.6 and why am I so excited?  I’m glad you asked.  For this guy, and on top of the list, it’s full support of all VAAI primitives.  Storage Center 5.5 and older boasted support of the block zeroing primitive.  Space Reclamation was there as well although that primitve alone did not satisfy the other component of the thin provisioning primitive which was STUN.

Shown below a Storage Center 5.5 datastore where I lack Atomic Test and Set (aka Hardware Assisted Locking) and XCOPY.  I have block zeroing and Space Reclamation using the Free Space Recovery agent for vSphere guest VMs using physical RDMs. VAAI support status can be obtained in full using esxcli:

Snagit Capture

Or in part using the vSphere Client GUI:

Snagit Capture

After the Storage Center 5.6 upgrade, I’ve got additional VAAI primitive support where Clone in most cases is going to be the biggest one in terms of fabric and host efficiency and performance. Not shown is support for Thin Provisioning Stun but that has been added as well:

Snagit Capture

The vSphere Client GUI now reflects full VAAI support after the 5.6 upgrade:

Snagit Capture

What else? Added support for vSphere 5.5 as an operating system type:

Snagit Capture

Last but not least, added support for Windows 2012 and some of its features including Offloaded Data Transfer, Thin Provisioning, Space Reclamation, and Server Objects:

Snagit Capture

Storage Center 5.6 also adds new storage features which are storage host agnostic such as Background Media Scans (BMS) as well as improved disk and HBA management for server objects.  And the bug fixes I mentioned earlier – refer to the SCOS 5.6 Release Notes for details.

To wrap this up, if you’ve got an older Storage Center model and you want support for these new features while avoiding a forklift upgrade, Storage Center Operating System 5.6 is the way to go.