Taking a look at my VM Storage Policies compliance in the vSphere Client, I was alerted to a situation that none of my configured virtual machines were compliant with their assigned VM Storage Policy named “Five Nines Compellent Storage”. Oddly enough, the virtual machine home directories and virtual disks were in fact on the correct datastores and showed as compliant a few days earlier. None had been migrated via Storage vMotion or SDRS.
Now you see it, now you don’t
I then verified my VASA configuration by looking at the status of my registered storage provider. The issue was not so much that the provider was malfunctioning, but rather it was missing completely from the registered storage providers list. This indeed explains the resulting Not Compliant status of my virtual machines.
I checked another upgraded environment where I know I had a registered VASA storage provider. It reflected the same symptom and confirmed my suspicion that the recent process of upgrading to vCenter Server 5.5 appliance to Update 1 (via the web repository method) may have unregistered the storage provider once the reboot of the appliance was complete.
I had one more similar environment remaining which I had not upgraded yet. I verified the storage provider was registered and functioning prior to the Update 1 upgrade. I proceeded with the upgrade and after the reboot completed the storage provider was no longer registered.
What remains a mystery at this point is the root cause of the unregistered storage provider. I was unable to find any VMware KB articles related to this issue.
Not the end of the world
The workaround is straightforward: re-register each of the missing storage providers. For Dell Compellent customers, the storage provider points to the CITV (Compellent Integration Tools for VMware) appliance and the URL is follows the format:
Dell Compellent customers should also keep the following in mind for VASA integration:
- the integration requires the CITV appliance and Enterprise Manager 6.1 and above.
- the out of box Windows Server Firewall configuration which Enterprise Manager sits on will block the initial VASA configuration in the CITV appliance. TCP 3033 incoming must be allowed or alternatively disable the Windows Firewall (not highly recommended).
Once the applicable storage provider(s) are added back, no additional VM Storage Policy reconfiguration is required other than to check for compliance. All VMs should fall back into compliance.
Once again, I am unsure at this point as to why applying vCenter 5.5 Update 1 to the appliance caused the registered storage providers to go missing or what that connection is. I will also add that I deployed additional vCenter 5.5 appliances under vCloud Director with a default configuration, no registered vSphere hosts, registered a VASA storage provider, upgraded to Update 1, rebooted, and the storage provider remained. I’m not sure what element in these subsequent tests caused the outcome to change but the problem itself now presents itself as inconsistent. If I do see it again and find a root cause, as per usual I will be sure to update this article. To reiterate, Update 1 was applied in this case via the web repository method. There are a few other methods available to apply Update 1 to the vCenter Server appliance and of course there is also the Windows version of vCenter Server – it is unknown by me if these other methods and versions are impacted the same way.
Looks like someone has a case of the Mondays
On a somewhat related note, during lab testing I did find that VM Storage Profiles configured via the legacy vSphere Client do not show up as configured VM Storage Policies in the next gen vSphere Web Client. Likewise, VM Storage Policies created in the next gen vSphere Web Client are missing in the legacy vSphere Client. However, registered storage providers themselves carry over from one client to the other – no issue there. I guess the lesson here is to stick with a consistent method of creating, applying, and monitoring Profile-Driven Storage in your vSphere environment from a vSphere Client perspective. As of the release of vSphere 5.5 going forward, that should be the next gen vSphere Web Client. However, this client still seems to lack the ability to identify VASA provided storage capabilities on any given datastore although the entire list of possible capability strings is available by diving into VM Storage Policy configuration.
Last but not least, VMware KB 2004098 vSphere Storage APIs – Storage Awareness FAQ provides useful bits of information about the VASA side of vSphere storage APIs. One item in that FAQ that I’ve always felt was worded a bit ambiguously in the context of vSphere consolidation is:
The Vendor Provider cannot run on the same host as the vCenter Server.
In most cases, the vCenter Server as well as the VASA integration component(s) will run as virtual machines. Worded above as is, it would seem the vCenter Server (whether that be Windows or appliance based) cannot reside on the same vSphere host as the VASA integration VM(s). That’s not at all what that statement implies and moreover it wouldn’t make much sense. What it’s talking about is the use case of a Windows based vCenter Server. In this case, Windows based VASA integration components must not be installed on the same Windows server being used to host vCenter Server. For Dell Compellent customers, the VASA integration comes by way of the CITV appliance which runs atop a Linux platform. However, the CITV appliances does communicate with the Windows based Enterprise Manager Data Collector for VASA integration. Technically, EM isn’t the provider, the CITV appliance is. Personally I’d keep the EM and vCenter Server installations separate. Both appreciate larger amounts of CPU and memory in larger environments and for the sake of performance, we don’t want these two fighting for resources during times of contention.