Posts Tagged ‘Security’

View 5.1 Upgrade Experience. Composer, Permissions, and SSL – Oh My!

August 8th, 2012

The other night I upgraded the VMware View 5.0.1 environment in the lab to 5.1 which was released on May 16th.  Normally when I upgrade the View environment, I don’t actually perform an inline upgrade of the Connection Server or database.  The environment is small enough that I can flatten it and rebuild fresh from scratch (including brand new VMs for the infrastructure components such as the Connection Server) for each new version VMware releases.  Due to VMware’s aggressive release schedule, I also embed the production version in the infrastructure server name which helps me keep track of where things are at in the lab.  Thus, with each new release, I’m building new infrastructure VMs with updated names, rather than recycling the previous infrastructure VMs, renaming them, remove/re-add to the domain, and even then I’m left with a VM name which doesn’t match the name on the datastore folder.  Pushing the reset button and starting fresh obliterates any bad DNA or cooties the previous environment might have had and it gives me a little extra peace of mind when I sleep at night.

I was running a little short on time so for this round I decided to perform an inline upgrade to 5.1 rather than going through the normal rebuild routine.  After all, most production environments don’t have the luxury of starting over so now was as good a time as ever to test the upgrade process of View in the lab.  Again – a fairly simple setup: a Connection Server, View Composer 2.7 installed on the vCenter Server which for the first time in many releases will be upgraded to 3.0, back end databases on an external SQL server, and 3 small pools.

The View Connection Server upgrade went as planned. No issues to speak of there (yet).  However, I did struggle with the View Composer upgrade.  The first run through uninstalled View Composer and failed with an error which I wasn’t quick enough to capture.  I re-ran through the Composer installation and it failed again with the same error:

The wizard was interrupted before VMware View Composer could be completely installed.

While I was perfrming some troubleshooting, a couple of gracious folks on Twitter by the name of Diego Quintana and Tim Washburn (@daquintana and @mittim12 respectively) pointed out VMware KB article 2017773 Installing or upgrading View Composer fails with error: The wizard was interrupted before VMware View Composer could be completely installed which resolved my issue.  The previous View Composer installation had placed one or more keys in the directory C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\ which my user account no longer had NTFS permissions to.  The resolution was to simply relax the NTFS permissions both on the MachineKeys folder as well as the files inside of the folder for good measure.

I thought I was out of the woods, but not quite yet.  SSL certificate issues followed.

Snagit Capture

VMware made some new changes with regards to SSL in View 5.1 which are documented at <View installation drive letter>:\Program Files\VMware\VMware View\Server\README.rtf

Copied and pasted verbatim, the release notes are:

Read These Notes!  Your View 5.1 Setup Will Be Easier!

You can read these notes in your language:
Français    Deutsch     简体中文     日本語     한국어

We made changes in View 5.1 that require you to configure View components a little differently than in the past.  These notes will help you to avoid potential pitfalls when you install or upgrade to View 5.1.

1)  You cannot downgrade View 5.1 Connection Server to previous versions.

In View 5.1, the View LDAP configuration is encrypted and cannot be used by earlier versions of View.

  • After you upgrade a View Connection Server instance to View 5.1, you cannot downgrade that instance to an earlier version.
  • After you upgrade all View Connection Server instances in a replicated group, you cannot add another instance that runs an earlier version of View.

Note: Downgrading was never supported, but in past releases it worked.  Now it won’t work.

2)  vCenter Server and View Composer hosts need valid SSL certificates.

  • Best choice: Ensure your vCenter Server and View Composer have Certificate Authority (CA)-provided certificates:

o  Install an SSL certificate, signed by a CA, on the Windows Server on which vCenter Server is installed.

o  Do the same for View Composer. If you install View Composer and vCenter Server on the same host, they can use the same certificate, but you must configure the certificate separately for each component.

* If you install the certificate before you install View Composer, you can select your certificate during the View Composer installation.

* If you replace the default certificate later, run the SviConfig ReplaceCertificate command to bind the new certificate to the port used by View Composer.

o  Make sure the CA for the new certificates, and any parent CAs, are trusted by each Windows server on which a View Connection Server instance is installed.

  • Alternative: After you add vCenter Server and View Composer to View, accept the thumbprint of the default certificate for View Composer by clicking Verify in View Administrator.  Do the same for vCenter Server.

More information: See “Configuring SSL Certificates for View Servers” in the View Installation guide.

3)  Security server and View Connection Server hosts need valid SSL certificates.

  • Best choice: After you install a View Connection Server instance or security server on a Windows Server host, open the Windows Server certificate store and take these steps:

o  Import an SSL certificate that is signed by a CA and that your clients can validate.

o  Make sure that the entire certificate chain, including intermediate certificates and root certificate, are installed.

o  Make sure the certificate has a private key, and mark the key as exportable.

o  Configure the certificate Friendly Name as vdm.

  • Alternative: Let the View server installer create a default certificate in the Windows Server certificate store. The certificate is self-signed and will be shown as invalid in View Administrator.
  • Upgrading to View 5.1: If your original View servers already have SSL certificates signed by a CA, you don’t have to do anything.  During the upgrade, View imports your certificates into the Windows Server certificate store.

If your original View servers have default certificates, upgrade your View servers and follow the Best choice steps shown above.

More information: See “Configuring SSL Certificates for View Servers” in the View Installation guide.

4)  Certificates for vCenter Server, View Composer, and View servers must include certificate revocation lists (CRLs).

View will not validate a certificate without a CRL.

  • Best Choice: lf needed, take these steps:

o  Add a CRL to your certificate.

o  Import the updated certificate into the Windows certificate store on the vCenter Server, View Composer, and View server host.

  • Alternative: Change the registry settings that control CRL checking.

More information: “Configuring Certificate Revocation Checking on Server Certificates” in the View Installation guide.

5)  Windows Firewall with Advanced Security must be enabled on Security Server and View Connection Server hosts. 

By default, IPsec rules govern connections between the View security server and View Connection Server and require Windows Firewall with Advanced Security to be enabled.

  • Best choice: Set Windows Firewall with Advanced Security to on before you install the View servers. Make sure it’s on for any active profiles; better still, set it to on for all profiles.
  • Alternative: Before you install security servers, open View Administrator and disable the Global Setting, Use IPsec for Security Server Connections, by setting it to no. (This is not recommended.)

6)  Back-end firewalls must be set up to support IPsec.

If you have a back-end firewall between security servers and View Connection Server instances, you must configure firewall rules to allow the connections to work.

More information: See “Configuring a Back-End Firewall to Support IPsec ” in the View Installation guide.

7)  View Clients must use HTTPS to connect to View.

View Connection Server instances and security servers use SSL for client connections.

  • If View clients connect via an SSL off-loading intermediate device, you must install the intermediate device’s SSL certificate on View Connection Server or security server.
  • The connection must be HTTPS whether or not a View client connects via an intermediate device such as a load balancer. If you use an intermediate device, and you want the connection between the intermediate device and View server to be over HTTP (SSL off-loading), configure the locked.properties file on the View server.
  • Older View clients that can choose not to use HTTPS will get an error if users select HTTP. Previously they were silently redirected to HTTPS. Clients that cannot make SSL connections will be unable to connect to View.

More information: See “Off-loading SSL Connections to Intermediate Servers” in the View Administration guide.

8)  Encrypted and cleansed View backups require new restore steps.

By default, View 5.1 backups are encrypted. You can also cleanse View backups (exclude passwords and other sensitive information from the backup data) or back up in plain text (not recommended).

  • To restore an encrypted backup, you must decrypt the data first. You must use the data recovery password that you provided when you installed View Connection Server.
  • Do not restore cleansed backups. Data such as passwords will be missing from your View LDAP configuration. View components will not function properly without this data. To restore normal functionality, you will have to use View Administrator to manually reset all passwords and other missing data items.

More information: See “Backing Up and Restoring View Configuration Data” in the View Administration guide.

9)  Before you can upgrade or reinstall a View 5.1 security server, you must remove the relevant IPsec rules from the paired View Connection Server instance so that fresh rules can be established.

  • In View Administrator, select the security server and click More Commands > Prepare for Upgrade or Reinstallation.

Note: You don’t need to remove a security server from View before you upgrade or reinstall the server.

More information: See “Prepare to Upgrade or Reinstall a Security Server” in the View Installation guide.

Ok, so basically VMware is pushing for the use of SSL certificates from a trusted CA whether that be externally (VeriSign, etc.) or internally (Microsoft Certificate Services) generated.  For the time being, I have ditched my internal Microsoft CA and wish to continue using the self signed certificates shipped and installed by View.  To do so, as explained in the README above, one must visit the System Health in the View Administrator Dashboard and verify the certificates for the vCenter Server as well as the View Composer Server (each will be seen in a red status in the dashboard).  The Connection Server certificate cannot be verified and will remain in a red status however from this point forward both the View Connection Server and View Composer will function normally.

Upgrading the View Agents and recomposing the pools was a non-issue and the upgrade was completed successfully.  After all is said and done, the environment is working and the upgrade was successful.  View 5.0.1 Clients have no problem connecting to the new 5.1 environment; I’ll get the clients upgraded in the near future and I’ll consider resurrecting the lab CA to generate trusted SSL certificates.

Father’s Day Fun With ESXi Annotations

June 19th, 2011

Tired of the same old DCUI look of your ESXi host?

SnagIt Capture

Change it with your own custom Annotation (found under Host|Configuration|Software|Advanced Settings):

SnagIt Capture

Viola!

SnagIt Capture

Clearly this is more fun than any one person should be allowed to have.

Clear the Annotation field to restore the original ESXi look.

Make an ESX Firewall Rule Manageable in the vSphere Client

June 25th, 2010

Make an ESX Firewall Rule Manageable in the vSphere Client.  To do so, you essentially need to create a new service in the firewall configuration XML file.

Open the file /etc/vmware/firewall/services.xml
Scroll to the bottom & note the last Service ID #
Copy an existing service section as a template (ie. faultTolerance)
Paste as new following proper XML formatting
Increment the Service ID # by 1 ensuring it’s unique
Customize to fit your new inbound/outbound port rule
Save and exit
Services do not need to be restarted

As an example, I took :

<service id=’0031′>
    <id>faultTolerance</id>
    <rule id=’0000′>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <port type=’dst’>80</port>
    </rule>
  </service>

and created a new service like so:

<service id=’0033′>
    <id>CoolFirewallRule</id>
    <rule id=’0000′>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <port type=’dst’>12345</port>
    </rule>
  </service>

The result is a firewall rule named CoolFirewallRule which can be toggled via the vSphere Client:

 6-22-2010 11-13-39 PM

Two Quick Announcements

April 19th, 2010

I’ve got several blog posts in the queue but unfortunately I haven’t had time to get them cranked out yet.  In the interim, here are a couple of hot items I wanted to help spread the word on.

There’s a 45 minute VMware certification podcast coming up early this Wednesday morning.  APAC Vitalization Roundtable (Certifications & the VCDX path).  Podcast guests are Andrew Mitchell, Duncan Epping, and Alastair Cooke.  The Talkshoe podcast is scheduled for the following time:
EDT (USA) – 7AM
PDT (USA) – 4AM
Perth (Australia) – 7PM
Hong Kong (Hong Kong) – 7PM
Kuala Lumpur (Malaysia) – 7PM
Tokio (Japan) – 8PM
Auckland (New Zealand) – 11PM
London (UK) – 12Noon
 

The vSphere 4.0 Hardening Guide, a collaborative effort which has been under development for several months with a few interim beta distributions, has been finalized and released.  The document is presented in a flexible format in that a few different levels of hardening can be used as low to high risk boundaries.  The user then ultimately decides on the appropriate hardening level depending on the level of risk the user is comfortable with assuming.The guide covers several key VMware Virtual Infrastructure areas.  To quote VMware’s announcement: 

Overall, there are more than 100 guidelines, with the following major sections: 

  • Introduction
  • Virtual Machines
  • Host (both ESXi and ESX)
  • vNetwork
  • vCenter
  • Console OS (for ESX only)

If you’ve recently deployed vSphere or if you are about to, check out these hardening guides to help secure your datacenter.  You can’t beat the price – free!

New ESX(i) 3.5 security patch released; scenarios and installation notes

April 11th, 2009

On Friday April 10th, VMware released two patches:

Both address the same issue:

A critical vulnerability in the virtual machine display function might allow a guest operating system to run code on the host. The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2009-1244 to this issue.

Hackers must love vulnerabilities like this because they can get a lot of mileage out of essentially a single attack. The ability to execute code on an ESX host can impact all running VMs on that host.

Although proper virtualization promises isolation, the reality is that no hardware or software vendor is perfect and from time to time we’re going to see issues like this. Products are under constant attack from hackers (both good and bad) to find exploits. In virtualized environments, it’s important to remember that guest VMs and guest operating systems are no different than their physical counterparts in that they need to be properly protected from the network. That means adequate virus protection, spyware protection, firewalls, encryption, packet filtering, etc.

This vulnerability in VMware ESX and ESXi is really a two factor attack. In order to compromise the ESX or ESXi host, the guest VM must first be vulnerable to compromise on the network to provide the entry point to the host. Once the guest VM is compromised, the next step is to get from the guest VM to the ESX(i) host. Hosts without the patch will be vulnerable to the next attack which we know from reading above will allow who knows what code to be executed on the host. If the host is patched, we maintain our guest isolation and the attack stops at the VM level. Unfortunately, the OS running in the guest VM is still compromised, again highlighting the need for adequate protection of the operating system and applications running in each VM.

The bottom line is this is an important update for your infrastructure. If your ESX or ESXi hosts are vulnerable, you’ll want to get this one tested and implemented as soon as possible.

I installed the updates today in the lab and discovered something interesting that is actually outlined in both of the KB articles above:

  • The ESXi version of the update requires a reboot. Using Update Manager, the patch process goes like this: Remediate -> Maintenance Mode -> VMotion VMs off -> Patch -> Reboot -> Exit Maintenance Mode. The duration of installation of the patch until exiting maintenance mode (including the reboot in between) took 12 minutes.
  • The ESX version of the update does not require a reboot. Using Update Manager, the patch process goes like this: Remediate -> Maintenance Mode -> VMotion VMs off -> Patch -> Exit Maintenance Mode. The duration of installation of the patch until exiting maintenance mode (with no reboot in between) took 1.5 minutes.

Given reboot times of the host, patching ESX hosts goes much quicker than patching ESXi hosts. Reboot times on HP Proliant servers aren’t too bad but I’ve been working with some powerful IBM servers lately and the reboot times on those are significantly longer than HP. Hopefully we’re not rebooting ESX hosts on a regular basis so with that in mind, reboot times aren’t a huge concern, but if you’ve got a large environment with a lot of hosts requiring reboots, the reboot times are going to be cumulative in most cases. Consider my environment above. A 6 node ESXi cluster is going to take 72 minutes to patch, not including VMotions. A 6 node ESX cluster is going to take 9 minutes to patch, not including VMotions. This may be something to really think about when weighing the decision of ESX versus ESXi for your environment.

Update: One more item critical to note is that although the ESX version of the patch requires no reboot, the patch does require three other patches to be installed, at least one of which requires a reboot.  If you already meet the requirements, no reboot will be required for ESX to install the new patch.

In closing, while we are on the subject of performing a lot of VMotions, take a look at a guest blog post from Simon Long called VMotion Performance. Simon shows us how to modify VirtualCenter (vCenter Server) to allow more simultaneous VMotions which will significantly cut down the amount of time spent patching ESX hosts in a cluster.

Three VirtualCenter security tips Windows administrators should know

January 15th, 2009

Good morning!  I’d like to take the opportunity to talk a bit about something that has been somewhat of a rock in my shoe as a seasoned Windows administrator from the NT 3.5 era:  The VirtualCenter (vCenter Server, VirtualCenter Management Server, VCMS, VC, etc.) security model, or more accurately, its unfamiliar mechanics that can catch Windows administrators off guard and leave them scratching their heads.

Tip #1: The VCMS security model revolves around privileges, roles, and objects.  The more than 100 privileges define rights, roles are a collection of privileges, and roles are assigned to objects which are entities in the virtual infrastructure as shown in the diagram borrowed below:

1-15-2009 11-24-45 AM

Windows administrators will be used to the concept of assigning NTFS permissions to files, folders, and other objects in Active Directory.  It is very common for Windows objects to contain more than one Access Control Entry (ACE) which can be a group (such as “Accounting”, “Marketing”, etc.) or an explicit user (such as “Bob”, Sally”, etc.)  The same holds true for assigning roles to object in VC.

In some instances, which are not uncommon at all, a user may be granted permission to an object by way of more than one ACE.  For example, if both the Accounting and Marketing groups were assigned rights, and Sally was a member of both those groups, Sally would have rights to the object through both of those groups.  Using this same example, if the two ACEs defined different permissions to an object, the end result is a cumulative, so long as the ACE doesn’t contain “deny” which is special:  Sally would have the combined set of permissions.  The same holds true in VC.

Let’s take the above example a step further.  In addition to the two groups, which Sally is a member of, being ACLd to an object, now let’s say Sally’s user account object itself is an explicit ACE in the ACL list.  In the Windows world, the effect is Sally’s rights are still cumulative combining the three ACEs.  This is where the fork in the road lies in the VirtualCenter security model.  Roles explicitly assigned to a user object trump all other assigned or inherited permissions to the same object.  If the explicit ACE defines less permissions, the effective result is Sally will have less permissions than what her group membership would have provided.  If the explicit ACE defines more permissions, the effective result is Sally will have more permissions than what her group membership would have provided.  This is where Windows based VC administrators will be dumbfounded when a user suddenly calls with tales of things gray’d out in VirtualCenter, not enough permissions, etc.  Of course the flip side of the coin is a junior administrator suddenly finds themselves with cool new options in VC.  “Let’s see what this datastore button does”

Moral of the story from a real world perspective:  Assigning explicit permissions to user accounts in VC without careful planning will yield somewhat unpredictable results when inheritance is enabled (which is typical).  To take this to extremes, assigning explicit permissions to user accounts in VC, especially where inheritance in the VC hierarchy is involved, is a security and uptime risk when a user ends up with the wrong permissions accidentally.  For security and consistency purposes, I would avoid assigning permissions explicitly to user accounts unless you have a very clear understanding of the impacts currently and down the road.

Tip #2: Beware the use of the built in role Virtual Machine Administrator.  It’s name is misleading and the permissions it has are downright scary and not much different than the built in Administrator role.  For instance, the Virtual Machine Administrator role:  can modify VC and ESX host licensing, has complete control over the VC folder structure, has complete control over Datacenter objects, has complete control over datastores (short of file management), can remove networks, has complete control over inventory items such as hosts and clusters.  This list goes on and on.  I have three words:  What The Hell?!  I don’t know – the way my brain works is those permissions stretch well beyond the boundaries of what I would delegate for a Virtual Machine Administrator.

Moral of the story from a real world perspective:  Use the Virtual Machine Administrator role with extreme caution.  There is little disparity between the Administrator role and the Virtual Machine Administrator role, minus some items for Update Manager and changing VC permissions themselves. Therefore, any user who has the Virtual Machine Administrator role is practically an administrator.  The Virtual Machine Administrator role should not be used unless you have delegations that would fit this role precisely.  Another option would be clone the role and strip some of the more datacenter impactful permissions out of it.

Tip #3: Audit your effective VirtualCenter permissions on a regular basis, especially if you have large implementation with many administrators “having their hands in the cookie jar” so to speak.  If you use groups to assign roles in VC, then that means you should be auditing these groups as well (above and beyond virtualization conversations, administrative level groups should be audited anyway as a best practice).  This whitepaper has a nice Perl script for dumping VirtualCenter roles and permissions using the VMware Infrastructure Perl Toolkit.  Use of the script will automate the auditing process quite a bit and help transform a lengthy mundane task into a quicker one.  While you’re at it, it wouldn’t be a bad idea to periodically check tasks and events to see who is doing what.  There should be no surprises there.

Moral of the story from a real world perspective:  Audit your VirtualCenter roles and permissions.  When an unexpected datacenter disaster occurs from users having elevated privileges, one of the first questions to be asked in the post mortem meeting will be what your audit process is.  Have a good answer prepared.  Even better, avoid the disaster and down time through the due diligence of auditing your virtual infrastructure security.

For more information about VirtualCenter security, check out this great white paper or download the .pdf version from this link.  Some of the information I posted above I gathered from this document.  The white paper was written by Charu Chaubal, a technical marketing manager at VMware and Ph.D. in numerical modeling of complex fluids, with contributions from Doug Clark, and Karl Rummelhart.

If VirtualCenter security talk really gets your juices flowing, you should check out a new podcast launched by well known and respected VMTN community member/moderator and book author Edward Haletky that starts today called Virtualization Security Round Table.  It is sure to be good!

VMware VI network communications and port usage diagram

November 6th, 2008

Nigel Metheringham has taken the information I posted here and updated it with information from page 179 of the VMware ESX Server 3 Configuration Guide

The result is a current diagram of VMware Virtual Infrastructure network communications and port usage which applies to both ESX and ESXi.  Nigel sent me the updated document via email so that I can update the information on the VMware Communities, however, they are currently unavailable due to planned maintenance so in the mean time I’m making the document available here.  Thank you Nigel!

Download from the link below:

vmware_network_ports