Monster VMs & ESX(i) Heap Size: Trouble In Storage Paradise

September 12th, 2012 by jason 20 comments »

While running Microsoft Exchange Server Jetstress on vSphere 5 VMs in the lab, tests were failing about mid way through initializing its several TBs of databases.  This was a real head scratcher.  Symptoms were unwritable storage or lack of storage capacity.  Troubleshooting yielding errors such as “Cannot allocate memory”.  After some tail chasing, the road eventually lead to VMware KB article 1004424: An ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage.

As it turns out, ESX(i) versions 3 through 5 have a statically defined per-host heap size:

  • 16MB for ESX(i) 3.x through 4.0: Allows a max of 4TB open virtual disk capacity (again, per host)
  • 80MB for ESX(i) 4.1 and 5.x: Allows a max of 8TB open virtual disk capacity (per host)

This issue isn’t specific to Jetstress, Exchange, Microsoft, or a specific fabric type, storage protocol or storage vendor.  Exceeding the virtual disk capacities listed above, per host, results in the symptoms discussed earlier and memory allocation errors.  In fact, if you take a look at the KB article, there’s quite a laundry list of possible symptoms depending on what task is being attempted:

  • An ESXi/ESX 3.5/4.0 host has more that 4 terabytes (TB) of virtual disks (.vmdk files) open.
  • After virtual machines are migrated by vSphere HA from one host to another due to a host failover, the virtual machines fail to power on with the error:vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.
  • You see warnings in /var/log/messages or /var/log/vmkernel.logsimilar to:vmkernel: cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
    vmkernel: cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)
    cpu15:11905)WARNING: Heap: 2525: Heap cow already at its maximum size. Cannot expand.
    cpu15:11905)WARNING: Heap: 2900: Heap_Align(cow, 6160/6160 bytes, 8 align) failed. caller: 0x41802fd54443
    cpu4:1959755)WARNING:Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
    cpu4:1959755)WARNING: Heap: 2900: Heap_Align(vmfs3, 2099200/2099200 bytes, 8 align) failed. caller: 0x418009533c50
    cpu7:5134)Config: 346: “SIOControlFlag2″ = 0, Old Value: 1, (Status: 0x0)
  • Adding a VMDK to a virtual machine running on an ESXi/ESX host where heap VMFS-3 is maxed out fails.
  • When you try to manually power on a migrated virtual machine, you may see the error:The VM failed to resume on the destination during early power on.
    Reason: 0 (Cannot allocate memory).
    Cannot open the disk ‘<>’ or one of the snapshot disks it depends on.
  • The virtual machine fails to power on and you see an error in the vSphere client:An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)
  • A similar error may appear if you try to migrate or Storage vMotion a virtual machine to a destination ESXi/ESX host on which heap VMFS-3 is maxed out.
  • Cloning a virtual machine using the vmkfstools -icommand fails and you see the error:Clone: 43% done. Failed to clone disk: Cannot allocate memory (786441)
  • In the /var/log/vmfs/volumes/DatastoreName/VirtualMachineName/vmware.log file, you may see error messages similar to:2012-05-02T23:24:07.900Z| vmx| FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory
    2012-05-02T23:24:07.900Z| vmx| AIOGNRC: Failed to open ‘/vmfs/volumes/xxxx-flat.vmdk’ : Cannot allocate memory (c00000002) (0x2013).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-VMFS : “/vmfs/volumes/xxxx-flat.vmdk” : failed to open (Cannot allocate memory): AIOMgr_Open failed. Type 3
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-LINK : “/vmfs/volumes/xxxx.vmdk” : failed to open (Cannot allocate memory).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-CHAIN : “/vmfs/volumes/xxxx.vmdk” : failed to open (Cannot allocate memory).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-LIB : Failed to open ‘/vmfs/volumes/xxxx.vmdk’ with flags 0xa Cannot allocate memory (786441).
    2012-05-02T23:24:07.900Z| vmx| DISK: Cannot open disk “/vmfs/volumes/xxxx.vmdk”: Cannot allocate memory (786441).
    2012-05-02T23:24:07.900Z| vmx| Msg_Post: Error
    2012-05-02T23:24:07.900Z| vmx| [msg.disk.noBackEnd] Cannot open the disk ‘/vmfs/volumes/xxxx.vmdk’ or one of the snapshot disks it depends on.
    2012-05-02T23:24:07.900Z| vmx| [msg.disk.configureDiskError] Reason: Cannot allocate memory.

While VMware continues to raise the scale and performance bar for it’s vCloud Suite, this virtual disk and heap size limitation becomes a limiting constraint for monster VMs or vApps.  Fortunately, there’s a fairly painless resolution (at least up until a certain point):  Increase the Heap Size beyond its default value on each host in the cluster and reboot each host.  The advanced host setting to configure is VMFS3.MaxHeapSizeMB.

Let’s take another look at the default heap size and with the addition of its maximum allowable heap size value:

  • ESX(i) 3.x through 4.0:
    • Default value: 16MB – Allows a max of 4TB open virtual disk capacity
    • Maximum value: 128MB – Allows a max of 32TB open virtual disk capacity per host
  • ESX(i) 4.1 and 5.x:
    • Default value: 80MB – Allows a max of 8TB open virtual disk capacity
    • Maximum value: 256MB – Allows a max of 25TB open virtual disk capacity per host

After increasing the heap size and performing a reboot, the ESX(i) kernel will consume additional memory overhead equal to the amount of heap size increase in MB.  For example, on vSphere 5, the increase of heap size from 80MB to 256MB will consume an extra 176MB of base memory which cannot be shared with virtual machines or other processes running on the host.

Readers may have also noticed an overall decrease in the amount of open virtual disk capacity per host supported in newer generations of vSphere.  While I’m not overly concerned at the moment, I’d bet someone out there has a corner case requiring greater than 25TB or even 32TB of powered on virtual disk per host.  With two of VMware’s core value propositions being innovation and scalability, I would tip-toe lightly around the phrase “corner case” – it shouldn’t be used as an excuse for its gaps while VMware pushes for 100% data virtualization and vCloud adoption.  Short term, the answer may be RDMs. Longer term: vVOLS.

Updated 9/14/12: There are some questions in the comments section about what types of stoarge the heap size constraint applies to.  VMware has confirmed that heap size and max virtual disk capacity per host applies to VMFS only. The heap size constraint does not apply to RDMs nor does it apply to NFS datastores.

Updated 4/4/13: VMware has released patch ESXi500-201303401-BG to address heap issues.  This patch makes improvements to both default and maximum limits of open VMDK files per vSphere host.  After applying the above patch to each host, the default heap size for VMFS-5 datastores becomes 640MB which supports 60TB of open VMDK files per host.  These new default configurations are also the maximum values as well.  For additional reading on other fine blogs, see A Small Adjustment and a New VMware Fix will Prevent Heaps of Issues on vSphere VMFS Heap and The Case for Larger Than 2TB Virtual Disks and The Gotcha with VMFS.

Updated 4/30/13: VMware has released vSphere 5.1 Update 1 and as Cormac has pointed out here, heap issue resolution has been baked into this release as follows:

  1. VMFS heap can grow up to a maximum of 640MB compared to 256MB in earlier release. This is identical to the way that VMFS heap size can grow up to 640MB in a recent patch release (patch 5) for vSphere 5.0. See this earlier post.
  2. Maximum heap size for VMFS in vSphere 5.1U1 is set to 640MB by default for new installations. For upgrades, it may retain the values set before upgrade. In such cases, please set the values manually.
  3. There is also a new heap configuration “VMFS3.MinHeapSizeMB” which allows administrators to reserve the memory required for the VMFS heap during boot time. Note that “VMFS3.MinHeapSizeMB” cannot be set more than 255MB, but if additional heap is required it can grow up to 640MB. It alleviates the heap consumption issue seen in previous versions, allowing the ~ 60TB of open storage on VMFS-5 volumes per host to be accessed.

When reached for comment, Monster VM was quoted as saying “I’m happy about these changes and look forward to a larger population of Monster VMs like myself.”

photo

VMworld 2012 Announcements – Part I

August 27th, 2012 by jason 7 comments »

VMworld 2012 is underway in San Francisco.  Once again, a record number of attendees is expected to gather at the Moscone Center to see what VMware and their partners are announcing.  From a VMware perspective, there is plenty.

Given the sheer quantity of announcements, I’m actually going to break up them up into a few parts, this post being Part I.  Let’s start with the release of vSphere 5.1 and some of its notable features.

Enhanced vMotion – the ability to now perform a vMotion as well as a Storage vMotion simultaneously. In addition, this becomes an enabler to perform vMotion without the shared storage requirement.  Enhanced vMotion means we are able to migrate a virtual machine stored on local host storage, to shared storage, and then to local storage again.  Or perhaps migrate virtual machines from one host to another with each having their own locally attached storage only.  Updated 9/5/12 The phrase “Enhanced vMotion” should be correctly read as “vMotion that has been enhanced”.  “Enhanced vMotion” is not an actual feature, product, or separate license.  It is an improvement over the previous vMotion technology and included wherever vMotion is bundled.

Snagit Capture

Enhanced vMotion Requirements:

  • Hosts must be managed by same vCenter Server
  • Hosts must be part of same Datacenter
  • Hosts must be on the same layer-2 network (and same switch if VDS is used)

Operational Considerations:

  • Enhanced vMotion is a manual process
  • DRS and SDRS automation do not leverage enhanced vMotion
  • Max of two (2) concurrent Enhanced vMotions per host
  • Enhanced vMotions count against concurrent limitations for both vMotion and Storage vMotion
  • Enhanced vMotion will leverage multi-NIC when available

Next Generation vSphere Client a.k.a. vSphere Web Client – An enhanced version of the vSphere Web Client which has already been available in vSphere 5.0.  As of vSphere 5.1, the vSphere Web Client becomes the defacto standard client for managing the vSphere virtualized datacenter.  Going forward, single sign-on infrastructure management will converge into a unified interface which any administrator can appreciate.  vSphere 5.1 will be the last platform to include the legacy vSphere client. Although you may use this client day to day while gradually easing into the Web Client, understand that all future development from VMware and its partners now go into the Web Client. Plug-ins currently used today will generally still function with the legacy client (with support from their respective vendors) but they’ll need to be completely re-written vCenter Server side for the Web Client.  Aside from the unified interface, the architecture of the Web Client has scaling advantages as well.  As VMware adds bolt-on application functionality to the client, VMware partners will now have the ability to to bring their own custom objects objects into the Web Client thereby extending that single pane of glass management to other integrations in the ecosystem.

Here is a look at that vSphere Web Client architecture:

Snagit Capture

Requirements:

  • Internet Explorer / FireFox / Chrome
  • others (Safari, etc.) are possible, but will lack VM console access

A look at the vSphere Web Client interface and its key management areas:

Snagit Capture

Where the legacy vSphere Client fall short and now the vSphere Web Client solves these issues:

  • Single Platform Support (Windows)
    • vSphere Web Client is Platform Agnostic
  • Scalability Limits
    • Built to handle thousands of objects
  • White Screen of Death
    • Performance
  • Inconsistent look and feel across VMware solutions
    • Extensibility
  • Workflow Lock
    • Pause current task and continue later right where you left off (this one is cool!)
    • Browser Behavior
  • Upgrades
    • Upgrade a Single serverside component

 vCloud Director 5.1

In the recent past, VMware aligned common application and platform releases to ease issues that commonly occurred with compatibility.  vCloud Director, the cornerstone of the vCloud Suite, is obviously the cornerstone in how VMware will deliver infrastructure, applications, and *aaS now and into the future. So what’s new in vCloud Director 5.1?  First an overview of the vCloud Suite:

Snagit Capture

And a detailed list of new features:

  • Elastic Virtual Datacenters – Provider vDCs can span clusters leveraging VXLAN allowing the distribution and mobility of vApps across infrastructure and the growing the vCloud Virtual Datacenter
  • vCloud Networking & Security VXLAN
  • Profile-Driven Storage integration with user and storage provided capabilities
  • Storage DRS (SDRS) integration
    • Exposes storage Pod as first class storage container (just like a datastores) making it visible in all workflows where a datastore is visible
    • Creation, modification, and deletion of spods not possible in vCD
    • Member datastore operations not permissible in VCD
  • Single level Snapshot & Revert support for vApps (create/revert/remove); integration with Chargeback
  • Integrated vShield Edge Gateway
  • Integrated vShield Edge Configuration
  • vCenter Single Sign-On (SSO)
  • New Features in Networking
    • Integrated Organization vDC Creation Workflow
    • Creates compute, storage, and networking objects in a single workflow
    • The Edge Gateway are exposed at Organization vDC level
    • Organization vDC networks replace Organization networks
    • Edge Gateways now support:
      • Multiple Interfaces on a Edge Gateway
      • The ability to sub-allocate IP pools to a Edge Gatewa
      • Load balancing
      • HA (not the same as vSphere HA)
        • Two edge VMs deployed in Active-Passive mode
        • Enabled at time of gateway creation
        • Can also be changed after the gateway has been completed
        • Gets deployed with first Organizational network created that uses this gateway
      • DNS Relay
        • Provides a user selectable checkbox to enable
        • If DNS servers are defined for the selected external network, DNS requests will be sent to the specified server. If not, then DNS requests will be sent to the default gateway of the external network.
      • Rate limiting on external interface
    • Organization networks replaced by Organization vDC Networks
      • Organization vDC Networks are associated with an Organization vDC
      • The network pool associated with Organization vDC is used to create routed and isolated Organization vDC networks
      • Can be shared across Organization vDCs in an Organization
    • Edge Gateways
      • Are associated with an Organization vDC, can not be shared across Organization vDCs
      • Can be connected to multiple external networks
        • Multiple routed Organization vDC networks will be connected to the same Edge Gateway
      • External network connectivity for the Organization vDC Network can be changed after creation by changing the external networks which the edge gateway is connected.
      • Allows IP pool of external networks to be sub-allocated to the Edge Gateway
        • Needs to be specified in case of NAT and Load Balancer
    • New Features in Gateway Services
      • Load balancer service on Edge Gateways
      • Ability to add multiple subnets to VPN tunnels
      • Ability to add multiple DHCP IP pools
      • Ability to add explicit SNAT and DNAT rules providing user with full control over address translation
      • IP range support in Firewall and NAT services
      • Service Configuration Changes
        • Services are configured on Edge Gateway instead of at the network level
        • DHCP can be configured on Isolated Organization vDC networks.
  • Usability Features
    • New default branding style
      • Cannot revert back to the Charcoal color scheme
      • Custom CSS files will require modification
    • Improved “Add vApp from Catalog” wizard workflow
    • Easy access to VM Quota and Lease Expirations
    • New dropdown menu that includes details and search
    • Redesigned catalog navigation and sub-entity hierarchy
    • Enhanced help and documentation links
  • Virtual Hardware Version 9
    • Supports features presented by HW9 (like 64 CPU support)
    • Supports Hardware Virtualization Calls
    • VT-x/EPT or AMD-V/RVI
    • Memory overhead increased, vMotion limited to like hardware
    • Enable/Disable exposed to users who have rights to create a vApp Template
  • Additional Guest OS Support
    • Windows 8
    • Mac OS 10.5, 10.6 and 10.7
  • Storage Independent of VM Feature
    • Added support for Independent Disks
    • Provides REST API support for actions on Independent Disks
      • As these consume disk space, the vCD UI was updated to show user when they are used:
      • Organizations List Page
      • A new Independent Disks count column is added.
      • Organization Properties Page
      • Independent Disks tab is added to show all independent disks belonging to vDC
      • Tab is not shown if no independent disk exists in the vDC.
      • Virtual Machine Properties Page
      • Hardware tab->Hard Disks section, attached independent disks are shown by their names and all fields for the disk are disabled as they are not editable.

That’s all I have time for right now.  As I said, there is more to come later on topics such as vDS enhancements, VXLAN, SRM, vCD Load Balancing, and vSphere Replication.  Stay tuned!

Patio Door Lock Replacement Tips

August 25th, 2012 by jason 1 comment »

1.  Make sure you purchase the correct replacement lock at your home improvement store.  This one looks like a match.

Snagit Capture

2.  Before removing the old lock or installing the new lock, insert the long door handle screw in the door.  Now it is safe to remove the two mounting screws located on the face of the lock.

Snagit Capture

3.  Failure to perform step #2 above allows gravity to cause the lock to fall down inside to the bottom of the door.  Not such a big deal if it’s the old lock being replaced, such as in my case, but you don’t want to drop your new replacement lock down there.  As far as I know, there is no easy way to retrieve it without removing the door.  Notice my new lock stays in place while I go grab the two mounting screws.

Snagit Capture

Well that’s it.  I hope I was able to save you that 2nd trip to the home improvement store.

Microsoft Windows Server 2012 Tips

August 16th, 2012 by jason 31 comments »

One of the benefits of working for Dell Compellent is having the privilege to collaborate with some very smart people who are subject matter experts in areas of technology I don’t get as much time to spend time on as I’d like to.  I get to share information with team members about vSphere, as well as Exchange, SQL, *nix, Oracle, and you might have guessed it… Microsoft Windows (including Hyper-V).  One of my colleagues has been working with Windows Server 2012 lately and he drew up a quick guide on some of the findings he had made.  Not only was he gracious enough to share it with his teammates, he was more than happy to share with the community when asked.  When I say community, of course I’m referring to readers of this blog.  So without further to do, here are some Windows Server 2012 (and perhaps even Windows 8) tips to get you started.

Navigating the New Server 2012 GUI

The look and feel of the Server 2012 GUI is quite different than Server 2008. While most of the familiar options and features are still available, the process of getting to them is quite different, and in some cases, more difficult.

Snagit Capture

1)      The “Start” button no longer exists in Server 2012.  To expose Start, jiggle your mouse in the lower left corner of the desktop and the Start option will appear as shown above.  This is a bit cumbersome in RDP sessions and takes some getting used to.

Snagit Capture

2)      The Start Menu presents applications and other options as tiles.

3)      To access Lock and Sign out, click on the User in the upper right for a drop-down menu.

Snagit Capture

4)      To access All Applications, right-click on any tile under Start, and then an options bar will appear at the bottom of the screen.  On this options bar, click on All Apps in the lower right.

Snagit Capture

5)      Under All Apps, you can find all the rest of the familiar (but now more difficult to find) options such as Command Prompt and Run.  To make these more easily accessible, pin them to the taskbar.

Snagit Capture

6)      Another hidden menu exits off the right side of the desktop.  To access it, move your mouse to the far right or lower-right corner of the screen and hold it there for a couple seconds.   Again, this is cumbersome in RDP sessions and takes some getting used to.

7)      As you can see above, the Restart and Shut down options are now buried a few layers deep so accessing them is a bit tedious.   Some customization suggestions below will help alleviate this.

Snagit Capture

8)      To stop the Server Manager window from automatically starting every time you log on, edit the Server Manager Properties and check the box Do not start Server Manager automatically at logon.

Customizations to Facilitate Better User Experience with Server 2012

You may find yourself a little frustrated with the changes introduced with the Server 2012 GUI because many apps/options/tools have been relocated and are therefore more difficult (and more time consuming) to find.

Below are some quick and simple customization changes to “restore” some of the of the Server 2008 look/feel/agility to the 2012 GUI.

1)      The first step is to install the Desktop Experience as found under Features.  Once installed, then the (My) Computer icon can be added back to the desktop.

Snagit Capture

a)      Launch Server Manager from the taskbar.

Snagit Capture

b)      Click on Add roles and features to launch the Add Roles and Features Wizard.  Under Features, check the box for Desktop Experience and then complete the wizard (requires a reboot).

Snagit Capture

c)       After rebooting, from the Desktop, right click and choose Personalize, Change Desktop Icons, and add the desired icons such as Computer and Control Panel.

d)      Right click on the Desktop again, and under View, set icon size to Small, and set Auto Arrange and Sort By options according to your preference.

Snagit Capture

2)      Customize the taskbar by pinning shortcuts for I.E., Run, Command Prompt, and other frequently used apps (as found under Start and All Apps) that you want to be quickly accessible.  For directions on how to access the Start and All Apps menus, see Page 2.

3)      Right click on the taskbar, select Properties, and select Use Small taskbar buttons, and under the Toolbars tab, add the Desktop toolbar.

4)      If you desire to add the Background Info (BGI) utility to your Windows 2012 server desktop, then complete the following steps:

Snagit Capture

  • From your network share or software repository containing BGInfo, copy the folder BGInfo to C:\BGInfo.  Edit the BGInfo.bgi config file to customize (if desired) the BGInfo settings.  (this is the latest 64-bit version of BGInfo)

Snagit Capture

  • To automatically refresh BGInfo each time you log on to the server, add a reg key (string value) called BGInfo with value of C:\BGInfo\LaunchBGI.batto:HKLM\Software\Microsoft\Windows\CurrentVersion\Run

Snagit Capture

  • If using mRemote, change the Display Wallpaper setting to Yes under the configuration settings for your server (the default setting is No).  Otherwise the BGInfo screen will not be passed to your display.

5)      To work around the cumbersome process of having to navigate to log-off, shutdown, or reboot commands under the hidden menus, place shortcuts to these operations on the Server 2012 desktop.  To make this process quick and easy, pre-defined shortcuts can be saved on a network share and copied down to each server installation.

 Snagit Capture

 

  • From the network share, copy the desktop shortcuts to Libraries\Documents\Public Documents on your 2012 server.

Snagit Capture

  • Once copied, open the Desktop_Icons folder, and copy and paste the icons found there to the public desktop (a hidden folder) which can be accessed at C:\Users\public\desktop (manually type this path in Windows Explorer as shown above to get to it).
  • Add or create other shortcuts as desired here so they will show on the public desktop.
  • By placing them on the public desktop, they will be there for all users, and will be preserved even when the server is sysprepped.

Snagit Capture

6)      When finished, your desktop will look similar to the above screen capture:

  • (My) Computer and Control Panel icons added to the desktop
  • Shutdown, Logoff, and Restart icons (which are shortcuts to the shutdown command) added to the desktop.  This is much quicker than having to access these options from the hidden menus on the left or right sides of the desktop, and it skips having to provide a reason for shutting down.
  • Shortcut to launch Disk Manager added to the desktop (add other shortcuts as desired)
  • Shortcuts to I.E., Run, and Command Prompt added to the taskbar
  • Desktop toolbar added to the taskbar
  • Background Info (BGInfo) provides for a blue background with the server name and other essential server specs on the desktop.  This will automatically refresh at each logon due to adding LaunchBGI.bat to Run in the system registry, and it can be refreshed manually at any time by clicking on the LaunchBGI icon on the public desktop.

Sysprep Suggestions

1)      When building a new gold image of a Windows 2012 server, include the above customizations before running Sysprep to allow cloned copies to boot with these modifications in place.  Most of the changes will be preserved in the sysprep image saving configuration time.

2)      Other suggested modifications you may want to consider making to a Windows 2012 image before sysprepping it to use as a gold image it include:

  1. Enable RDP
  2. Install Adobe Reader
  3. Using Roles and Features, install .Net 3.5 (set the path to \sources\sxs when prompted); Failover Clustering, MPIO, and Hyper-V
  4. Disable the firewall
  5. Disable I.E. security
  6. Disable User Account Control security (set to never notify)
  7. Fully patch the server
  8. If a physical server, run the applicable driver and firmware management/update utility to apply the latest drivers and firmware.
  9. Set the time zone to Central
  10. Install JRE (version of your choice, both the 32bit and 64bit versions)
  11. Other apps and features as desired

StarWind and Cirrus Tech Partner to Deliver Cutting Edge Technologies to the Cloud Computing Market

August 12th, 2012 by jason No comments »

Press Release

StarWind Solutions Become Available Through a Leading Canadian Web Hosting Company

Burlington, MA – 6 August 2012StarWind Software Inc., an innovative provider of storage virtualization software and VM backup technology, announced today a new partnership agreement with Cirrus Tech Ltd., a Canadian web hosting company specializing in VPS, VM and cloud hosting services. Companies collaborate to deliver best-in-breed Cloud services that help customers accelerate their businesses.

According to the agreement, Cirrus Tech extends its portfolio with StarWind storage virtualization software and will offer it to their customers as a dedicated storage platform that delivers a highly available and high performance scalable storage infrastructure that is capable of supporting heterogeneous server environments; as Cloud storage for private clouds as well as a robust solution for building Disaster Recovery (DR) plans.

StarWind SAN solutions deliver a wide variety of enterprise-class features, such as High Availability (HA), Synchronous Data Mirroring, Remote Asynchronous Replication, CDP/Snapshots, Thin Provisioning, Global Deduplication, etc., that make the stored data highly available, simplify storage management, and ensure business continuity and disaster recovery.

“Companies are increasingly turning to cloud services to gain efficiencies and respond faster to today’s changing business requirements.” said Artem Berman, Chief Executive Officer of StarWind Software, Inc. “We are pleased to combine our forces with Cirrus Tech in order to deliver our customers a wide range of innovative cloud services that will help their transition to a flexible and efficient shared IT infrastructure.”

“Every business needs to consider what would happen in the event of a disaster,” shares Cirrus CEO Ehsan Mirdamadi. “By bringing StarWind’s SAN solution to our customers, we are helping them to ease the burden of disaster recovery planning by offering powerful and affordable storage options. You never want to think of the worst, but when it comes to your sensitive data and business critical web operations, it’s always better to be safe than sorry. Being safe just got that much easier for Cirrus customers.”

To find out more about Cirrus’ web hosting services visit http://www.cirrushosting.com or call 1.877.624.7787.
For more information about StarWind, visit www.starwindsoftware.com

About Cirrus Hosting
Cirrus Tech Ltd. has been a leader in providing affordable, dependable VHS and VPS hosting services in Canada since 1999. They have hosted and supported hundreds of thousands of websites and applications for Canadian businesses and clients around the world. As a BBB member with an A+ rating, Cirrus Tech is a top-notch Canadian web hosting company with professional support, rigorous reliability and easily upgradable VPS solutions that grow right alongside your business. Their Canadian data center is at 151 Front Street in Toronto.

About StarWind Software Inc.
StarWind Software is a global leader in storage management and SAN software for small and midsize companies. StarWind’s flagship product is SAN software that turns any industry-standard Windows Server into a fault-tolerant, fail-safe iSCSI SAN. StarWind iSCSI SAN is qualified for use with VMware, Hyper-V, XenServer and Linux and Unix environments. StarWind Software focuses on providing small and midsize companies with affordable, highly availability storage technology which previously was only available in high-end storage hardware. Advanced enterprise-class features in StarWind include Automated HA Storage Node Failover and Failback (High Availability), Replication across a WAN, CDP and Snapshots, Thin Provisioning and Virtual Tape management.

Since 2003, StarWind has pioneered the iSCSI SAN software industry and is the solution of choice for over 30,000 customers worldwide in more than 100 countries and from small and midsize companies to governments and Fortune 1000 companies.

For more information on StarWind Software Inc., visit: www.starwindsoftware.com

View 5.1 Upgrade Experience. Composer, Permissions, and SSL – Oh My!

August 8th, 2012 by jason 2 comments »

The other night I upgraded the VMware View 5.0.1 environment in the lab to 5.1 which was released on May 16th.  Normally when I upgrade the View environment, I don’t actually perform an inline upgrade of the Connection Server or database.  The environment is small enough that I can flatten it and rebuild fresh from scratch (including brand new VMs for the infrastructure components such as the Connection Server) for each new version VMware releases.  Due to VMware’s aggressive release schedule, I also embed the production version in the infrastructure server name which helps me keep track of where things are at in the lab.  Thus, with each new release, I’m building new infrastructure VMs with updated names, rather than recycling the previous infrastructure VMs, renaming them, remove/re-add to the domain, and even then I’m left with a VM name which doesn’t match the name on the datastore folder.  Pushing the reset button and starting fresh obliterates any bad DNA or cooties the previous environment might have had and it gives me a little extra peace of mind when I sleep at night.

I was running a little short on time so for this round I decided to perform an inline upgrade to 5.1 rather than going through the normal rebuild routine.  After all, most production environments don’t have the luxury of starting over so now was as good a time as ever to test the upgrade process of View in the lab.  Again – a fairly simple setup: a Connection Server, View Composer 2.7 installed on the vCenter Server which for the first time in many releases will be upgraded to 3.0, back end databases on an external SQL server, and 3 small pools.

The View Connection Server upgrade went as planned. No issues to speak of there (yet).  However, I did struggle with the View Composer upgrade.  The first run through uninstalled View Composer and failed with an error which I wasn’t quick enough to capture.  I re-ran through the Composer installation and it failed again with the same error:

The wizard was interrupted before VMware View Composer could be completely installed.

While I was perfrming some troubleshooting, a couple of gracious folks on Twitter by the name of Diego Quintana and Tim Washburn (@daquintana and @mittim12 respectively) pointed out VMware KB article 2017773 Installing or upgrading View Composer fails with error: The wizard was interrupted before VMware View Composer could be completely installed which resolved my issue.  The previous View Composer installation had placed one or more keys in the directory C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\ which my user account no longer had NTFS permissions to.  The resolution was to simply relax the NTFS permissions both on the MachineKeys folder as well as the files inside of the folder for good measure.

I thought I was out of the woods, but not quite yet.  SSL certificate issues followed.

Snagit Capture

VMware made some new changes with regards to SSL in View 5.1 which are documented at :\Program Files\VMware\VMware View\Server\README.rtf

Copied and pasted verbatim, the release notes are:

Read These Notes!  Your View 5.1 Setup Will Be Easier!

You can read these notes in your language:
Français    Deutsch     简体中文     日本語     한국어

We made changes in View 5.1 that require you to configure View components a little differently than in the past.  These notes will help you to avoid potential pitfalls when you install or upgrade to View 5.1.

1)  You cannot downgrade View 5.1 Connection Server to previous versions.

In View 5.1, the View LDAP configuration is encrypted and cannot be used by earlier versions of View.

  • After you upgrade a View Connection Server instance to View 5.1, you cannot downgrade that instance to an earlier version.
  • After you upgrade all View Connection Server instances in a replicated group, you cannot add another instance that runs an earlier version of View.

Note: Downgrading was never supported, but in past releases it worked.  Now it won’t work.

2)  vCenter Server and View Composer hosts need valid SSL certificates.

  • Best choice: Ensure your vCenter Server and View Composer have Certificate Authority (CA)-provided certificates:

o  Install an SSL certificate, signed by a CA, on the Windows Server on which vCenter Server is installed.

o  Do the same for View Composer. If you install View Composer and vCenter Server on the same host, they can use the same certificate, but you must configure the certificate separately for each component.

* If you install the certificate before you install View Composer, you can select your certificate during the View Composer installation.

* If you replace the default certificate later, run the SviConfig ReplaceCertificate command to bind the new certificate to the port used by View Composer.

o  Make sure the CA for the new certificates, and any parent CAs, are trusted by each Windows server on which a View Connection Server instance is installed.

  • Alternative: After you add vCenter Server and View Composer to View, accept the thumbprint of the default certificate for View Composer by clicking Verify in View Administrator.  Do the same for vCenter Server.

More information: See “Configuring SSL Certificates for View Servers” in the View Installation guide.

3)  Security server and View Connection Server hosts need valid SSL certificates.

  • Best choice: After you install a View Connection Server instance or security server on a Windows Server host, open the Windows Server certificate store and take these steps:

o  Import an SSL certificate that is signed by a CA and that your clients can validate.

o  Make sure that the entire certificate chain, including intermediate certificates and root certificate, are installed.

o  Make sure the certificate has a private key, and mark the key as exportable.

o  Configure the certificate Friendly Name as vdm.

  • Alternative: Let the View server installer create a default certificate in the Windows Server certificate store. The certificate is self-signed and will be shown as invalid in View Administrator.
  • Upgrading to View 5.1: If your original View servers already have SSL certificates signed by a CA, you don’t have to do anything.  During the upgrade, View imports your certificates into the Windows Server certificate store.

If your original View servers have default certificates, upgrade your View servers and follow the Best choice steps shown above.

More information: See “Configuring SSL Certificates for View Servers” in the View Installation guide.

4)  Certificates for vCenter Server, View Composer, and View servers must include certificate revocation lists (CRLs).

View will not validate a certificate without a CRL.

  • Best Choice: lf needed, take these steps:

o  Add a CRL to your certificate.

o  Import the updated certificate into the Windows certificate store on the vCenter Server, View Composer, and View server host.

  • Alternative: Change the registry settings that control CRL checking.

More information: “Configuring Certificate Revocation Checking on Server Certificates” in the View Installation guide.

5)  Windows Firewall with Advanced Security must be enabled on Security Server and View Connection Server hosts. 

By default, IPsec rules govern connections between the View security server and View Connection Server and require Windows Firewall with Advanced Security to be enabled.

  • Best choice: Set Windows Firewall with Advanced Security to on before you install the View servers. Make sure it’s on for any active profiles; better still, set it to on for all profiles.
  • Alternative: Before you install security servers, open View Administrator and disable the Global Setting, Use IPsec for Security Server Connections, by setting it to no. (This is not recommended.)

6)  Back-end firewalls must be set up to support IPsec.

If you have a back-end firewall between security servers and View Connection Server instances, you must configure firewall rules to allow the connections to work.

More information: See “Configuring a Back-End Firewall to Support IPsec ” in the View Installation guide.

7)  View Clients must use HTTPS to connect to View.

View Connection Server instances and security servers use SSL for client connections.

  • If View clients connect via an SSL off-loading intermediate device, you must install the intermediate device’s SSL certificate on View Connection Server or security server.
  • The connection must be HTTPS whether or not a View client connects via an intermediate device such as a load balancer. If you use an intermediate device, and you want the connection between the intermediate device and View server to be over HTTP (SSL off-loading), configure the locked.properties file on the View server.
  • Older View clients that can choose not to use HTTPS will get an error if users select HTTP. Previously they were silently redirected to HTTPS. Clients that cannot make SSL connections will be unable to connect to View.

More information: See “Off-loading SSL Connections to Intermediate Servers” in the View Administration guide.

8)  Encrypted and cleansed View backups require new restore steps.

By default, View 5.1 backups are encrypted. You can also cleanse View backups (exclude passwords and other sensitive information from the backup data) or back up in plain text (not recommended).

  • To restore an encrypted backup, you must decrypt the data first. You must use the data recovery password that you provided when you installed View Connection Server.
  • Do not restore cleansed backups. Data such as passwords will be missing from your View LDAP configuration. View components will not function properly without this data. To restore normal functionality, you will have to use View Administrator to manually reset all passwords and other missing data items.

More information: See “Backing Up and Restoring View Configuration Data” in the View Administration guide.

9)  Before you can upgrade or reinstall a View 5.1 security server, you must remove the relevant IPsec rules from the paired View Connection Server instance so that fresh rules can be established.

  • In View Administrator, select the security server and click More Commands > Prepare for Upgrade or Reinstallation.

Note: You don’t need to remove a security server from View before you upgrade or reinstall the server.

More information: See “Prepare to Upgrade or Reinstall a Security Server” in the View Installation guide.

Ok, so basically VMware is pushing for the use of SSL certificates from a trusted CA whether that be externally (VeriSign, etc.) or internally (Microsoft Certificate Services) generated.  For the time being, I have ditched my internal Microsoft CA and wish to continue using the self signed certificates shipped and installed by View.  To do so, as explained in the README above, one must visit the System Health in the View Administrator Dashboard and verify the certificates for the vCenter Server as well as the View Composer Server (each will be seen in a red status in the dashboard).  The Connection Server certificate cannot be verified and will remain in a red status however from this point forward both the View Connection Server and View Composer will function normally.

Upgrading the View Agents and recomposing the pools was a non-issue and the upgrade was completed successfully.  After all is said and done, the environment is working and the upgrade was successful.  View 5.0.1 Clients have no problem connecting to the new 5.1 environment; I’ll get the clients upgraded in the near future and I’ll consider resurrecting the lab CA to generate trusted SSL certificates.

VCDXs To Recieve New Storage Book

July 25th, 2012 by jason 2 comments »

Snagit Capture

Last fall at the VMworld book store in Las Vegas, I picked up sample chapter 6 ALUA from Mostafa Khalil‘s (@MostafaVMW) upcoming book “Storage Design and Implementation in vSphere 5.0“.  The level of detail looked fantastic and I could hardly wait for the rest of the book to be released.

Fast forward to a month ago, nearing completion I joked with Mostafa that perhaps VCDXs could receive a copy of his new book. Mostafa gracefully accepted the challenge and delivered the good news tonight that he has arranged with VMware Press for all current VCDXs to receive a free copy of his book via e-copy or print.

What Mostafa has done is extremely generous and I’m really looking forward to receiving a copy of his new storage book once it is released (which should be very soon) so that I can read the rest.