Posts Tagged ‘Exchange’

Monster VMs & ESX(i) Heap Size: Trouble In Storage Paradise

September 12th, 2012

While running Microsoft Exchange Server Jetstress on vSphere 5 VMs in the lab, tests were failing about mid way through initializing its several TBs of databases.  This was a real head scratcher.  Symptoms were unwritable storage or lack of storage capacity.  Troubleshooting yielding errors such as “Cannot allocate memory”.  After some tail chasing, the road eventually lead to VMware KB article 1004424: An ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage.

As it turns out, ESX(i) versions 3 through 5 have a statically defined per-host heap size:

  • 16MB for ESX(i) 3.x through 4.0: Allows a max of 4TB open virtual disk capacity (again, per host)
  • 80MB for ESX(i) 4.1 and 5.x: Allows a max of 8TB open virtual disk capacity (per host)

This issue isn’t specific to Jetstress, Exchange, Microsoft, or a specific fabric type, storage protocol or storage vendor.  Exceeding the virtual disk capacities listed above, per host, results in the symptoms discussed earlier and memory allocation errors.  In fact, if you take a look at the KB article, there’s quite a laundry list of possible symptoms depending on what task is being attempted:

  • An ESXi/ESX 3.5/4.0 host has more that 4 terabytes (TB) of virtual disks (.vmdk files) open.
  • After virtual machines are migrated by vSphere HA from one host to another due to a host failover, the virtual machines fail to power on with the error:vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.
  • You see warnings in /var/log/messages or /var/log/vmkernel.logsimilar to:vmkernel: cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
    vmkernel: cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)
    cpu15:11905)WARNING: Heap: 2525: Heap cow already at its maximum size. Cannot expand.
    cpu15:11905)WARNING: Heap: 2900: Heap_Align(cow, 6160/6160 bytes, 8 align) failed. caller: 0x41802fd54443
    cpu4:1959755)WARNING:Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
    cpu4:1959755)WARNING: Heap: 2900: Heap_Align(vmfs3, 2099200/2099200 bytes, 8 align) failed. caller: 0x418009533c50
    cpu7:5134)Config: 346: “SIOControlFlag2” = 0, Old Value: 1, (Status: 0x0)
  • Adding a VMDK to a virtual machine running on an ESXi/ESX host where heap VMFS-3 is maxed out fails.
  • When you try to manually power on a migrated virtual machine, you may see the error:The VM failed to resume on the destination during early power on.
    Reason: 0 (Cannot allocate memory).
    Cannot open the disk ‘<<Location of the .vmdk>>’ or one of the snapshot disks it depends on.
  • The virtual machine fails to power on and you see an error in the vSphere client:An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)
  • A similar error may appear if you try to migrate or Storage vMotion a virtual machine to a destination ESXi/ESX host on which heap VMFS-3 is maxed out.
  • Cloning a virtual machine using the vmkfstools -icommand fails and you see the error:Clone: 43% done. Failed to clone disk: Cannot allocate memory (786441)
  • In the /var/log/vmfs/volumes/DatastoreName/VirtualMachineName/vmware.log file, you may see error messages similar to:2012-05-02T23:24:07.900Z| vmx| FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory
    2012-05-02T23:24:07.900Z| vmx| AIOGNRC: Failed to open ‘/vmfs/volumes/xxxx-flat.vmdk’ : Cannot allocate memory (c00000002) (0x2013).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-VMFS : “/vmfs/volumes/xxxx-flat.vmdk” : failed to open (Cannot allocate memory): AIOMgr_Open failed. Type 3
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-LINK : “/vmfs/volumes/xxxx.vmdk” : failed to open (Cannot allocate memory).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-CHAIN : “/vmfs/volumes/xxxx.vmdk” : failed to open (Cannot allocate memory).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-LIB : Failed to open ‘/vmfs/volumes/xxxx.vmdk’ with flags 0xa Cannot allocate memory (786441).
    2012-05-02T23:24:07.900Z| vmx| DISK: Cannot open disk “/vmfs/volumes/xxxx.vmdk”: Cannot allocate memory (786441).
    2012-05-02T23:24:07.900Z| vmx| Msg_Post: Error
    2012-05-02T23:24:07.900Z| vmx| [msg.disk.noBackEnd] Cannot open the disk ‘/vmfs/volumes/xxxx.vmdk’ or one of the snapshot disks it depends on.
    2012-05-02T23:24:07.900Z| vmx| [msg.disk.configureDiskError] Reason: Cannot allocate memory.

While VMware continues to raise the scale and performance bar for it’s vCloud Suite, this virtual disk and heap size limitation becomes a limiting constraint for monster VMs or vApps.  Fortunately, there’s a fairly painless resolution (at least up until a certain point):  Increase the Heap Size beyond its default value on each host in the cluster and reboot each host.  The advanced host setting to configure is VMFS3.MaxHeapSizeMB.

Let’s take another look at the default heap size and with the addition of its maximum allowable heap size value:

  • ESX(i) 3.x through 4.0:
    • Default value: 16MB – Allows a max of 4TB open virtual disk capacity
    • Maximum value: 128MB – Allows a max of 32TB open virtual disk capacity per host
  • ESX(i) 4.1 and 5.x:
    • Default value: 80MB – Allows a max of 8TB open virtual disk capacity
    • Maximum value: 256MB – Allows a max of 25TB open virtual disk capacity per host

After increasing the heap size and performing a reboot, the ESX(i) kernel will consume additional memory overhead equal to the amount of heap size increase in MB.  For example, on vSphere 5, the increase of heap size from 80MB to 256MB will consume an extra 176MB of base memory which cannot be shared with virtual machines or other processes running on the host.

Readers may have also noticed an overall decrease in the amount of open virtual disk capacity per host supported in newer generations of vSphere.  While I’m not overly concerned at the moment, I’d bet someone out there has a corner case requiring greater than 25TB or even 32TB of powered on virtual disk per host.  With two of VMware’s core value propositions being innovation and scalability, I would tip-toe lightly around the phrase “corner case” – it shouldn’t be used as an excuse for its gaps while VMware pushes for 100% data virtualization and vCloud adoption.  Short term, the answer may be RDMs. Longer term: vVOLS.

Updated 9/14/12: There are some questions in the comments section about what types of stoarge the heap size constraint applies to.  VMware has confirmed that heap size and max virtual disk capacity per host applies to VMFS only. The heap size constraint does not apply to RDMs nor does it apply to NFS datastores.

Updated 4/4/13: VMware has released patch ESXi500-201303401-BG to address heap issues.  This patch makes improvements to both default and maximum limits of open VMDK files per vSphere host.  After applying the above patch to each host, the default heap size for VMFS-5 datastores becomes 640MB which supports 60TB of open VMDK files per host.  These new default configurations are also the maximum values as well.  For additional reading on other fine blogs, see A Small Adjustment and a New VMware Fix will Prevent Heaps of Issues on vSphere VMFS Heap and The Case for Larger Than 2TB Virtual Disks and The Gotcha with VMFS.

Updated 4/30/13: VMware has released vSphere 5.1 Update 1 and as Cormac has pointed out here, heap issue resolution has been baked into this release as follows:

  1. VMFS heap can grow up to a maximum of 640MB compared to 256MB in earlier release. This is identical to the way that VMFS heap size can grow up to 640MB in a recent patch release (patch 5) for vSphere 5.0. See this earlier post.
  2. Maximum heap size for VMFS in vSphere 5.1U1 is set to 640MB by default for new installations. For upgrades, it may retain the values set before upgrade. In such cases, please set the values manually.
  3. There is also a new heap configuration “VMFS3.MinHeapSizeMB” which allows administrators to reserve the memory required for the VMFS heap during boot time. Note that “VMFS3.MinHeapSizeMB” cannot be set more than 255MB, but if additional heap is required it can grow up to 640MB. It alleviates the heap consumption issue seen in previous versions, allowing the ~ 60TB of open storage on VMFS-5 volumes per host to be accessed.

When reached for comment, Monster VM was quoted as saying “I’m happy about these changes and look forward to a larger population of Monster VMs like myself.”


Configuring Zimbra Mobile

February 19th, 2012

Shortly before VMware Partner Exchange, Microsoft Certificate Services in my long-in-the-tooth lab domain died and unfortunately took a handful of integrated services with it.  One of those dependent services was Microsoft Exchange 2010.  After attempts to repair and later rebuild and restore the Microsoft CA failed, I decided the time had finally come to ditch the boche.mcse Windows Active Directory domain which I originally built when Windows 2000 and Active Directory first launched.  There were a number of other lingering issues with the domain which I haven’t been able to repair and the time investment and frustration really wasn’t worth it.  I had been waiting for a catalyst in addition to a block of weekend time and today was the day.

I had been hosting Microsoft Exchange Server since version 5.5 and although I had accumulated quite a bit of experience over the years in building, migrating, and maintaining a small Exchange environment, the truth is Exchange has gotten way too complex and bloated for me to keep up with.  Being a VMware guy, I’ve had the itch to give Zimbra a try since the acquisition.  It seemed like a win-win solution: Ditch the complexities of Exchange and replace with a VMware messaging solution.  So after building the new domain this afternoon Exchange 2010 was retired and Zimbra was born.

Zimbra is incredibly easy to set up, especially when deploying the appliance version in .OVF format hosted on vSphere 5.  I was up and running with incoming and outgoing email in very little time.  One area I was concerned with when replacing Exchange with Zimbra was the integration and end user experience.  Zimbra ships with an HTML Outlook Web Access like interface which can be used for processing mail on a regular basis.  Would I lose the ability to use Outlook as a mail client or mobile device compatibility?  The answer was no on both accounts.  In addition to the web interface, I can continue to use Outlook 2010 connected to the Zimbra server via IMAP.  This is just about as easy to set up as attaching Outlook to a native Exchange environment.  Outlook detects the mail services provided by Zimbra and negotiates an IMAP configuration.  Done.

Configuring Zimbra Mobile wasn’t as straight forward but I was able to piece together the steps.  Zimbra’s documentation is decent but in some areas it feels explicitly vague, lacking step by step configuration detail.  With Outlook and the web client functioning well, the last step was to integrate Zimbra with my mobile devices: the iPhone and iPad.  These are the steps I followed:

Log into the Zimbra Collaboration Suite Appliance web console (https://<f.q.d.n.>:5480/).  On the Zimbra Administration tab, change the pull down box from All addresses or Accounts to Profiles.

Snagit Capture

In a default out-of-box deployment, there should be just one profile named ‘default’.  Modify that profile by enabling Mobile Sync on the Zimbra Mobile tab.  Save the changes to the profile.

Snagit Capture

Now that we have Mobile Sync enabled in a profile, we can apply that profile to each Zimbra user who needs messaging access from their mobile handheld.  Going back to the Dashboard tab, change the pull down box from Profiles to Accounts.  Modify the account in which you’d like to grant Zimbra Mobile access.  On the General Information tab for the user account, uncheck the box labeled ‘auto‘ and then begin typing the first few letters of the profile name of ‘default‘ in the Profile box.  After the first few letters of the profile are typed, the default profile name will appear and that can be selected with the mouse.  Now save the changes.

Snagit Capture

Now it’s time to configure the iPad and iPhone to connect to Zimbra Mobile.  The first step is to create a new mail account.  The type will be Exchange.

Snagit Capture

I found configuring the account with Zimbra Mobile easier than with Exchange 2010.  Provide the following values:

2. Email – this should match the email address tied to the account in Zimbra.

3. Server – this will be the f.q.d.n of the Zimbra server.  By default an SSL connection over port 443 will be attempted which Zimbra supports.  You can optionally specify an alternative port.  For example, I could come in externally over port 8080 and then have that translated by NAT to port 443 and sent to the Zimbra server.  Without a port defined, 443 is implied.

     Domain – not required with internal Zimbra authentication.  If authenticating with LDAP integration, this field would likely need to be populated.

4. Username – the name tied to the account in Zimbra.

5. Password – 12345

     Description – not used in Zimbra Mobile configuration.  It’s merely a differentiator for the account on a mobile device which may have multiple accounts set up.

Snagit Capture

If things are working correctly, you’ll be able to tap the blue Next button and you’ll be prompted to verify the synchronization of all three types of items: Mail, Contacts, and Calendars.  Once the account configuration is saved you’ll also see an additional slider showing that SSL is ON (the default).

Next up is to configure my wife’s Kindle Fire.  I don’t expect any issues there.

Microsoft Exchange 2003 to Exchange 2010 Upgrade Notes

May 14th, 2010

Last weekend I successfully upgraded, ahem, migrated the lab infrastructure from Microsoft Exchange 2003 to Exchange 2010.  This upgrade has been on my agenda for quite some time but I had been delaying it mainly due to lack of time and thorough knowledge of the steps.  I had a purchased the Microsoft Exchange Server 2010 Administrator’s Pocket Consultant (ISBN: 978-0-7356-2712-3) in January and marked up a few pages with a highlighter.  However, the deeper I got in the book, the more daunting the task seemed to have become, even for a simple one-server environment like mine.  In my mind, Exchange has always been somewhat of a beast, with increasing levels of difficulty as new editions emerged.  The pocket consultant series of books are wonderfully technical, but they haven’t been able to fit in my pocket for about a decade. They contain so much content that it has become difficult to rely on them as a CliffsNotes guide for platform upgrades, especially when it comes to Exchange.

Then two things happened miraculously at the same time.  First, I was invited to a private beta test of a virtualization related iPad application.  As part of this test, I needed to be able to send email from my iPad.  I had been unsuccessful thus far in getting Microsoft Exchange ActiveSync to work with the iPad (even after following Stephen Foskett’s steps) and could only assume that it was due to several years of wear and tear on my Exchange 2003 Server.  I needed to get that upgrade to Exchange 2010 done quickly.  Second, the May 2010 issue of Windows IT Pro magazine showed up in my mailbox.  To my delight, it was chock full of Exchange 2010 goodness, including a cover story of “Exchange 2003 to Exchange 2010 Step-by-Step Exchange Migration”. I’m pretty sure this was divine intervention with the message being “Get it done this weekend, you can do this.”

The upgrade article by Michael B. Smith started on page 26 and was 100% in scope.  The focus was a single server Exchange environment upgrade from 2003 to 2010.  I read the seven page artile in its entirety, marking up key “to-do” steps with a highlighter.  Following are some things I learned along the way:

  1. Naturally the Exchange server is virtualized on VMware vSphere.
  2. My Exchange environment is built upon a foundation that dates back as far as Exchange 5.5 (pre-Active Directory).  There would be no in place upgrades.  Exchange hasn’t provided an upgrade since Exchange 2003.  That suited me just fine as the Exchange 2003 server has been through so much neglect, although it had gotten pretty slow, it’s a miracle it was still functional.  The Exchange migration will consist of bringing up a fresh OS with a new installation of Exchange, and then migrating the mailboxes and services, and then retiring the old Exchange Server.  Microsoft calls this a migration rather than an upgrade.
  3. Exchange must be running in Native mode.  Not a problem, I was already there.
  4. Pre-migration, there exists a hotfix from Microsoft which is recommended to be installed on the Exchange 2003 server.
  5. The Schema Master mast be running Windows Server 2003 SP1 or higher.
  6. There needs to be at least one Global Catalog server at Windows Server 2003 SP1 or higher in the Exchange site.
  7. The AD forest needs to be at Server 2003 Forest Functional Level or higher.
  8. The AD domain needs to be at Server 2003 Domain Functional Level or higher.
  9. For migration flexibility purposes, Exchange 2003 and Exchange 2010 both support DFL and FFL up to Server 2008 R2.
  10. Exchange 2010 requires 64-bit hardware.  No problem, that requirement was met with vSphere .
  11. Exchange 2010 can be installed on Windows Server 2008 or Windows Server 2008 R2.  I naturally opted for R2.  No sense in deploying a two-year old OS when a more current one exists and is supported.  Plus, I personally need more exposure to 2008 and R2… 2003 is getting long in the tooth.
  12. Copy the Exchange DVD to a data/utility drive on the server.  Reason being, you can drop the most recent rollup available into the \Updates\ folder and basically perform a slipstream installation of Exchange with the most recent rollup applied out of the gate.  As of this writing, the most current is Rollup 3.
  13. Here’s a big time saver, install the server roles and features Exchange 2010 requires using the provided script on the DVD:
    \scripts\ServerManagerCmd -ip Exchange-Typical.xml -restart
    Other sample pre-requisite installer scripts can be found here.
  14. The 2007 Office System Converter: Microsoft Filter Pack (x64) is required to be installed.  This is downloadable from Microsoft’s website.  A little strange, but I’ll play along.  It’s required for the Exchange full-text search engine to search Office format documents.
  15. Run the following commands for good measure. It may or may not be required depending on what’s been done to the server so far:
    sc config NetTcpPortSharing start= auto
    net start NetTcpPortSharing
  16. Setup logs for Exchange are found in C:\ExchangeSetupLogs\  The main one is ExchangeSetup.log.  Hopefully you won’t have to rely on these logs and you are blessed with a trouble-free installation.
  17. There are the usual Active Directory preparatory steps to expand the Schema which seem to have increased in quantity but I could be hallucinating:
    1. /PrepareLegacyExchangePermissions
    2. /PrepareSchema
    3. /PrepareAD
    4. /PrepareAllDomains
  18. Installation can be invoked by CLI with /mode:install /roles:ca,ht,mb however, I chose a GUI installation which was more intuitive for me.
  19. The article stated the installation would take at least 20 minutes on fast hardware.  My installation took less than 15 minutes on a VM hosted by four year old servers attached to fibre channel EMC Celerra storage – bitchin.
  20. A Send connector is required before Exchange 2010 will route mailto the internet.
  21. Exchange 2010 ships with two Receive connectors but they must be configured before they will accept anonymous email from the internet.
  22. Exchange 2010 is managed by the Exchange Management Console which is called the EMC for short.  That will be easy to remember.
  23. Exchange 2010 is also managed by PowerShell scripts (also called an Exchange Management Shell, or EMS for short).  There are some configuration tasks which can only be made via PowerShell script and not via the EMC.
  24. Lend your end users and Helpdesk staff a hand by creating a meta-refresh document in C:\inetpub\wwwroot\ which points to https://<mail_server_fqdn>/owa effectively teleporting them into Outlook Web App (did you catch the name change? no more Outlook Web Access)
  25. Mailboxes are no longer moved online due to their potential size and problems which may occur if a mailbox is accessed during migration.  Mailbox migrations are now handled via EMC by way of a Move Request (either local [same org] or remote[different org]).  When a move request is submitted, the process begins immediately but may take some time to complete obviously based on the size of the mailbox as well as the quantity of mailboxes multiple selected for the move request.  Tony Redmond wrote a decent article on how this is done.  Scheduled move requests can be instantiated via PowerShell script.
  26. One of the final steps of a successful migration is properly decommissioning the old Exchange 2003 environment.  This is where things got a little hairy, and I half wasn’t surprised.  Upon attempting to uninstall Exchange 2003 to properly remove its tentacles from Active Directory and the Exchange organization, I was greeted by two errors in the following message:
    5-9-2010 9-16-31 PM
    In the legacy Exchange 2003 System Manager, there are two Recipient Update policies which exist.  Going from memory, one was for the domain which I was able to remove easily, and one was an Enterprise policy which cannot be removed via the System Manager.  Follow the instructions near the end of this article for the procedure to modify Active Directory with adsiedit.
    The second error message deals with removal of the legacy Routing Group Connector.  There were actually two which needed to be removed.  The only way to remove the Routing Group Connector is via PowerShell and it is also described towards the end of this article.
  27. After addressing the issues above, the uninstaller ran briefly and then failed for an unknown reason.  Upon attempting to re-run the uninstall, I noticed the ability uninstall Exchange 2003 via Add/Remove Programs in the Control Panel had disappeared, as if it was successfully uninstalled. Clearly it was not as the Exchange services still existed, were running, and I could launch System Manager and manage the organization.
  28. ActiveSync doesn’t work out of the box on privileged administrator level accounts due to security reasons.  If you accept the risk, this behavior can be changed by enabling the inheritance checkbox on the user account security property sheet.

I’m pretty happy with the results.  The process took took quite a few steps but I am nonetheless pleased.  Careful work following a very nicely outlined procedure by Michael B. Smith has yielded both a snappy-fast Exchange 2010 server on Windows Server 2008 R2 as well as ActiveSync integration with my iPad.  Exchange 2010 is a beast.  I can’t imagine tackling an Exchange project for anything larger than the smallest of environments.  I’m not sure how I can have so many years experience managing my own small Exchange environment yet still lack the confidence in the technology.  I guess it mostly runs itself and as I said earlier, it’s quite resilient meaning it doesn’t require much care and feeding from me.  And thank God for that.

How VMware virtualized Exchange 2007

January 8th, 2009

I often hear questions or concerns about virtualizing Exchange.  E-Oasis found a new VMware white paper and provides a nice lead in explaining how VMware corporate took their physical servers and migrated to virtual, reducing aggregrate hardware usage.

One might ask why VMware’s Exchange servers were not virtualized before this, particularly when VMware was a smaller company with less mailboxes?  Perhaps they decided earlier versions of Exchange were not virtualization candidates?  Maybe limitations in earlier version of ESX made it less than attractive?  I don’t know why but it would have been cooler to see VMware put their money where their mouth is earlier on.  Perhaps someone from VMware can chime in on a comment here.

At any rate, it’s an absolutely beautiful white paper and I’m actually surprised at the level of detail some of the diagrams get into providing network host names and IP addresses for the infrastructure.  I suppose they could be ficticious, but the names look rather authentic and not made up to me.  Kudos.

Take a look at VMware’s whitepaper here.