vSphere Integration With EMC Unisphere

February 14th, 2011 by jason 6 comments »

SnagIt CaptureIf you manage EMC unified storage running at least FLARE 30 and DART 6, or if you’re using a recent version of the UBER VSA, or if you’re one of the fortunate few who have had your hands on the new VNX series, then chances are you’re familiar with or you’ve at least experienced Unisphere, which is EMC’s single pane of glass approach to managing its multi protocol arrays.  For what is essentially a 1.0 product, I think EMC did a great job with Unisphere.  It’s modern.  It’s fast.  It has a cool sleek design and flows well.  They may have cut a few corners where it made sense (one can still see a few old pieces of Navisphere code here and there) but what counts for me the most at the end of the day is the functionality and efficiency gained by a consolidation of tools.

You’re probably reading this because you have a relationship with VMware virtualization.  Anyone who designs, implements, manages, or troubleshoots VMware virtual infrastructure also has a relationship with storage, most often shared storage.  Virtualization has been transforming the datacenter, and not just it’s composition.  The way we manage and collaborate from a technology perspective is also evolving.  Virtualization has brought about an intersection of technologies which is redefining roles and delegation of responsibilities.  One of the earlier examples of this was virtual networking.  With the introduction of 802.1Q VST in ESX, network groups found themselves fielding requests for trunked VLANs to servers and having to perform the associated design, capacity, and security planning.  Managing access to VLANs was a shift in delegated responsibility from the network team to the virtualization platform team.  Some years later, implementation of the Cisco Nexus 1000V in vSphere pulled most of the network related tasks back under the control of the network team.

Storage is another broad reaching technology upon which most of today’s computing relies upon, including virtualization.  Partners work closely with VMware to develop tools which provide seamless integration of overlapping technologies.  Unisphere is one of several products in the EMC portfolio which boasts this integration.  Granted, some of these VMware bits existed in Unisphere’s ancestor Navisphere.  However, I think it’s still worth highlighting some of the capabilities found in Unisphere.  EMC has been on an absolute virtualization rampage.  I can only imagine that with their commitment, these products will get increasingly better.

So what does this Unisphere/vSphere integration look like?  Let’s take a look…

In order to bring vSphere visibility into Unisphere, we need to make Unisphere aware of our virtual environment.  From the Host Management menu pane in Unisphere, choose Hypervisor Information Configuration Wizard:

SnagIt Capture

Classic welcome to the wizard.  Next:

SnagIt Capture

Select the EMC array in which to integrate a hypervisor configuration:

SnagIt Capture

In the following screen, we’re given the option to integrate either standalone ESX(i) hosts, vCenter managed hosts, or both.  In this case, I’ll choose vCenter managed hosts:

SnagIt Capture

Unisphere needs the IP address of the vCenter Server along with credentials having sufficient permissions to collect virtual infrastructure information.  FQDN of virtual infrastructure doesn’t work here (Wish list item), however, hex characters are accepted which tells me it’s IPv6 compatible:

SnagIt Capture

I see your infrastructure.  Would you like to add or remove items?

SnagIt Capture

Last step.  This is the virtual infrastructure we’re going to tie into.  Choose Finish:

SnagIt Capture

Congratulations.  Success.  Click Finish once more:

SnagIt Capture

Once completed, I see that the vCenter server I added has nested in the ESX host which it manages.  Again we see only the IP address representing a vCenter Server, rather than the FQDN itself.  This could get a little hairy in larger environments where a name is more familiar and friendlier than an IP address.  However, in Unisphere’s defense, at the time of adding a host we do have the option of adding a short description which would show up here.  Highlighting the ESX host reveals the VMs which are running on the host.  Nothing Earth shattering yet, but the good stuff lies ahead:

SnagIt Capture

Let’s look at the ESX host properties.  Here’s where the value starts to mount (storage pun intended).  The LUN Status tab reveals information of LUNs in use by the ESX host, as well as the Storage Processor configuration and status.  This is useful information for balance and performance troubleshooting purposes:

SnagIt Capture

Moving on to the Storage tab, more detailed information is provided about the LUN characteristics and how the LUNs are presented to the ESX host:

SnagIt Capture

The Virtual Machines tab is much the same as the VMware Infrastructure summary screen with the information that it provides.  However, it does provide the ability to drill down to specific VM information by way of hyperlinks:

SnagIt Capture

Let’s take a look at the VM named vma41 by clicking on the vma41 hyperlink from the window above.  The General tab provides some summary information about the VM and the storage, but nothing that we probably don’t already know at this point.  Onward:

SnagIt Capture

The LUN Status tab provides the VM to storage mapping and Storage Processor.  Once again, this is key information for performance troubleshooting.  Don’t get me wrong.  This information alone isn’t necessarily going to provide conclusive troubleshooting data.  Rather, it should be combined with other information collected such as  storage or fabric performance reports:

SnagIt Capture

Similar to the host metrics, the Storage tab from the VM point of view provides more detailed information about the datastore as well as the VM disk configuration.  Note the Type column which shows that the VM was thinly provisioned:

SnagIt Capture

There are a few situations which can invoke the age old storage administrator’s question: “What’s using this LUN?”  From the Storage | LUNs | Properties drill down (or from Storage | Pools/RAID Groups), Unisphere ties in the ESX hosts connected to the LUN as well as the VMs  living on the LUN.  Example use cases where this information is pertinent would be performance troubleshooting, storage migration or expansion, replication and DR/BCP planning.

SnagIt Capture

VM integration also lends itself to the Unisphere Report Wizard.  Here, reports can be generated for immediate display in a web browser, or they can be exported in .CSV format to be massaged further.

SnagIt Capture

If you’d like to see more, EMC has made available a three minute EMC Unisphere/VMware Integration Demo video which showcases integration and the flow of information:

In addition to that, you can download the FREE UBER VSA and give Unisphere a try for yourself.  Other EMC vSpecialist demos can be found at Everything VMware At EMC.

With all of this goodness and as with any product, there is room for improvement.  I mentioned before that by and large the vSphere integration code appears to be legacy which came from Navisphere.  Navisphere manages CLARiiON block storage only (fibre channel and native CLARiiON iSCSI).  What this means is that there is a gap in Unisphere/vSphere integration with respect to Celerra NFS and iSCSI.  For NFS, EMC has a vSphere plugin which Chad Sakac introduced about a year ago on his blog here and here.  While it’s not Unisphere integration, it does do some cool and useful things which are outlined in this product overview

In medium to large sized environments where teams can be siloed, it’s integration like this which can provide a common language, bridging the gap between technologies which have close dependencies with one another.  These tools work in the SMB space as well where staff will have both virtualization and storage areas of responsibility.  vSphere integration with Unisphere can provide a fair amount insight and efficiency.  I think this is just a slight representation of what future integration will be capable of.  VMware’s portfolio of virtualization, cloud, and data protection products continues to expand.  Each and every product VMware delivers is dependent on storage.  There is a tremendous opportunity to leverage each of these attach points for future integration.

vSphere 4.1 Update 1 Upgrade File Issues

February 11th, 2011 by jason 14 comments »

I began seeing this during upgrade testing last night in my lab but decided to wait a day to see if other people were having the same problems I was.  It is now being reported in various threads in the vSphere Upgrade & Install forum that vSphere 4.1 Update 1 upgrade files are failing to import into VMware Update Manager (VUM).  What I’m consistently seeing in multiple environements is:

  • .zip files which upgrade ESX and ESX from 4.0 to 4.1u1 will import into VUM successfully. 
  • .zip files which upgrade ESX and ESX from 4.1 to 4.1u1 fail to import into VUM.
  • I have not tested the upgrade files for ESX(i) 3.5 to 4.1u1.

The success and error message for all four .zip file imports are shown below.  Two successful.  Two failures.

SnagIt Capture

MD5SUM comparisons with VMware’s download site all result in matches.  I believe there is invalid metadata or corrupted .zip files being made available for download.

The workaround is to create a patch baseline in VUM which will instruct VUM to download the necessary upgrade files itself which is an alternative method to utilizing upgrade bundles and upgrade baselines in VUM.

VMware Releases vSphere 4.1 Update 1

February 10th, 2011 by jason 2 comments »

I’ve just been informed by my VMware Update Manager (VUM) that VMware has released vSphere 4.1 Update 1, including:

  • vCenter 4.1 Update 1
  • ESXi 4.1 Update 1
  • ESX 4.1 Update 1
  • vShield Zones 4.1 Update 1??

Will this be the last release of the ESX hypervisor in history?  Thus far, I haven’t seen that the HP, IBM, and Dell versions of ESXi 4.1 Update 1 are available for download yet.  They typically follow the VMware GA release by a few weeks.

Grab your copy now!

SnagIt Capture

The number of patch definitions downloaded (15 critical/28 total):

ID: ESX410-201101201-SG  Impact: HostSecurity  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 Core and CIM components

ID: ESX410-201101202-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 VMware-webCenter

ID: ESX410-201101203-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 esxupdate library

ID: ESX410-201101204-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 mptsas device driver

ID: ESX410-201101206-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 bnx2xi device driver

ID: ESX410-201101207-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 bnx2x device driver

ID: ESX410-201101208-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 sata device driver

ID: ESX410-201101211-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 VMware-esx-remove-rpms

ID: ESX410-201101213-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 net-enic device driver

ID: ESX410-201101214-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 qla4xxx device driver

ID: ESX410-201101215-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 net-nx-nic device driver

ID: ESX410-201101216-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 vaai plug-in

ID: ESX410-201101217-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 e1000e device driver

ID: ESX410-201101218-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 net-cdc-ether driver

ID: ESX410-201101219-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 e1000 device driver

ID: ESX410-201101220-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0 Updates the ESX 4.1 igb, tg3, scsi-fnic

ID: ESX410-201101221-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 HP SAS Controllers

ID: ESX410-201101222-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates ESX 4.1 mptsas, mptspi drivers

ID: ESX410-201101223-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0

3w-9xxx: scsi driver for VMware ESX

ID: ESX410-201101224-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: esx 4.1.0

vxge: net driver for VMware ESX

ID: ESX410-201101225-UG  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 Updates vmware-esx-pam-config library

ID: ESX410-201101226-SG  Impact: HostSecurity  Release date: 2011-02-10  Products: esx 4.1.0 Updates glibc packages

ID: ESX410-Update01  Impact: Critical  Release date: 2011-02-10  Products: esx 4.1.0 VMware ESX 4.1 Complete Update 1

ID: ESXi410-201101201-SG  Impact: HostSecurity  Release date: 2011-02-10  Products: embeddedEsx 4.1.0 Updates the ESXi 4.1 firmware

ID: ESXi410-201101202-UG  Impact: Critical  Release date: 2011-02-10  Products: embeddedEsx 4.1.0 Updates the ESXi  4.1 VMware Tools

ID: ESXi410-201101223-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: embeddedEsx 4.1.0

3w-9xxx: scsi driver for VMware ESXi

ID: ESXi410-201101224-UG  Impact: HostGeneral  Release date: 2011-02-10  Products: embeddedEsx 4.1.0

vxge: net driver for VMware ESXi

ID: ESXi410-Update01  Impact: HostGeneral  Release date: 2011-02-10  Products: embeddedEsx 4.1.0 VMware ESXi 4.1 Complete Update 1

Veeam FastSCP “Agents failed to start” During Copy

February 8th, 2011 by jason 3 comments »

Quick fix here for an operational task error I encountered in Veeam FastSCP 3.0.3.  I was trying to copy a file from the VMware vMA 4.1 appliance to a Windows folder using Veeam FastSCP 3.0.3.  In Veeam, the vMA appliance is registered as a Linux server & is recognized in the interface as the server object with the penguin.  In this example, I’m trying to copy /etc/motd to my local C: drive on Windows 7 Ultimate 64 bit:

SnagIt Capture

After a delay of several seconds, the error message is displayed:
Agents failed to start, server “vma41.boche.mcse”, client “localhost” Cannot connect to server [x.x.x.x:2500].

SnagIt Capture

The problem is an iptables daemon which is responsible for blocking communication on port 2500.  The workaround I used is to temporarily disable the iptables daemon as follows:

[vi-admin@vma41 etc]$ sudo service iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]

SnagIt Capture

Immediately after the iptables daemon is stopped, I’m able to copy the file:

SnagIt Capture

Now that my file is copied, I’ll undo the workaround, ensuring the vMA appliance is left in the state I had found it with its firewall rules applied:

[vi-admin@vma41 etc]$ sudo service iptables start
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_n[ OK ]

SnagIt Capture

Left alone, the workaround would persist until the next reboot.  Other workarounds to deal with this issue in a more permanent fashion would be to open port 2500 or use chkconfig to permanently disable the iptables daemon as follows:

sudo chkconfig iptables off
sudo service iptables save
sudo service iptables stop

VCP4 Exam Cram: VMware Certified Professional (2nd Edition)

February 7th, 2011 by jason 3 comments »

DSC04094

Tonight I was grateful to have received from Pearson Education the book VCP4 Exam Cram: VMware Certified Professional (2nd Edition).  The book is authored by Elias Khnaser (Twitter WWW) along with Technical Editors Brian Atkinson (WWW) and my friend Gabrie van Zanten (Twitter WWW).  This 2nd edition is 340 pages in length and ships with a cardboard fact filled cram sheet in the front as well as a CD in the back which contains VCP4 practice exams and an electronic versions of the cardboard cram sheet in case your friends are jealous of your intimate VMware vSphere knowledge and decide to swipe yours.

The book (ISBN-10: 0789740567) includes 10 chapters along with an appendix.  The chapter layout is as follows:

  1. Introducing vSphere 4
  2. Planning, Installing, and Configuring ESX/ESXi 4.1
  3. vNetworking Operations
  4. vStorage Operations
  5. Administration with vCenter
  6. Virtual Machine Operations
  7. vSphere Security and Web Access
  8. Managing vSphere Resources
  9. Monitoring vSphere Resources
  10. Backup and High Availability

There are a very few number of books which focus specifically on the VCP-410 exam.  This one is hot off the presses; published on January 31, 2011.  As such I would expect to find the most recent and relevant vSphere as well as exam information contained within.  Admittedly, I have not read this book cover to cover and have no plans to having passed the VCP4 17 months ago.  However, I have thumbed through several sections admiring the quality and thorough coverage of exam blueprint objectives.  This one looks pretty good and based on my positive experience with Exam Cram books in the past (Active Directory Services Design), I would recommend this book.  As a certification focused text, it contains practice questions at the end of each chapter and a comprehensive 75 question practice exam at the end of the book to test your knowledge.  Elias is open in that this book is recommended as supplemental learning in addition to the VMware ICM class (required for the certification) and a few other sources such as the VMware PDF documentation, but to be honest I think he sells himself a little short.  At well over 300 pages of exam focused material, I think it will go a long way towards passing the VCP-410 written test.

The book is available at Amazon in paperback for $28.40 or the Kindle version for $25.56.  Pearson was kind enough to ship me several copies which I will make available an the upcoming Minneapolis VMware User Group meeting.

What are you still doing here?  You should be reading this book.  Go.

StarWind Partners with OnApp Cloud Hosting

February 3rd, 2011 by jason 1 comment »

Press Release

Burlington, Mass. – February 02, 2010StarWind Software Inc., a global leader in developing iSCSI SAN software for small and midsize companies and OnApp, a leading developer of cloud management software for hosts, have joined forces to provide OnApp’s hosting cloud customers with an affordable and highly available SAN, free for one year and for a low monthly fee after the 1st year. Storage in the cloud helps overcome the burden of purchasing an expensive SAN solution, delaying or preventing IT’s migration to the cloud.

OnApp cloud software enables hosting providers to deploy clouds on commodity hardware, and manage cloud resources, failover, users and billing through a simple point-and-click interface. Cost-efficient and reliable storage is essential for stable business continuity in the cloud. The StarWind iSCSI SAN’s architecture and rich feature set provide an ideal solution for OnApp’s hosting customers. Since StarWind software can be installed on commodity servers the price point enables cloud services to be affordable, and eliminates vendor lock in associated with proprietary SAN hardware vendors.

“There’s a huge need for cost-effective storage in the hosting mass market, and especially in the fast-growing cloud hosting market,” said Carlos Rego, MD and Chief Architect of OnApp. “With StarWind we’re making it easy for hosts to add high performance cloud storage without huge up-front investment in SANs. Our special free for the 1st year  licensing offer helps reduce entry costs to cloud hosting, and with OnApp’s monthly licensing, another vital component of cloud hosting infrastructure is moved from CAPEX to OPEX.”

The StarWind partnership allows OnApp customers to deploy High Availability storage with a free one-year license for StarWind Enterprise HA 16TB or Unlimited TB editions, for up to two servers. After the 1st year, OnApp customers can license StarWind’s HA 16TB or Unlimited TB editions for a low monthly fee.

“Cloud hosting is the future of hosting and high availability storage is critical to provide server and application redundancy and uptime. In cooperation with our partner OnApp, we are pleased to contribute to the growing the cloud space. OnApp provides cost-effective and flexible cloud platforms and StarWind Software guarantees affordable and highly available SAN storage in the cloud,” said Art Berman, CEO of StarWind Software, Inc.

About OnApp

OnApp develops cloud management software for the hosting industry. OnApp software was developed from the ground up to enable mass-market hosts to build their own cloud hosting services. It enables hosts to deploy clouds in the datacenter using commodity hardware; provides rich functionality for cloud deployment, resource management, user management, failover and utility billing; has a high density design to maximize a host’s margins; and features pre-built integration to leading hosting billing engines, including WHMCS, Ubersmith and HostBill.

OnApp launched in July 2010 after two years of development. OnApp has offices in the US and Europe, employs more than 40 staff and can be found online at www.onapp.com.

For more information about OnApp, please contact:

Robert van der Meulen
+44 208 846 0855
press@onapp.com

 

About StarWind Software Inc.

StarWind Software is a global leader in storage management and SAN software for small and midsize companies. StarWind’s flagship product is SAN software that turns any industry-standard Windows Server into a fault-tolerant, fail-safe iSCSI SAN. StarWind iSCSI SAN is qualified for use with VMware, Hyper-V, XenServer and Linux and Unix environments. StarWind Software is focused on providing small and midsize companies with affordable, highly availability storage technology which previously was only available in high-end storage hardware. Advanced enterprise-class features in StarWind include Automated Storage Node Failover and Failback, Replication across a WAN, CDP and Snapshots, Thin Provisioning and Virtual Tape management.

Since 2003 StarWind has pioneered the iSCSI SAN software industry and is the solution of choice for over 30,000 customers worldwide in over 100 countries, from small and midsize companies to governments and Fortune 1000 companies.

Press Contacts:
StarWind Software Inc.
+1 (617) 449-7717
info@starwindsoftware.com


 

Social Media Links

Twitter: http://twitter.com/starwindsan

LinkedIn: http://www.linkedin.com/companies/starwind-software-inc.

Facebook: http://www.facebook.com/StarWind.Software

Jumbo Frames Comparison Testing with IP Storage and vMotion

January 24th, 2011 by jason 51 comments »

Are you thinking about implementing jumbo frames with your IP storage based vSphere infrastructure?  Have you asked yourself why or thought about the guaranteed benefits? Various credible sources discuss it (here’s a primer).  Some will highlight jumbo frames as a best practice but the majority of what I’ve seen and heard talk about the potential advantages of jumbo frames and what the technology might do to make your infrastructure more efficient.  But be careful to not interpret that as an order of magnitude increase in performance for IP based storage.  In almost all cases, that’s not what is being conveyed, or at least, that shouldn’t be the intent.  Think beyond SPEED NOM NOM NOM.  Think efficiency and reduced resource utilization which lends itself to driving down overall latency.  There are a few stakeholders when considering jumbo frames.  In no particular order:

  1. The network infrastructure team: They like network standards, best practices, a highly performing and efficient network, and zero down time.  They will likely have the most background knowledge and influence when it comes to jumbo frames.  Switches and routers have CPUs which will benefit from jumbo frames because processing less frames but more payload overall makes the network device inherently more efficient while using less CPU power and consequently producing less heat.  This becomes increasingly important on 10Gb networks.
  2. The server and desktop teams: They like performance and unlimited network bandwidth provided by magic stuff, dark spirits, and friendly gnomes.  These teams also like a postive end user experience.  Their platforms, which include hardware, OS, and drivers, must support jumbo frames.  Effort required to configure for jumbo frames increases with a rising number of different hardware, OS, and driver combinations.  Any systems which don’t support network infrastructure requirements will be a showstopper.  Server and desktop network endpoints benefit from jumbo frames much of the same way network infrastructure does: efficiency and less overhead which can lead to slightly measurable amounts of performance improvement.  The performance gains more often than not won’t be noticed by the end users except for process that historically take a long amount of time to complete.  These teams will generally follow infrastructure best practies as instructed by the network team.  In some cases, these teams will embark on an initiative which recommends or requires a change in network design (NIC teaming, jumbo frames, etc.).
  3. The budget owner:  This can be a project sponsor, departmental manager, CIO, or CEO.  They control the budget and thus spending.  Considerable spend thresholds require business justification.  This is where the benefit needs to justify the cost.  They are removed from the most of the technical persuasions.  Financial impact is what matters.  Decisions should align with current and future architectural strategies to minimize costly rip and replace.
  4. The end users:  Not surprisingly, they are interested in application uptime, stability, and performance.  They could care less about the underlying technology except for how it impacts them.  Reduction in performance or slowness is highly visible.  Subtle increases in performance are rarely noticed.  End user perception is reality.

The decision to introduce jumbo frames should be carefully thought out and there should be a compelling reason, use case, or business justification which drives the decision.  Because of the end to end requirements, implementing jumbo frames can bring with it additional complexity and cost to an existing network infrastructure.  Possibly the single best one size fits all reason for a jumbo frames design is a situation where jumbo frames is already a standard in the existing network infrastructure.  In this situation, jumbo frames becomes a design constraint or requirement.  The evangelistic point to be made is VMware vSphere supports jumbo frames across the board.  Short of the previous use case, jumbo frames is a design decision where I think it’s important to weigh cost and benefit.  I can’t give you the cost component as it is going to vary quite a bit from environment to environment depending on the existing network design.  This writing speaks more to the benefit component.  Liberal estimates claim up to 30% performance increase when integrating jumbo frames with IP storage.  The numbers I came up with in lab testing are nowhere close to that.  In fact, you’ll see a few results where IO performance with jumbo frames actually decreased slightly.  Not only do I compare IO with or without jumbo frames, I’m also able to compare two storage protocols with and without jumbo frames which could prove to be an interesting sidebar discussion.

I’ve come across many opinions regarding jumbo frames.  Now that I’ve got a managed switch in the lab which supports jumbo frames and VLANs, I wanted to see some real numbers.  Although this writing is primarily regarding jumbo frames, by way of the testing regimen, it is in some ways a second edition to a post I created one year ago where I compared IO performance of the EMC Celerra NS-120 among its various protocols. So without further ado, let’s get onto the testing.

 

Lab test script:

To maintain as much consistency and integrity as possible, the following test criteria was followed:

  1. One Windows Server 2003 VM with IOMETER was used to drive IO tests.
  2. A standardized IOMETER script was leveraged from the VMTN Storage Performance Thread which is a collaboration of storage performance results on VMware virtual infrastructure provided by VMTN Community members around the world.  The thread starts here, was locked due to length, and continues on in a new thread here.  For those unfamiliar with the IOMETER script, it basically goes like this: each run consists of a two minute ramp up followed by five minutes of disk IO pounding.  Four different IO patterns are tested independently.
  3. Two runs of each test were performed to validate consistent results.  A third run was performed if the first two were not consistent.
  4. One ESXi 4.1 host with a single IOMETER VM was used to drive IO tests.
  5. For the mtu1500 tests, IO tests were isolated to one vSwitch, one vmkernel portgroup, one vmnic, one pNIC (Intel NC360T PCI Express), one Ethernet cable, and one switch port on the host side.
  6. For the mtu1500 tests, IO tests were isolated to one cge port, one datamover, one Ethernet cable, and one switch port on the Celerra side.
  7. For the mtu9000 tests, IO tests were isolated to the same vSwitch, a second vmkernel portgroup configured for mtu9000, the same vmnic, the same pNIC (Intel NC360T PCI Express), the same Ethernet cable, and the same switch port on the host side.
  8. For the mtu9000 tests, IO tests were isolated to a second cge port configured for mtu9000, the same datamover, a second Ethernet cable, and a second switch port on the Celerra side.
  9. Layer 3 routes to between host and storage were removed to lessen network burden and to isolate storage traffic to the correct interfaces.
  10. 802.1Q VLANs were used isolate traffic and categorize standard traffic versus jumbo frame traffic.
  11. RESXTOP was used to validate storage traffic was going through the correct vmknic.
  12. Microsoft Network Monitor and Wireshark were used to validate frame lengths during testing.
  13. Activities known to introduce large volumes of network or disk activity were suspended such as backup jobs.
  14. Dedupe was suspended on all Celerra file systems to eliminate datamover contention.
  15. All storage tests were performed on thin provisioned virtual disks and datastores.
  16. The same group of 15 spindles were used for all NFS and iSCSI tests.
  17. The uncached write mechanism was enabled on the NFS file system for all NFS tests.  You can read more about that in the following EMC best practices document VMware ESX Using EMC Celerra Storage Systems

Lab test hardware:

SERVER TYPE: Windows Server 2003 R2 VM on ESXi 4.1
CPU TYPE / NUMBER: 1 vCPU / 512MB RAM (thin provisioned)
HOST TYPE: HP DL385 G2, 24GB RAM; 2x QC AMD Opteron 2356 Barcelona
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC Celerra NS-120 / 15x 146GB 15K / 3x RAID5 5×146
SAN TYPE: / HBAs: NFS / swiSCSI / 1Gb datamover ports (sorry, no FCoE)
OTHER: 3Com SuperStack 3 3870 48x1Gb Ethernet switch

 

Lab test results:

NFS test results.  How much better is NFS performance with jumbo frames by IO workload type?  The best result seen here is about a 7% performance increase by using jumbo frames, however, 100% read is a rather unrealistic representation of a virtual machine workload.  For NFS, I’ll sum it up as a 0-3% IOPS performance improvement by using jumbo frames.

SnagIt Capture

SnagIt Capture

iSCSI test results.  How much better is iSCSI performance with jumbo frames by IO workload type?  Here we see that iSCSI doesn’t benefit from the move to jumbo frames as much as NFS.  In two workload pattern types, performance actually decreased slightly.  Discounting the unrealistic 100% read workload as I did above, we’re left with a 1% IOPS performance gain at best by using jumbo frames with iSCSI.

SnagIt Capture

SnagIt Capture

NFS vs iSCSI test results.  Taking the best results from each protocol type, how do the protocol types compare by IO workload type?  75% of the best results came from using jumbo frames.  The better performing protocol is a 50/50 split depending on the workload pattern.  One interesting observation to be made in this comparison is how much better one protocol performs over the other.  I’ve heard storage vendors state that the IP protocol debate is a snoozer, they preform roughly the same.  I’ll grant that in two of the workload types below, but in the other two, iSCSI pulls a significant performance lead over NFS. Particularly in the Max Throughput-50%Read workload where iSCSI blows NFS away.  That said, I’m not outright recommending iSCSI over NFS.  If you’re going to take anything away from these comparisons, it should be “it depends”.  In this case, it depends on the workload pattern, among a handful of other intrinsic variables.  I really like the flexibility in IP based storage and I think it’s hard to go wrong with either NFS or iSCSI.

SnagIt Capture

SnagIt Capture

vMotion test results.  Up until this point, I’ve looked at the impact of jumbo frames on IP based storage with VMware vSphere.  For curiosity sake, I wanted to to address the question “How much better is vMotion performance with jumbo frames enabled?”  vMotion utilizes a VMkernel port on ESXi just as IP storage does so the ground work has already been established making this a quick test.  I followed roughly the same lab test script outlined above so that the most consistent and reliable results could be produced.  This test wasn’t rocket science.  I simply grabbed a few different VM workload types (Windows, Linux) with varying sizes of RAM allocated to them (2GB, 3GB, 4GB).  I then performed three batches of vMotions of two runs each on non jumbo frames (mtu1500) and jumb frames (mtu9000).  Results varied.  The first two batches showed that jumbo frames provided a 7-15% reduction in elapsed vMotion time.  But then the third and final batch contrasted previous results with data revealing a slight decrease in vMotion efficiency with jumbo frames.  I think there’s more variables at play here and this may be a case where more data sampling is needed to form any kind of reliable conclusion.  But if you want to go by these numbers, vMotion is quicker on jumbo frames more often than not.

SnagIt Capture

SnagIt Capture

The bottom line:

So what is the bottom line on jumbo frames, at least today?  First of all my disclaimer:  My tests were performed on an older 3Com network switch.  Mileage may vary on newer or different network infrastructure.  Unfortunately I did not have access to a 10Gb lab network to perform this same testing.  However, I believe my findings are consistent with the majority of what I’ve gathered from the various credible sources.  I’m not sold on jumbo frames as a provider of significant performance gains.  I wouldn’t break my back implementing the technology without an undisputable business justification.  If you want to please the network team and abide by the strategy of an existing jumbo frames enabled network infrastructure, then use jumbo frames with confidence.  If you want to be doing everything you possibly can to boost performance from your IP based storage network, use jumbo frames.  If you’re betting the business on IP based storage, use jumbo frames.  If you need a piece of plausible deniability when IP storage performance hits the fan, use jumbo frames. If you’re looking for the IP based storage performance promise land, jumbo frames doesn’t get you there by itself.  If you come across a source telling you otherwise, that jumbo frames is the key or sole ingredient to the Utopia of incomprehendable speeds, challenge the source.  Ask to see some real data.  If you’re in need of a considerable performance boost of your IP based storage, look beyond jumbo frames.  Look at optimizing, balancing, or upgrading your back end disk array.  Look at 10Gb.  Look at fibre channel.  Each of these alternatives are likely to get you better overall performance gains than jumbo frames alone.  And of course, consult with your vendor.