Posts Tagged ‘Hardware’

Free VMware vSphere Client for iPad Available

March 18th, 2011

SnagIt CaptureIt has been an exciting couple of months for VMware in terms of product releases.  Now, VMware has done it again.  Effective immediately, the vSphere Client for iPad is announced and is generally available for download from the Apple App Store.  Leave your wallet and iTunes gift cards parked.  Similar to the VMware View Client for iPad, this app is also brought to the community free of charge.  From anywhere, we can now view key performance metrics and perform essential management tasks in a simplified and portable interface.

The new client is not meant to be functionally equivalent to the existing vSphere Client for Windows.  Rather, the idea is to be able to perform the most common vSphere administrator tasks.  This release is version 1.0.1.  As such, not all of the desired features and functionality is baked in.  Future development will be an iterative process from the GA release point forward. Feedback from end users will be collected and improvements will be built into future versions.  vMotion will perhaps be the most desired feature but unfortunately it did not make GA release.  VMware promises it will be the next feature added so that is more good news to look forward to on the horizon. 

Other potential wish list items which didn’t make the GA build are ESX Service Console, ESXi DCUI, and guest VM console access.  In my opinion, I wouldn’t look for console features any time soon.  I believe the spirit of the vSphere Client for iPad is to provide simplified management through an easy to use interface ala knobs and buttons.  Console access falls into that last 20% of advanced troubleshooting which extends beyond the intended use case of the iPad Client.

Architecture

So what’s under the hood?  Let’s take a look.  Aside from the foundational vSphere infrastructure (which is available as a free 60-day evaluation), there are two components, both free, which enable the delivery of portable management bliss:  the vCMA and the client for iPad itself.  To connect with the client from a remote location via the internet, a VPN connection on the iPad placing it local on the destination network is required.  Like the View Client for iPad, the vSphere Client for iPad is developed for iPad only.  No iPhone, iOther, etc.  The logic is built into the vCMA which will make it extensible for Android in the future.  Additionally, the vCMA will eventually be retired and its functionality will be rolled natively into vCenter Server.  I like this idea because my lab is getting to be somewhat appliance heavy which limits capacity to run the traditional VMs I want to be testing with.  Following is a visual overview of the architecture:

SnagIt Capture

SnagIt Capture

As mentioned earlier, future development will be an iterative process based on customer feedback.  These discussions can be aired in the vSphere Client for iPad VMTN Community forums located at the URL below.  Do not be shy.  VMware WANTS your feedback:

http://communities.vmware.com/community/vmtn/vsphere/ipadclient

Now let’s take a bit of a deeper dive by looking at the installation process and the management capabilities of the app.

Installation and Configuration

  1. Download the vSphere Client for iPad application from the iTunes Store.
  2. Once the vCMA virtual appliance (available for free at http://labs.vmware.com/flings/vcma) powers on, on the home screen of the iPad go to “Settings”, scroll down and tap on “vSphere Client” (an example this screen is shown below).
  3. Enter the IP Address of the vCMA virtual appliance in the “Web Server” field (again, see the sample image below).
  4. Ensure your iPad has connectivity to the vCMA virtual appliance (note: as of this writing, the vCMA has SSL enabled by default). This may entail configuring the iPad’s built-in VPN client. Consult Apple’s documentation on configuring the built-in VPN client.
  5. Launch the vSphere Client for iPad application and enter the host, username and password for the vCenter Server or vSphere Host you wish to connect to.

SnagIt Capture

Management Capabilities

Search for vSphere hosts and virtual machines.�
Reboot vSphere hosts or put them into maintenance mode.

SnagIt Capture

Manage virtual machines with the ability to start, stop and suspend.�
View and restore virtual machines’ snapshots

SnagIt Capture

Monitor the performance of vSphere hosts and virtual machines:

SnagIt Capture

Diagnose vSphere hosts and virtual machines using built-in ping and traceroute tools:

SnagIt Capture

Videos

Following are a few short video clips which VMware has made available covering the vSphere Client for iPad.

Configure the vCMA Virtual Appliance:

httpv://www.youtube.com/watch?v=msjXKWFdgcM

Configure & use the iPad app:

httpv://www.youtube.com/watch?v=6kRalVLzMvE

Summary of the iPad development by VMware at VMworld in Copenhagen October 2010:

httpv://www.youtube.com/watch?v=UseseTSNOP0

VMware is sure to gain popularity by offering virtualization and cloud management tools for portable devices… and at the right price.  VMware is listening to feedback and has already reacted with a modified list price in this GA release.  I think last week’s launch of the View Client for iPad was a big hit.  It will be interesting to see how well received this app is, particularly by the *nix folks who have been patiently waiting their turn for some client development love.

Updated 3/20/11:  Srinivas Krishnamurti, Senior Director for Mobile Solutions at VMware, has written a piece on his blog over at the Office of the CTO.  Read it here: VMware vSphere Client for iPad has left the building…

VMware Talk Puzzler

March 15th, 2011

SnagIt Capture

I’ve been a fan of the Car Talk radio program since I was introduced to it in 1993.  I hope the Tappet brothers don’t mind if I borrow the theme from one of their popular segments appropriately called Puzzler.  It seemed fitting for this article which I’m going to call VMware Talk Puzzler.  Not surprising, the goal of the Car Talk Puzzler is to listen to the problem (which is typically not simple), then provide the root cause.  In this adaptation, I’ll present a real life vSphere problem.  If you choose to take a stab, your job is three fold:

1) Identify the root cause of the problem.

2) Identify the solution.

3) Identify the unique tasks or chain of events which lead to the problem.

Here we go.

I was called in to help troubleshoot a problem.  “Carl” had created a virtual machine in a VMware vSphere 4.1 Update 1 cluster.  The problem Carl was experiencing was that the VM would not power on.  Error messages in vCenter include but are not limited to:

  • “Failed to find a host for powering on the virtual machine.”
  • “The following faults explain why the registered host is not compatible.”
  • “The number of virtual CPUs may be limited by the guest OS selected for the virtual machine or by the licensing for the host.”

I asked if the ESXi cluster and vCenter were licensed.  Carl confirmed by showing me that vCenter was licensed with Standard Edition and the hosts which wouldn’t power on the VM were still using 60 day Evaluation licensing as they were just recently built.  I further verified the Evaluation licensing had not yet expired.

I asked Carl to show me details of the VM.  He proceeded to show me the VM was created with the following shell specifications:

  • Guest OS: Microsoft Windows Server 2008 (64-bit)
  • VM Version: 4
  • CPU: 8
  • Memory: 4096MB 

 

1) Identify the root cause of Carl’s vSphere problem.

2) Identify the solution for Carl.

3) Identify the tasks or chain of events which lead to Carl’s problem.

If you think you know the answer, write it on the top of a shrink wrapped pallet containing a Cisco UCS 5100 Series Blade Server Chassis, fully loaded with UCS B200 M2 Blade Servers each with 192GB RAM, UCS 6100 Series Fabric Interconnects, and a pair of Cisco Nexus 5548P next generation 10GbE switches and send to my mailing address.

Or… reply in the comments section below.

The first correct and complete answer (I hope there is just one) will receive internet recognition, real life respect, and if I can find one, a prize.  No promises on that last one but I’ll see what I can do.

VMware View Client for iPad Released

March 9th, 2011

SnagIt CaptureThere’s an old saying which goes “The best things in life are free”.  Even better are those things which will forever remain free.  Such is the case with the new VMware View Client for iPad, announced and made available this morning!  By the time you read this, the bits will already be available for download in the Apple App Store.  GET IT NOW!

Development efforts for the new client stem from VMware Product Manager Tedd Fox who is no stranger to iPad Apps.  Tedd also lead the development and is on the patent for the Citrix client for the iPad.  Tedd’s policy?  “I never charge for clients”.  So long as Tedd is at VMware, this client (and future versions, of which there are going to be many, rapid fired) will be free.

Following are some notable product features, frequently asked questions, as well as current limitations (and from here on out I’m going to refer to the VMware View Client as the vVC in the interest of less typing [by the way, I just made that up so if VMware adopts vVC, you heard it here first folks]):

  • The vVC for iPad will compete with Wyse PocketCloud.  A few of the differences between the two apps are:
    • vVC is purpose built for the VMware View use case and associated connectivity.  I think this will be important to keep in mind as the product is run through its paces and feature requests start to roll in.  VMware is going to pay more attention to feature requests which tie to its use with View and align with the VMware Enterprise Desktop architectural and strategic direction.
    • Instead of a hockey puck like cursor, the vVC sports a rendered track pad on the iPad surface.  VMware believes this no nonsense approach leads to a better user experience. The track pad, as well as other dockable modules such as function keys, can be moved around the display or hidden.
    • Wyse PocketCloud = $14.99 plus additional bolt on feature costs
    • vVC = $FREE
    • Other than the price tag, protocol is the biggie:  vVC supports PCoIP only.  Whereas PocketCloud supports Terminal Services/Remote Desktop RDP, View (RDP) and VNC.  We’ll see if this drives VMware View 4.6 upgrades/deployments which boast the required PCoIP gateway feature.  Alternatively, I’m assuming vVC PCoIP via VPN tunnel will also work with VMware View versions 4.6 and prior.
  • The vVC is currently available for iPad only with Android tablets targeted mid year.  There are no plans to support the smaller 7″ range of devices.  Tedd explains “the app just doesn’t feel right on smaller devices.”  No comment as of yet on HP TouchPad futures.
  • iPad 2 compatibility?  The honest answer is nobody knows at this time.  Nobody but Apple has an iPad 2 today.  vVC will likely work on the iPad 2, but there is a chance it won’t.  With future versions of vVC scheduled to come fast and furious, I doubt the wait would be long for full functionality on iPad 2, if it doesn’t already work out of the gate on March 11th when iPad 2 is released.  What we do know is that PCoIP does not currently support cameras, iPad 2 or otherwise.
  • Video and Audio:
    • vVC will support unidirectional audio. However, due to lack of Teradici integration, there will be no bidirectional audio support for this release.
    • 1024 x 768 video out is supported with the Apple VGA adapter (sold separately).
  • vVC supports connectivity to multiple brokers and multiple sessions, but not simultaneously.  Not until there’s a compelling use case.
  • There is no iPad multitasking support in the GA version but it is being worked on.  Wyse PocketCloud doesn’t have this either, or at least it doesn’t work for me as sessions are always disconnected when I multitask.
  • Dock keyboard and Bluetooth keyboard pairing support.
  • Local/LAN printing from the VDI session is supported, Apple/Air printing is not.
  • The VMware View for iPad VMTN community forum has been created at:

So enough socializing.  Feast your eyes on some candy captured by an iPad running the new View Client for iPad.

The vVC is launched and prompting for a broker.  The only information needed to get up and running with this app is a View broker URL and credentials:

photo-4-print

Previously visited sessions are available for selection along with a thumbnail of the desktop.  I believe the way this works is that the thumbnail is captured when the previous session is disconnected.  I don’t believe this is a dynamic representation of what’s currently displayed on the desktop.  The latter wouldn’t be very practical if desktops were locked or screen saver enabled:

photo-1-print

Wyse PocketCloud and iPad users in general will find the finger gestures familiar.  Comparing the two apps, there are both similarities and minor differences in how the gestures map to functions.

photo-2-print

Displayed here are some floating modules:  the track pad and two sets of function keys.  Also visible at the top is a pull down menu for the vVC:

SnagIt Capture 

Not much to say here so I’ll add some evangelism:  I’m so pumped about a free VMware App that I’ll probably forget about Enterprise Plus and per VM licensing for at least a day:

photo-print

Here’s a demo video from VMware which showcases some of the features:

Apple has strict protocols for its App Store.  Nobody outside of the development company gets pre-release copies or BETA software.  Nobody outside of VMware has had their hands on this app yet, including myself, so I write this piece from information gathered from those at VMware who have developed and worked with the product quite extensively already.  As I stated before, I’m overwhelmed with confidence in Tedd and his passion for client technology and from what I’ve seen, this client looks very promising.  I’m looking forward to grabbing this app ASAP and I wish Tedd and VMware a very successful launch.  I also look forward to the future releases and features Tedd promises.  After all, upgrading apps on the iPad isn’t nearly the bummer that Windows or other platform application upgrades are what with the reboots, compatibility issues, etc.  I’ll end with another quote from an old friend of mine who used to commonly say “What do you want for free?”  In this case, it would seem VMware has done a pretty good job with the GA version of vVC.  At this time I couldn’t ask for much more but ask me in a few weeks once I’ve had some seat time with it.

vSphere Integration With EMC Unisphere

February 14th, 2011

SnagIt CaptureIf you manage EMC unified storage running at least FLARE 30 and DART 6, or if you’re using a recent version of the UBER VSA, or if you’re one of the fortunate few who have had your hands on the new VNX series, then chances are you’re familiar with or you’ve at least experienced Unisphere, which is EMC’s single pane of glass approach to managing its multi protocol arrays.  For what is essentially a 1.0 product, I think EMC did a great job with Unisphere.  It’s modern.  It’s fast.  It has a cool sleek design and flows well.  They may have cut a few corners where it made sense (one can still see a few old pieces of Navisphere code here and there) but what counts for me the most at the end of the day is the functionality and efficiency gained by a consolidation of tools.

You’re probably reading this because you have a relationship with VMware virtualization.  Anyone who designs, implements, manages, or troubleshoots VMware virtual infrastructure also has a relationship with storage, most often shared storage.  Virtualization has been transforming the datacenter, and not just it’s composition.  The way we manage and collaborate from a technology perspective is also evolving.  Virtualization has brought about an intersection of technologies which is redefining roles and delegation of responsibilities.  One of the earlier examples of this was virtual networking.  With the introduction of 802.1Q VST in ESX, network groups found themselves fielding requests for trunked VLANs to servers and having to perform the associated design, capacity, and security planning.  Managing access to VLANs was a shift in delegated responsibility from the network team to the virtualization platform team.  Some years later, implementation of the Cisco Nexus 1000V in vSphere pulled most of the network related tasks back under the control of the network team.

Storage is another broad reaching technology upon which most of today’s computing relies upon, including virtualization.  Partners work closely with VMware to develop tools which provide seamless integration of overlapping technologies.  Unisphere is one of several products in the EMC portfolio which boasts this integration.  Granted, some of these VMware bits existed in Unisphere’s ancestor Navisphere.  However, I think it’s still worth highlighting some of the capabilities found in Unisphere.  EMC has been on an absolute virtualization rampage.  I can only imagine that with their commitment, these products will get increasingly better.

So what does this Unisphere/vSphere integration look like?  Let’s take a look…

In order to bring vSphere visibility into Unisphere, we need to make Unisphere aware of our virtual environment.  From the Host Management menu pane in Unisphere, choose Hypervisor Information Configuration Wizard:

SnagIt Capture

Classic welcome to the wizard.  Next:

SnagIt Capture

Select the EMC array in which to integrate a hypervisor configuration:

SnagIt Capture

In the following screen, we’re given the option to integrate either standalone ESX(i) hosts, vCenter managed hosts, or both.  In this case, I’ll choose vCenter managed hosts:

SnagIt Capture

Unisphere needs the IP address of the vCenter Server along with credentials having sufficient permissions to collect virtual infrastructure information.  FQDN of virtual infrastructure doesn’t work here (Wish list item), however, hex characters are accepted which tells me it’s IPv6 compatible:

SnagIt Capture

I see your infrastructure.  Would you like to add or remove items?

SnagIt Capture

Last step.  This is the virtual infrastructure we’re going to tie into.  Choose Finish:

SnagIt Capture

Congratulations.  Success.  Click Finish once more:

SnagIt Capture

Once completed, I see that the vCenter server I added has nested in the ESX host which it manages.  Again we see only the IP address representing a vCenter Server, rather than the FQDN itself.  This could get a little hairy in larger environments where a name is more familiar and friendlier than an IP address.  However, in Unisphere’s defense, at the time of adding a host we do have the option of adding a short description which would show up here.  Highlighting the ESX host reveals the VMs which are running on the host.  Nothing Earth shattering yet, but the good stuff lies ahead:

SnagIt Capture

Let’s look at the ESX host properties.  Here’s where the value starts to mount (storage pun intended).  The LUN Status tab reveals information of LUNs in use by the ESX host, as well as the Storage Processor configuration and status.  This is useful information for balance and performance troubleshooting purposes:

SnagIt Capture

Moving on to the Storage tab, more detailed information is provided about the LUN characteristics and how the LUNs are presented to the ESX host:

SnagIt Capture

The Virtual Machines tab is much the same as the VMware Infrastructure summary screen with the information that it provides.  However, it does provide the ability to drill down to specific VM information by way of hyperlinks:

SnagIt Capture

Let’s take a look at the VM named vma41 by clicking on the vma41 hyperlink from the window above.  The General tab provides some summary information about the VM and the storage, but nothing that we probably don’t already know at this point.  Onward:

SnagIt Capture

The LUN Status tab provides the VM to storage mapping and Storage Processor.  Once again, this is key information for performance troubleshooting.  Don’t get me wrong.  This information alone isn’t necessarily going to provide conclusive troubleshooting data.  Rather, it should be combined with other information collected such as  storage or fabric performance reports:

SnagIt Capture

Similar to the host metrics, the Storage tab from the VM point of view provides more detailed information about the datastore as well as the VM disk configuration.  Note the Type column which shows that the VM was thinly provisioned:

SnagIt Capture

There are a few situations which can invoke the age old storage administrator’s question: “What’s using this LUN?”  From the Storage | LUNs | Properties drill down (or from Storage | Pools/RAID Groups), Unisphere ties in the ESX hosts connected to the LUN as well as the VMs  living on the LUN.  Example use cases where this information is pertinent would be performance troubleshooting, storage migration or expansion, replication and DR/BCP planning.

SnagIt Capture

VM integration also lends itself to the Unisphere Report Wizard.  Here, reports can be generated for immediate display in a web browser, or they can be exported in .CSV format to be massaged further.

SnagIt Capture

If you’d like to see more, EMC has made available a three minute EMC Unisphere/VMware Integration Demo video which showcases integration and the flow of information:

In addition to that, you can download the FREE UBER VSA and give Unisphere a try for yourself.  Other EMC vSpecialist demos can be found at Everything VMware At EMC.

With all of this goodness and as with any product, there is room for improvement.  I mentioned before that by and large the vSphere integration code appears to be legacy which came from Navisphere.  Navisphere manages CLARiiON block storage only (fibre channel and native CLARiiON iSCSI).  What this means is that there is a gap in Unisphere/vSphere integration with respect to Celerra NFS and iSCSI.  For NFS, EMC has a vSphere plugin which Chad Sakac introduced about a year ago on his blog here and here.  While it’s not Unisphere integration, it does do some cool and useful things which are outlined in this product overview

In medium to large sized environments where teams can be siloed, it’s integration like this which can provide a common language, bridging the gap between technologies which have close dependencies with one another.  These tools work in the SMB space as well where staff will have both virtualization and storage areas of responsibility.  vSphere integration with Unisphere can provide a fair amount insight and efficiency.  I think this is just a slight representation of what future integration will be capable of.  VMware’s portfolio of virtualization, cloud, and data protection products continues to expand.  Each and every product VMware delivers is dependent on storage.  There is a tremendous opportunity to leverage each of these attach points for future integration.

Jumbo Frames Comparison Testing with IP Storage and vMotion

January 24th, 2011

Are you thinking about implementing jumbo frames with your IP storage based vSphere infrastructure?  Have you asked yourself why or thought about the guaranteed benefits? Various credible sources discuss it (here’s a primer).  Some will highlight jumbo frames as a best practice but the majority of what I’ve seen and heard talk about the potential advantages of jumbo frames and what the technology might do to make your infrastructure more efficient.  But be careful to not interpret that as an order of magnitude increase in performance for IP based storage.  In almost all cases, that’s not what is being conveyed, or at least, that shouldn’t be the intent.  Think beyond SPEED NOM NOM NOM.  Think efficiency and reduced resource utilization which lends itself to driving down overall latency.  There are a few stakeholders when considering jumbo frames.  In no particular order:

  1. The network infrastructure team: They like network standards, best practices, a highly performing and efficient network, and zero down time.  They will likely have the most background knowledge and influence when it comes to jumbo frames.  Switches and routers have CPUs which will benefit from jumbo frames because processing less frames but more payload overall makes the network device inherently more efficient while using less CPU power and consequently producing less heat.  This becomes increasingly important on 10Gb networks.
  2. The server and desktop teams: They like performance and unlimited network bandwidth provided by magic stuff, dark spirits, and friendly gnomes.  These teams also like a postive end user experience.  Their platforms, which include hardware, OS, and drivers, must support jumbo frames.  Effort required to configure for jumbo frames increases with a rising number of different hardware, OS, and driver combinations.  Any systems which don’t support network infrastructure requirements will be a showstopper.  Server and desktop network endpoints benefit from jumbo frames much of the same way network infrastructure does: efficiency and less overhead which can lead to slightly measurable amounts of performance improvement.  The performance gains more often than not won’t be noticed by the end users except for process that historically take a long amount of time to complete.  These teams will generally follow infrastructure best practies as instructed by the network team.  In some cases, these teams will embark on an initiative which recommends or requires a change in network design (NIC teaming, jumbo frames, etc.).
  3. The budget owner:  This can be a project sponsor, departmental manager, CIO, or CEO.  They control the budget and thus spending.  Considerable spend thresholds require business justification.  This is where the benefit needs to justify the cost.  They are removed from the most of the technical persuasions.  Financial impact is what matters.  Decisions should align with current and future architectural strategies to minimize costly rip and replace.
  4. The end users:  Not surprisingly, they are interested in application uptime, stability, and performance.  They could care less about the underlying technology except for how it impacts them.  Reduction in performance or slowness is highly visible.  Subtle increases in performance are rarely noticed.  End user perception is reality.

The decision to introduce jumbo frames should be carefully thought out and there should be a compelling reason, use case, or business justification which drives the decision.  Because of the end to end requirements, implementing jumbo frames can bring with it additional complexity and cost to an existing network infrastructure.  Possibly the single best one size fits all reason for a jumbo frames design is a situation where jumbo frames is already a standard in the existing network infrastructure.  In this situation, jumbo frames becomes a design constraint or requirement.  The evangelistic point to be made is VMware vSphere supports jumbo frames across the board.  Short of the previous use case, jumbo frames is a design decision where I think it’s important to weigh cost and benefit.  I can’t give you the cost component as it is going to vary quite a bit from environment to environment depending on the existing network design.  This writing speaks more to the benefit component.  Liberal estimates claim up to 30% performance increase when integrating jumbo frames with IP storage.  The numbers I came up with in lab testing are nowhere close to that.  In fact, you’ll see a few results where IO performance with jumbo frames actually decreased slightly.  Not only do I compare IO with or without jumbo frames, I’m also able to compare two storage protocols with and without jumbo frames which could prove to be an interesting sidebar discussion.

I’ve come across many opinions regarding jumbo frames.  Now that I’ve got a managed switch in the lab which supports jumbo frames and VLANs, I wanted to see some real numbers.  Although this writing is primarily regarding jumbo frames, by way of the testing regimen, it is in some ways a second edition to a post I created one year ago where I compared IO performance of the EMC Celerra NS-120 among its various protocols. So without further ado, let’s get onto the testing.

 

Lab test script:

To maintain as much consistency and integrity as possible, the following test criteria was followed:

  1. One Windows Server 2003 VM with IOMETER was used to drive IO tests.
  2. A standardized IOMETER script was leveraged from the VMTN Storage Performance Thread which is a collaboration of storage performance results on VMware virtual infrastructure provided by VMTN Community members around the world.  The thread starts here, was locked due to length, and continues on in a new thread here.  For those unfamiliar with the IOMETER script, it basically goes like this: each run consists of a two minute ramp up followed by five minutes of disk IO pounding.  Four different IO patterns are tested independently.
  3. Two runs of each test were performed to validate consistent results.  A third run was performed if the first two were not consistent.
  4. One ESXi 4.1 host with a single IOMETER VM was used to drive IO tests.
  5. For the mtu1500 tests, IO tests were isolated to one vSwitch, one vmkernel portgroup, one vmnic, one pNIC (Intel NC360T PCI Express), one Ethernet cable, and one switch port on the host side.
  6. For the mtu1500 tests, IO tests were isolated to one cge port, one datamover, one Ethernet cable, and one switch port on the Celerra side.
  7. For the mtu9000 tests, IO tests were isolated to the same vSwitch, a second vmkernel portgroup configured for mtu9000, the same vmnic, the same pNIC (Intel NC360T PCI Express), the same Ethernet cable, and the same switch port on the host side.
  8. For the mtu9000 tests, IO tests were isolated to a second cge port configured for mtu9000, the same datamover, a second Ethernet cable, and a second switch port on the Celerra side.
  9. Layer 3 routes to between host and storage were removed to lessen network burden and to isolate storage traffic to the correct interfaces.
  10. 802.1Q VLANs were used isolate traffic and categorize standard traffic versus jumbo frame traffic.
  11. RESXTOP was used to validate storage traffic was going through the correct vmknic.
  12. Microsoft Network Monitor and Wireshark were used to validate frame lengths during testing.
  13. Activities known to introduce large volumes of network or disk activity were suspended such as backup jobs.
  14. Dedupe was suspended on all Celerra file systems to eliminate datamover contention.
  15. All storage tests were performed on thin provisioned virtual disks and datastores.
  16. The same group of 15 spindles were used for all NFS and iSCSI tests.
  17. The uncached write mechanism was enabled on the NFS file system for all NFS tests.  You can read more about that in the following EMC best practices document VMware ESX Using EMC Celerra Storage Systems

Lab test hardware:

SERVER TYPE: Windows Server 2003 R2 VM on ESXi 4.1
CPU TYPE / NUMBER: 1 vCPU / 512MB RAM (thin provisioned)
HOST TYPE: HP DL385 G2, 24GB RAM; 2x QC AMD Opteron 2356 Barcelona
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC Celerra NS-120 / 15x 146GB 15K / 3x RAID5 5×146
SAN TYPE: / HBAs: NFS / swiSCSI / 1Gb datamover ports (sorry, no FCoE)
OTHER: 3Com SuperStack 3 3870 48x1Gb Ethernet switch

 

Lab test results:

NFS test results.  How much better is NFS performance with jumbo frames by IO workload type?  The best result seen here is about a 7% performance increase by using jumbo frames, however, 100% read is a rather unrealistic representation of a virtual machine workload.  For NFS, I’ll sum it up as a 0-3% IOPS performance improvement by using jumbo frames.

SnagIt Capture

SnagIt Capture

iSCSI test results.  How much better is iSCSI performance with jumbo frames by IO workload type?  Here we see that iSCSI doesn’t benefit from the move to jumbo frames as much as NFS.  In two workload pattern types, performance actually decreased slightly.  Discounting the unrealistic 100% read workload as I did above, we’re left with a 1% IOPS performance gain at best by using jumbo frames with iSCSI.

SnagIt Capture

SnagIt Capture

NFS vs iSCSI test results.  Taking the best results from each protocol type, how do the protocol types compare by IO workload type?  75% of the best results came from using jumbo frames.  The better performing protocol is a 50/50 split depending on the workload pattern.  One interesting observation to be made in this comparison is how much better one protocol performs over the other.  I’ve heard storage vendors state that the IP protocol debate is a snoozer, they preform roughly the same.  I’ll grant that in two of the workload types below, but in the other two, iSCSI pulls a significant performance lead over NFS. Particularly in the Max Throughput-50%Read workload where iSCSI blows NFS away.  That said, I’m not outright recommending iSCSI over NFS.  If you’re going to take anything away from these comparisons, it should be “it depends”.  In this case, it depends on the workload pattern, among a handful of other intrinsic variables.  I really like the flexibility in IP based storage and I think it’s hard to go wrong with either NFS or iSCSI.

SnagIt Capture

SnagIt Capture

vMotion test results.  Up until this point, I’ve looked at the impact of jumbo frames on IP based storage with VMware vSphere.  For curiosity sake, I wanted to to address the question “How much better is vMotion performance with jumbo frames enabled?”  vMotion utilizes a VMkernel port on ESXi just as IP storage does so the ground work has already been established making this a quick test.  I followed roughly the same lab test script outlined above so that the most consistent and reliable results could be produced.  This test wasn’t rocket science.  I simply grabbed a few different VM workload types (Windows, Linux) with varying sizes of RAM allocated to them (2GB, 3GB, 4GB).  I then performed three batches of vMotions of two runs each on non jumbo frames (mtu1500) and jumb frames (mtu9000).  Results varied.  The first two batches showed that jumbo frames provided a 7-15% reduction in elapsed vMotion time.  But then the third and final batch contrasted previous results with data revealing a slight decrease in vMotion efficiency with jumbo frames.  I think there’s more variables at play here and this may be a case where more data sampling is needed to form any kind of reliable conclusion.  But if you want to go by these numbers, vMotion is quicker on jumbo frames more often than not.

SnagIt Capture

SnagIt Capture

The bottom line:

So what is the bottom line on jumbo frames, at least today?  First of all my disclaimer:  My tests were performed on an older 3Com network switch.  Mileage may vary on newer or different network infrastructure.  Unfortunately I did not have access to a 10Gb lab network to perform this same testing.  However, I believe my findings are consistent with the majority of what I’ve gathered from the various credible sources.  I’m not sold on jumbo frames as a provider of significant performance gains.  I wouldn’t break my back implementing the technology without an undisputable business justification.  If you want to please the network team and abide by the strategy of an existing jumbo frames enabled network infrastructure, then use jumbo frames with confidence.  If you want to be doing everything you possibly can to boost performance from your IP based storage network, use jumbo frames.  If you’re betting the business on IP based storage, use jumbo frames.  If you need a piece of plausible deniability when IP storage performance hits the fan, use jumbo frames. If you’re looking for the IP based storage performance promise land, jumbo frames doesn’t get you there by itself.  If you come across a source telling you otherwise, that jumbo frames is the key or sole ingredient to the Utopia of incomprehendable speeds, challenge the source.  Ask to see some real data.  If you’re in need of a considerable performance boost of your IP based storage, look beyond jumbo frames.  Look at optimizing, balancing, or upgrading your back end disk array.  Look at 10Gb.  Look at fibre channel.  Each of these alternatives are likely to get you better overall performance gains than jumbo frames alone.  And of course, consult with your vendor.

IBM x3850 M2 shows 8 sockets on ESXi 4.1

December 9th, 2010

Working with an IBM x3850 M2, I noticed VMware ESXi 4.1 was reporting 8 processor sockets when I know this model has only 4 sockets.  It was easily noticable as I ran out of ESX host licensing prematurely.  The problem is also reported with the IBM x3950 M2 in this thread.

SnagIt Capture

SnagIt Capture

Here’s the fix:  Reboot the host and flip a setting in the BIOS.

POST -> F1 -> Advanced Setup -> CPU Options -> Clustering Technology. Toggle the Clustering Technology configuration from Logical Mode to Physical Mode.

After the above change is made, sanity is restored in that ESXi 4.1 will properly see 4 sockets and licenses will be consumed appropriately.

SnagIt Capture

SnagIt Capture

Flow Control

November 29th, 2010

Thanks to the help from blog sponsorship, I’m able to maintain a higher performing lab environment than I ever had been up until this point.  One area which I hadn’t invested much in, at least from a lab standpoint, is networking.  In the past, I’ve always had some sort of small to mid density unmanageable Ethernet switch.  And this was fine.  Household name brand switches like Netgear and SMC from Best Buy and NewEgg performed well enough and survived for years in the higher temperature lab environment.  Add to that, by virtue of being unmanaged, they were plug and play.  No time wasted fighting a mis configured network. 

I recently picked up a 3Com SuperStack 3 Switch 3870 (48 1GbE ports).  It’s not 10GbE but it does fit my budget along with a few other networking nice-to-haves like VLANs and Layer 3 routing.  Because this switch is managed, I can now apply some best practices from the IP based storage realm.  One of those best practices is configuring Flow Control for VMware vSphere with network storage.  This blog post is mainly to record some pieces of information I’ve picked up along the way and to open a dialog with network minded readers who may have some input.

So what is network Flow Control? 

NetApp defines Flow Control in TR-3749 as “the process of managing the rate of data transmission between two nodes to prevent a fast sender from over running a slow receiver.”  NetApp goes on to advise that Flow Control can be set at the two endpoints (ESX(i) host level and the storage array level) and at the Ethernet switch(es) in between.

Wikipedia is in agreement with the above and adds more meat to the discussion including the following “The overwhelmed network element will send a PAUSE frame, which halts the transmission of the sender for a specified period of time. PAUSE is a flow control mechanism on full duplex Ethernet link segments defined by IEEE 802.3x and uses MAC Control frames to carry the PAUSE commands. The MAC Control opcode for PAUSE is 0X0001 (hexadecimal). Only stations configured for full-duplex operation may send PAUSE frames.

What are network Flow Control best practices as they apply to VMware virtual infrastructure with NFS or iSCSI network storage?

Both NetApp and EMC agree that Flow Control should be enabled in a specific way at the endpoints as well as at the Ethernet switches which support the flow of traffic:

  • Endpoints (that’s the ESX(i) hosts and the storage arrays) should be configured with Flow Control send/tx on, and receive/rx off.
  • Supporting Ethernet switches should be configured with Flow Control “Desired” or send/tx off and receive/rx on.

One item to point out here is that although both mainstream storage vendors recommend these settings for VMware infrastructures as a best practice, neither of their multi protocol arrays ship configured this way.  At least not the units I’ve had my hands on which includes the EMC Celerra NS-120 and the NetApp FAS3050c.  The Celerra is configured out of the box with Flow Control fully disabled and I found the NetApp configured for Flow Control set to full (duplex?).

Here’s another item of interest.  VMware vSphere hosts are configured out of the box to auto negotiate Flow Control settings.  What does this mean?  Network interfaces are able to advertise certain features and protocols which they were purpose built to understand following the OSI model and RFCs of course.  One of these features is Flow Control.  VMware ESX ships with a Flow Control setting which adapts to its environment.  If you plug an ESX host into an unmanaged switch which doesn’t advertise Flow Control capabilities, ESX sets its tx and rx flags to off.  These flags tie specifically to PAUSE frames mentioned above.  When I plugged in my ESX host into the new 3Com managed switch and configured the ports for Flow Control to be enabled, I subsequently found out using the ethtool -a vmnic0 command that both tx and rx were enabled on the host (the 3Com switch has just one Flow Control toggle: enabled or disabled).  NetApp provides a hint to this behavior in their best practice statement which says “Once these [Flow Control] settings have been configured on the storage controller and network switch ports, it will result in the desired configuration without modifying the flow control settings in ESX/ESXi.”  Jase McCarty pointed out back in January a “feature” of the ethtool in ESX.  Basically, ethtool can be used to display current Ethernet adapter settings (including Flow Control as mentioned above) and it can also be used to configure settings.  Unfortunately, when ethtool is used to hard code a vmnic for a specific Flow Control configuration, that config lasts until the next time ESX is rebooted.  After reboot, the modified configuration does not persist and it reverts back to auto/auto/auto.  I tested with ESX 4.1 and the latest patches and the same holds true.  Jase offers a workaround in his blog post which allows the change to persist by embedding it in /etc/rc.local.

Third item of interest.  VMware KB 1013413 talks about disabling Flow Control using esxcfg-module for Intel NICs and ethtool for Broadcom NICs.  This article specifically talks about disabling Flow Control when PAUSE frames are identified on the network.  If PAUSE frames are indicative of a large amount of traffic which a receiver isn’t able to handle, it would seem to me we’d want to leave Flow Control enabled (by design to mediate the congestion) and perform root cause analysis on exactly why we’ve hit a sustained scaling limit (and what do we do about it long term).

Fourth.  Flow Control seems to be a simple mechanism which hinges on PAUSE frames to work properly.  If the Wikipedia article is correct in that only stations configured for full-duplex operation may send PAUSE frames, then it would seem to me that both network endpoints (in this case ESX(i) and the IP based storage array) should be configured with Flow Control set to full duplex, meaning both tx and rx ON.  This conflicts with the best practice messages from EMC and NetApp although it does align with the FAS3050 out of box configuration.  The only reasonable explanation is that I’m misinterpreting the meaning of full-duplex here.

Lastly, I’ve got myself all worked up into a frenzy over the proper configuration of Flow Control because I want to be sure I’m doing the right thing from both a lab and infrastructure design standpoint, but in the end Flow Control is like the Shares mechanism in VMware ESX(i):  The values or configurations invoked apply only during periods of contention.  In the case of Flow Control, this means that although it may be enabled, it serves no useful purpose until a receiver on the network says “I can’t take it any more” and sends the PAUSE frames to temporarily suspend traffic.  I may never reach this tipping point in the lab but I know I’ll sleep better at night knowing the lab is configured according to VMware storage vendor best practices.