Hello gang! For anyone who doesn’t specifically follow the NetApp blogs, this is just a quick heads up to let you know that NetApp has updated its popular NetApp and VMware vSphere Storage Best Practices book and is offering 1,000 free copies of the new Version 2.0 edition.
The free copies are available while supplies last so get registered for yours soon!
It’s the last presentation of the day and the last presentation overall for Gestalt IT Tech Field Day Seattle. We’ve made a short journey from the Microsoft store in Redmond, WA to to NEC in Bellevue. Anyone who knows the NEC brand is aware of their diverse portfolio of products and perhaps their services. Today’s discussion, however, will focus on Storage Solutions.
First a bit of background information on NEC as a corporation:
Storage. NEC opened up with some of today’s storage challenges faced by many. Enter HYDRAstor, a two-tier grid architecture comprised of the following key building blocks:
HYDRAstor delivers the following industry standard benefits:
A few of other key selling points about HYDRAstor:
I know some of the other delegates were really excited about HYDRAstor and its enabling technologies. Sorry NEC, I wasn’t feeling it. HYDRAstor’s approach seems to consume more rack space than the competition, more cabling, and based on today’s lab walkthru, more cooling.
Note : Tech Field Day is a sponsored event. Although I receive no direct compensation and take personal leave to attend, all event expenses are paid by the sponsors through Gestalt IT Media LLC. No editorial control is exerted over me and I write what I want, if I want, when I want, and how I want.
Compellent’s hardware approach is a modular one. Many of the components, such as drives and interfaces (Ethernet, FC, etc.), are easily replacable and hot swappable, eliminating the need to “rip and replace” the entire frame of hardware and providing the ability to upgrade components without taking down the array.
In April of 2010, Compellent introduced the new zNAS solution:
Compellent introduces the new zNAS solution, which consolidates file and block storage on a single, intelligent platform. The latest unified storage offering from Compellent integrates next-generation ZFS software, high-performance hardware and Fluid Data architecture to actively manage and move data in a virtual pool of storage, regardless of the size and type of block, file or drive. Enterprises can simplify management, intelligently scale capacity, improve performance for critical applications and reduce complexity and costs.
Fluid Data Storage is Compellent’s granular approach to data management
Volume Creation
Volume Recovery
Volume Management
Integration
Compellent performed a live demo of their Replay (Snapshot) feature with a LUN presented to a Windows host. It worked slick and as expected. Compellent’s Windows based storage management UI has a fresh, no-nonsense, 21st century feel to it which I can appreciate.
We closed discussion answering the question “Why Compellent?” Top Reasons:
Follow them on Twitter at @Compellent.
Thank you Compellent for the presentation and I’m sure I’ll see you back in Minnesota!
Note : Tech Field Day is a sponsored event. Although I receive no direct compensation and take personal leave to attend, all event expenses are paid by the sponsors through Gestalt IT Media LLC. No editorial control is exerted over me and I write what I want, if I want, when I want, and how I want.
We’re on to our 3rd and final presentation here at Gestalt IT Tech Field Day. After a short road trip into beautiful downtown Seattle, we’ve arrived at F5. At 1,800 employees strong, F5 was named one of the best places to work in the Seattle area. From a high level, F5′s business goal is to optimize the end user experience.
Today, F5 showed us simulated long distance vMotion. F5 enables this with mid-range BIG-IP appliances stretching a Layer 2 network between two geographically disbursed datacenters along with providing WAN Optimization to access IP based storage between datacenters. In addition, the hardware appliances expose APIs which VMware Orchestrator uses to assist the F5 into directing traffic between sites. F5 has tested at up to 300ms round trip latency and a 10Mbps link. This is what it looks like:
Another thing I learned today is that just a few months ago, in March 2010, F5 released the BIG-IP LTM VE. This is a virtual appliance that falls in the BIG-F5 family of products.
Today that appliance is supported on only one virtualization platform and it should come as no surprise that the hypervisor of choice is VMware.
BIG-IP® Local Traffic Manager™ (LTM) Virtual Edition (VE) takes your Application Delivery Network virtual. You get the agility you need to create a mobile, scalable, and adaptable infrastructure for virtualized applications. And like physical BIG‑IP devices, BIG-IP LTM VE is a full proxy between users and application servers, providing a layer of abstraction that secures, optimizes, and load balances application traffic.
Speaking of F5 and VMware, Why would you want F5 for VMware vSphere?
•F5 Management Plug-In for VMware vSphere
The F5 Management Plug-in simplifies common BIG-IP LTM administrative tasks in a vSphere environment, reduces the risk of error and enables basic automation.
•Integration with vCenter Server
Respond automatically to changes in the infrastructure with seamless integration between VMware and F5.
•Increased VM density by up to 60 percent
Free up server resources by offloading CPU-intensive operations to achieve maximum utilization and consolidation.
•Long-distance vMotion
Enable fully automated long-distance VMotion and Storage VMotion events between data centers without downtime or user disruption.
•Acceleration of VMotion and Storage VMotion
Accelerate VMotion events over the WAN up to 10x by compressing, deduplicating, and optimizing traffic.
•Other virtualization considerations with F5
File Virtualization
Infrastructure Virtualization
Server Virtualization
What about F5 and Cloud Benefits?
•Reduce Complexity
With a reusable framework of services that can be leveraged across static, dedicated servers as well as across multi-site cloud deployments, you immediately gain value that grows as your applications grow.
•Increased Control
By integrating traffic management, dynamic provisioning, access control, and management, you can more readily outsource the processing of applications and data without giving up ownership and control.
•Context Awareness
Having a complete picture of the user, network, application, and services gives you a unique ability to use context to determine how applications and data are delivered.
•Reduced Switching Costs
With a centrally controlled method of delivering applications and data, you can move resources anywhere at a moment’s notice without worrying about the capabilities of host locations.
This was a great session where I think I picked up the most information so far. F5 is one of those technologies I see a lot in the datacenter but I’ve not worked intimately with. I like their value-added integration with virtualization and adoption of a cloud vision.
Note : Tech Field Day is a sponsored event. Although I receive no direct compensation and take personal leave to attend, all event expenses are paid by the sponsors through Gestalt IT Media LLC. No editorial control is exerted over me and I write what I want, if I want, when I want, and how I want.
Gestalt IT was gracious enough to invite me back as a delegate for Tech Field Day Seattle which is happening… well… now, not to put too fine a point on it. I’m really excited about this opportunity! For the next two days, I’ll be at the Microsoft campus in Redmond, WA taking in vendor presentations and participating in peer discussions spanning a few different technology verticals.
We kicked things off tonight with dinner, discussion, and door prizes at Cedarbrook Lodge in Seatac, WA. There are a lot of new faces in this group of delegates. I don’t know most of the guys but that makes for a great opportunity to meet new people and network. In a word, Cedarbrook is gorgeous. It has more of a resort feel to it than a hotel. It’s too bad I won’t be spending more time here but the show must go on.
Tomorrow (Thursday), the other delegates and I will be meeting with Veeam, F5, and a stealth company which officially launches in our very presence tomorrow. I’m familiar with most of Veeam’s offerings but as a virtualization guy, I’m hoping to see more about their SureBackup technology. I’ve known of F5 for many years but just recently I’m seeing them push their way into the virtualization arena. Just last week they have expressed interest in participating in the Minneapolis VMUG. I’m anxious in seeing what value they bring to the virtualized datacenter. We cap off the day with a party at the Museum of Flight which should be really cool.
Moving into Friday, we’ll hear from Compellent on what they have been up to in the storage arena and how they are doing things differently than other storage vendors such as EMC, NetApp, Hitachi, HP, IBM, 3PAR, Dell, FalconStor, Pillar, etc. We’ll also be spending some time with NEC. I’m real curious as to what they are going to present. Talk about a diverse portfolio of products (as well as professional services). Whatever it is, I’ll be looking for virtualization relevance. Not only that, but will we see a landscape that continues to cater to cloud agility? Cloud has picked up a lot of momentum. It’s real. Adopt, adapt, integrate, or get run over by it. There may be one more vendor on Friday… that remains to be seen at this point. We end Friday with dinner in the evening and then some of us will start our journey back home.
I’m looking forward to a couple of great days.
Note : Tech Field Day is a sponsored event. Although I receive no direct compensation and take personal leave to attend, all event expenses are paid by the sponsors through Gestalt IT Media LLC. No editorial control is exerted over me and I write what I want, if I want, when I want, and how I want.
Sometimes I take things for granted. For instance, the health and integrity of the lab environment. Although it is “lab”, I do run some workloads which are key to keep online on a regular basis. Primarily the web server which this blog is served from, the email server which is where I do a lot of collaboration, and the Active Directory Domain Controllers/DNS Servers which provide the authentication mechanisms, mailbox access, external host name resolution to fetch resources on the internet, and internal host name resolution.
The workloads and infrastructure in my lab are 100% virtualized. The only “physical” items I have are type 1 hypervisor hosts, storage, and network. By this point I’ll assume most are familiar with the benefits of consolidation. The downside is that when the wheels come off in a highly consolidated environment, the impacts can be severe as they fan out and tip over down stream dependencies like dominos.
A few weeks ago I had decided to recarve the EMC Celerra fibre channel SAN storage. The VMs which were running on the EMC fibre channel block storage were all moved to NFS on the NetApp filer. Then last week, the Gb switch which supports all the infrastructure died. Yes it was a single point of failure – it’s a lab. The timing for that to happen couldn’t have been worse since all lab workloads were running on NFS storage. All VMs had lost their virtual storage and the NFS connections on the ESX(i) hosts eventually timed out.
The network switch was replaced later that day and since all VMs were down and NFS storage had disconnected, I took the opportunity to gracefully reboot the ESX(i) hosts; good time for a fresh start. Not surprised, I had to use the vSphere Client to connect to each host by IP address since at that point I had no functional DNS name resolution in the lab whatsoever. When the hosts came back online, I was about to begin powering up VMs, but instead I encountered a situation which I hadn’t planned for – all the VMs were grayed out, esentially disconnected. I discovered the cause of this was that after the host reboot, the NFS storage hadn’t come back online – both NetApp and EMC Celerra – on both hosts. There’s no way both storage cabinets and/or both hosts were having a problem at the same time so I assumed it was a network or cabling problem. With the NFS mounts in the vSphere client staring back at me in their disconnected state, it dawned on me – lack of DNS name resolution was preventing the hosts from connecting to the storage. The hosts could not resolve the FQDN name of the EMC Celerra or the NetApp filer storage. I modified /etc/hosts on each ESX(i) host, adding the TCP/IP address and FQDN for the NetApp filer and Celerra Data Movers. Shortly after I was back in business.
What did I learn? Not much. It was more a reiteration of important design considerations which I was already aware of:
Upgrading your virtual infrastructure to vSphere? Be sure to check out this handy reference from VMware: vSphere Upgrade Prerequisites Checklist. There are several areas which need to be considered and this document covers them all, including both requirements and recommendations. If you’re a consultant who visits new customer sites on a regular basis, it wouldn’t be a bad idea to bring this with to each engagement, or at least a condensed version of it.
If you’re reading this, that’s good news because it means last night’s P2V completed successfully. I took the last remaining non-virtualized physical infrastructure server in the lab and made it a virtual machine. Resource and role wise, this was the largest physical lab server next to the ESX hosts themselves.
Resources:
Roles:
I’m shutting down this last remaining physical server as well as the tape library. They’ll go in the pile of other physical assets which are already for sale or they will be donated as sales for 32-bit server hardware are slow right now. This is a milestone because this server, named SKYWALKER – you may have heard me mention it from time to time, has been a physical staple in the lab for as long as the lab has existed (circa 1995). Granted it has gone through several physical hardware platform migrations, its logical role is historic and its composition has always been physical. To put it into perspective, at one point in time SKYWALKER was a Compaq Prosignia 300 server with a Pentium Pro processor and a single internal Barracuda 4.3GB SCSI drive. Before my abilities to acquire server class hardware, it was hand-me-down whitebox parts from earlier gaming rigs.
The P2V (using VMware Converter) took a little over 5 hours for 500GB of storage. So the only physical servers remaining in the lab are the ESX hosts themselves. 2 DL385 G2s and 2 DL385s which typically remain powered down, earmarked for special projects. A successful P2V is a great start to a weekend if you ask me. Now I’m off to my daughter’s T-ball game. 
The NetApp filer in the lab recently encountered a failed disk. With the failed disk confirmed dead and removed, and the replacement disk added, I made my first attempt at replacing a failed disk in a NetApp filer.
fas3050clow*> disk assign 0a.29
disk 0a.29 (S/N 3HY0T1GG00007342W9NJ) is already owned by system cr2conffd03 (ID
84173417).
disk assign: Assign failed for one or more disks in the disk list.
Detour. The following parsed output confirmed this disk had ownership information from a previous filer in its DNA:
fas3050clow*> disk show -a
DISK OWNER POOL SERIAL NUMBER
———— ————- —– ————-
0a.29 cr2conffd03(84173417) Pool0 3HY0T1GG00007342W9NJ
Quick help from the community set me in the right direction. A few commands accomplished the required task:
fas3050clow*> priv set advanced
fas3050clow*> disk assign 0a.29 -s unowned -f
Note: Disks may be automatically assigned to this node, since option disk.auto_a
ssign is on.
fas3050clow*> disk assign 0a.29
Thu May 13 13:30:56 CDT [fas3050clow: diskown.changingOwner:info]: changing owne
rship for disk 0a.29 (S/N 3HY0T1GG00007342W9NJ) from unowned (ID -1) to fas3050c
low (ID 101175198)
Thu May 13 13:30:56 CDT [fas3050clow: HTTPPool00:warning]: HTTP XML Authenticati
on failed from 192.168.110.71.
fas3050clow*> Thu May 13 13:30:56 CDT [fas3050clow: diskown.RescanMessageFailed:
warning]: Could not send rescan message to fas3050clow. Please type disk show on
the console of fas3050clow for it to scan the newly inserted disks.
Thu May 13 13:30:56 CDT [fas3050clow: raid.assim.label.upgrade:info]: Upgrading
RAID labels.
Thu May 13 13:30:57 CDT [fas3050clow: disk.fw.downrevWarning:warning]: 1 disks h
ave downrev firmware that you need to update.
Thu May 13 13:31:00 CDT [fas3050clow: monitor.globalStatus.ok:info]: The system’
s global status is normal.
Shortly after, the firmware on the replacement disk was automatically upgraded:
Thu May 13 13:31:18 CDT [fas3050clow: dfu.firmwareDownloading:info]: Now downloa
ding firmware file /etc/disk_fw/X274_SCHT6146F10.NA16.LOD on 1 disk(s) of plex [
Pool0]…
I confirmed via NetApp System Manager (my GUI crutch), that the replaced disk is now a spare for the two aggregates configured on/owned by the head. I then updated the storage array spreadsheet I maintain which tracks disks, spares, arrays, luns, aggregates, volumes, exports, groups, pools, etc. for the various lab storage.
One additional item I learned from a NetApp Engineer is that spares are not to remain static. Rather, the role is designed to float around to different disks as failures can and will occur. This is a habit I’m learning to break which contradicts management of older storage arrays where spares instantiated to active duty were later deactivated when a failed disk was replaced.
As Erick Moore suggests in the comments, don’t forget to exit privileged mode when done:
fas3050clow*> priv set
Jason Langer, the spreadsheet is really nothing special. Merely a tool I use to keep track of the storage configurations. Following is a screenshot:
Quick tip if you use NetApp filer storage and you’d like to enable Deduplication (dedupe) and actually have it work as it was designed: Size the volumes and aggregates according to NetApp Deduplication for FAS and V-Series Deployment and Implementation Guide (TR-3505). What happens if you don’t provide enough breathing room for dedupe to run? In my experience, it runs for a second or two and completes successfully, but it does not deduplicate any data.
The deduplication metadata overhead space required boils down to a few variables: Data ONTAP version, volumes, and data within the volumes. All three make up the calculation. Specifically look at the tail end of section 3.3 on pages 17-18.
For Data ONTAP 7.3.x, which is what I have on the NetApp FAS 3050c filer, the following calculation applies:
1. Volume deduplication overhead – for each volume with deduplication enabled, up to 2% of the logical amount of data written to that volume will be required in order to store volume dedupe metadata. This is free space needed in the volume.
2. Aggregate deduplication overhead – for each aggregate that contains any volumes with dedupe enabled, up to 4% of the logical amount of data contained in all of those volumes with dedupe enabled will be required in order to store the aggregate dedupe metadata. This is free space needed in the aggregate.
An example used in the document:
If 100GB of data is to be deduplicated within a single volume, then there should be 2GB worth of available space within the volume and 4GB of space available within the aggregate.
Could be visualized as:
A second example with multiple volumes:
Consider a 2TB aggregate with 4 volumes each 400GB’s in size within the aggregate where three volumes are to be deduplicated, with 100GB of data, 200GB of data and 300GB of data respectively. The volumes will need 2GB, 4GB, and 6GB of space within the respective volumes; and, the aggregate will need a total of 24GB ((4% of 100GB) + (4% of 200GB) + (4%of 300GB) = 4+8+12 = 24GB) of space available within the aggregate.
Could be visualized as:
If you’ve got a filer in which to carve out some storage which needs to be deduplicated, you can go about the calculation from a few different directions.
I take full credit for the MS Excel diagrams above. Eat your heart out Hany Michael
Update 5/13/10: Here’s another item I stumbled on… Once dedupe completes, it may not reflect any savings in the “Space saved” field highlighted below. In my case, this occurred because the iSCSI LUN carved out of the volume was not thin provisioned.
Vaughn Stewart of NetApp explained it as follows:
With NetApp, thick provisioned LUNs reserve space in the FlexVol. In other words it is a storage accounting function and not a fully written out file (like a VMDK).
If data in the LUN is deduped, the savings cannot be displayed if the thick LUN reservation is in place. Toggle the LUN to thin and savings should magically appear.
There is absolutely no change in the data layout or performance with thick or thin LUNs (thus why you can toggle the function).
This was resolved by editing the LUN properties and unchecking the “Reserved” box, and then rerunning the deduplication process on the volume.
SANTA CLARA, CA – April 6, 2010 – Data Robotics, Inc., the company that delivers the best data storage experience, today introduced Drobo FS, a breakthrough Drobo designed for simple, expandable file sharing. By providing network file sharing capabilities along with automated data protection, the Drobo FS greatly simplifies shared storage for connected home, home office and small office users. Based on the revolutionary BeyondRAID technology and a flexible platform for adding features and capacity as needed, the Drobo FS can quickly and easily be customized and scaled to meet current or future storage requirements.
“Adding to Data Robotics’ offering of self-managing storage solutions, the Drobo FS offers users the ability to share data between computers quickly and easily,” said Liz Conner, research analyst for IDC’s storage systems and personal storage device & systems. “More than ever before, home users and small offices want to access and share a growing amount of data, but they don’t need a large, expensive system that requires specific expertise or extensive management. With the Drobo FS, Data Robotics addresses a need for which many Drobo users have been looking forward.”
Drobo FS extends automated data protection across connected systems using Data Robotics’ award-winning BeyondRAID virtualized storage technology. Drobo FS features a one-click toggle between single- and dual-drive redundancy and provides protection against up to two simultaneous drive failures. In addition, the Drobo FS enables users to add storage on-the-fly without ever losing access to data.
“We have been using Drobo solutions to store our data for almost three years and continue to be happy with their performance and simplicity. With four computers on our network, we also needed a solution that would allow us to share data between users. Typical shared storage devices were complicated or too expensive, so we’re thrilled that Data Robotics has created the Drobo FS,” said Seth Resnick, Co-Founder at D65. “Drobo FS comes with just the functionality we need, so it is easy to use while still providing the automated data protection that comes with BeyondRAID. The additional DroboApps allow us to add new features as we need them, which is really unique.”
Drobo FS Features and Benefits
- Plug In and Share – The Drobo FS connects directly to any Gigabit Ethernet network for a true plug in and share set-up experience. Supports standard data transfer protocols including Apple File Protocol (AFP) and Microsoft Common Internet File System (CIFS).
- 5-Drive Capacity and Instant Expansion to 10TB and Beyond – Customers with growing storage requirements can easily add data capacity by simply inserting a new hard drive or replacing the smallest drive with a larger one, even when all five drive bays are full. With Drobo FS, expansion is automatic, instantaneous, and access to data is always maintained.
- Single- and Dual-Drive Redundancy – The Drobo FS dual drive redundancy option protects against the simultaneous failure of up to two hard drives. Customers can engage this option with a single click without ever losing access to their data.
- Self-Healing Technology – With BeyondRAID, the Drobo FS continually examines data blocks and sectors on each drive to flag potential issues. The preemptive “scrubbing” helps ensure data is being written only to healthy drive areas and automatically keeps data in the safest state possible – even when a drive fails.
- Customizable Storage – Utilizing the growing library of DroboApps, including media and web applications, users can customize the Drobo FS to further enhance their sharing experience.
“This is the decade of being connected, no matter where you are or what kind of data you are storing. The Drobo FS was designed to best serve the needs of our customers with file sharing needs – from small offices and home offices to connected homes. By reducing the complexity of data sharing and providing a truly flexible platform for adding capacity and features, the Drobo FS is ideal for users that have the need to share ever increasing amounts of data,” said Tom Buiocchi, CEO of Data Robotics.
Pricing and Availability Drobo FS is currently available at a starting price of $699 MSRP, with multiple configurations to $1,449 for a 10TB (5x 2TB drives) bundle. Drobo FS is available now from authorized partners worldwide and on www.drobostore.com. To learn more about Drobo FS, please visit www.drobo.com/drobo-fs.
About Data Robotics
Data Robotics, Inc., the company that delivers the best storage experience ever, develops automated storage products designed to ensure data is always protected, accessible, and simple to manage. The award-winning Drobo storage arrays are the first to provide the protection of traditional RAID without the complexity. The revolutionary BeyondRAID technology frees users from making the difficult and confining choice of “Which RAID level to deploy?” by providing an unprecedented combination of advanced features and automation, including single- and dual-drive redundancy, instant expansion, self-monitoring, data awareness, self-healing, and an easy-to-understand visual status and alert panel. For more information, visit Data Robotics at www.datarobotics.com.
I spent some time with Data Robotics listening about the Drobo FS in addition to their existing storage offerings. Here are some of the things I learned about the FS:
MSRP Pricing:
| Product | Configuration | Pricing USD | Pricing GBP | Pricing EUR |
| Drobo FS Base | Base appliance, no drives | $699 | £469 | 519€ |
| Drobo FS 4.5TB | 4.5TB (3 x 1.5TB) | $999 | £669 | 749€ |
| Drobo FS 7.5TB | 7.5TB (5 x 1.5TB) | $1,149 | £769 | 859€ |
| Drobo FS 10TB | 10TB (5 x 2TB) | $1,449 | £969 | 1079€ |
The Drobo FS looks looks and sounds like a nice product. Watch for competitive pricing. If I’m able to get my hands on a unit, I’ll perform some lab testing and further writing I’m sure.
This is one of those “I’m documenting it for my own purposes” articles. Yes I read my own blog once in a while to find information on past topics. Here I’m basically copying a VMware KB article but I’ll provide a brief introduction.
So your wondering if you should use VMware Paravirtual SCSI? I’ve gotten this question a few times. PVSCSI is one of those technologies where “should I implement it” could be best answered with the infamous consulting reply “it depends”. One person asked if it would be good to use as a default configuration for all VMs. One notion that I would agree on by and large is that I feel the support complexity increases when using PVSCSI and it should only be used as needed for VMs which need an additional bit of performance squeezed from the disk subsystem. This is not a technology I would implement by default on all VMs. Dissecting the practical beneifts and ROI of implementing PVSCSI should be performed, but before that, your valuable time may be better spent finding out if your environment will support it to begin with. Have a look at VMware KB Article 1010398 which is where the following information comes from, verbatim.
It’s important to identify the guest VMs which support PVSCSI:
Paravirtual SCSI adapters are supported on the following guest operating systems:
It’s important to further identify more ambiguous type situations where PVSCSI may or may not not fit:
Paravirtual SCSI adapters also have the following limitations:
For more information on PVSCSI, including installation steps, see VMware KB Article 1010398. One more important thing to note is that in some operating system types, to install PVSCSI, you need to create a virtual machine with the LSI controller, install VMware Tools, then change to the drives to paravirtualized mode.
I recently set out to enable VMware ESX(i) 4 Round Robin load balancing with EMC Celerra (CLARiiON) fibre channel storage. Before I get to the details of how I did it, let me preface this discussion with a bit about how I interpret Celerra storage architecture.
The Celerra is built on CLARiiON fibre channel storage and as such, it leverages the benefits and successes CLARiiON has built over the years. I believe most CLARiiON’s are, by default, active/passive arrays from VMware’s perspective. Maybe more accurately stated, all controllers are active, however, each controller has sole ownership of a LUN or set of LUNs. If a host wants access to a LUN, it is preferable to go through the owning controller (the preferred path). Attempts to access a LUN through any other controller than the owning controller will result in a “Trespass” in EMC speak. A Trespass is shift in LUN ownership from one controller to another in order to service an I/O request from a fabric host. When I first saw Trespasses in Navisphere, I was alarmed. I soon learned that they aren’t all that bad in moderation. EMC reports that a Trespass occurs EXTREMELY quickly and in almost all cases will not cause problems. However, as with any array which adopts the LUN ownership model, stacking up enough I/O requests which force a race condition between controllers for LUN access, will cause a condition known as thrashing. Thrashing causes storage latency and queuing as controllers play tug of war for LUN access. This is why it is important for ESX hosts, which share LUN access, to consistently access LUNs via the same controller path.
As I said, the LUN ownership model above is the “out-of-box” configuration for the Celerra, also known as Failover Mode 1 in EMC Navisphere. The LUN path going through the owning controller will be the Active path from a VMware perspective. Other paths will be Standby. This is true for both MRU and Fixed path selection policies. What I needed to know was how to enable Round Robin path selection in VMware. Choosing Round Robin in the vSphere Client is easy enough, however, there’s more to it than that because the Celerra is still operating in Failover Mode 1 where I/O can only go through the owning controller.
So the first step in this process is to read the CLARiiON/VMware Applied Technology Guide which says I need to change the Failover Mode of the Celerra from 1 to 4 using Navisphere (FLARE release 28 version 04.28.000.5.704 or later may be required). A value of 4 tells the CLARiiON to switch to the ALUA (Asymmetric Logical Unit Access or Active/Active) mode. In this mode, the controller/LUN ownership model still exists, however, instead of transferring ownership of the LUN to the other controller with a Trespass, LUN access is allowed through the non-owning controller. The I/O is passed by the non-owning controller to the owning controller via the backplane and then to the LUN. In this configuration, both controllers are Active and can be used to access a LUN without causing ownership contention or thrashing. It’s worth mentioning right now that although both controllers are active, the Celerra will report to ESX the owning controller as the optimal path, and the non-owning controller as the non-optimal path. This information will be key a little later on. Each ESX host needs to be configured for Failover Mode 4 in Navisphere. The easiest way to do this is to run the Failover Setup Wizard. Repeat the process for each ESX host. One problem I ran into here is that after making the configuration change, each host and HBA still showed a Failover Mode of 1 in the Navisphere GUI. It was as if the Failover Setup Wizard steps were not persisting. I failed to accept this so I installed the Navisphere CLI and verified each host with the following command:
naviseccli -h <SPA_IP_ADDRESS> port -list –all
Output showed that Failover Mode 4 was configured:
Information about each HBA:
HBA UID: 20:00:00:00:C9:8F:C8:C4:10:00:00:00:C9:8F:C8:C4
Server Name: lando.boche.mcse
Server IP Address: 192.168.110.5
HBA Model Description:
HBA Vendor Description: VMware ESX 4.0.0
HBA Device Driver Name:
Information about each port of this HBA:�
SP Name: SP A
SP Port ID: 2
HBA Devicename: naa.50060160c4602f4a50060160c4602f4a
Trusted: NO
Logged In: YES
Source ID: 66560
Defined: YES
Initiator Type: 3
StorageGroup Name: DL385_G2
ArrayCommPath: 1
Failover mode: 4
Unit serial number: Array
Unfortunately, the CLARiiON/VMware Applied Technology Guide didn’t give me the remaining information I needed to actually get ALUA and Round Robin working. So I turned to social networking and my circle of VMware and EMC storage experts on Twitter. They put me on to the fact that I needed to configure SATP for VMW_SATP_ALUA_CX, something I wasn’t familiar with yet.
So the next step is a multistep procedure to configure the Pluggable Storage Architecture on the ESX hosts. More specifically, SATP (Storage Array Type Plugin) and the PSP (Path Selection Plugin), in that order. Duncan Epping provides a good foundation for PSA which can be learned here.
Configuring the SATP tells the PSA what type of array we’re using, and more accurately, what failover mode the array is running. In this case, I needed to configure the SATP for each LUN to VMW_SATP_ALUA_CX which is the EMC CLARiiON (CX series) running in ALUA mode (active/active failover mode 4). The command to do this must be issued on each ESX host in the cluster for each active/active LUN and is as follows:
#set SATP
esxcli nmp satp setconfig –config VMW_SATP_ALUA_CX –device naa.50060160c4602f4a50060160c4602f4a
esxcli nmp satp setconfig –config VMW_SATP_ALUA_CX –device naa.60060160ec242700be1a7ec7a208df11
esxcli nmp satp setconfig –config VMW_SATP_ALUA_CX –device naa.60060160ec242700bf1a7ec7a208df11
esxcli nmp satp setconfig –config VMW_SATP_ALUA_CX –device naa.60060160ec2427001cac9740a308df11
esxcli nmp satp setconfig –config VMW_SATP_ALUA_CX –device naa.60060160ec2427001dac9740a308df11
The devices you see above can be found in the vSphere Client when looking at the HBA devices discovered. You can also find devices with the following command on the ESX Service Console:
esxcli nmp device list
I found that changing the SATP requires a host reboot for the change to take effect (thank you Scott Lowe). After the host is rebooted, the same command used above should reflect that the SATP has been set correctly:
esxcli nmp device list
Results in:
naa.60060160ec2427001dac9740a308df11
Device Display Name: DGC Fibre Channel Disk (naa.60060160ec2427001dac9740a308df11)
Storage Array Type: VMW_SATP_ALUA_CX
Storage Array Type Device Config: {navireg=on, ipfilter=on}{implicit_support=on;explicit_ow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPat=0,numBytesPending=0}
Working Paths: vmhba1:C0:T0:L61
Once the SATP is set, it is time to configure the PSP for each LUN to Round Robin. You can do this via the vSphere Client, or you can issue the commands at the Service Console:
#set PSP per device
esxcli nmp psp setconfig –config VMW_PSP_RR –device naa.60060160ec242700be1a7ec7a208df11
esxcli nmp psp setconfig –config VMW_PSP_RR –device naa.60060160ec242700bf1a7ec7a208df11
esxcli nmp psp setconfig –config VMW_PSP_RR –device naa.60060160ec2427001cac9740a308df11
esxcli nmp psp setconfig –config VMW_PSP_RR –device naa.60060160ec2427001dac9740a308df11
#set PSP for device
esxcli nmp device setpolicy –psp VMW_PSP_RR –device naa.50060160c4602f4a50060160c4602f4a
esxcli nmp device setpolicy –psp VMW_PSP_RR –device naa.60060160ec242700be1a7ec7a208df11
esxcli nmp device setpolicy –psp VMW_PSP_RR –device naa.60060160ec242700bf1a7ec7a208df11
esxcli nmp device setpolicy –psp VMW_PSP_RR –device naa.60060160ec2427001cac9740a308df11
esxcli nmp device setpolicy –psp VMW_PSP_RR –device naa.60060160ec2427001dac9740a308df11
Once again, running the command:
esxcli nmp device list
Now results in:
naa.60060160ec2427001dac9740a308df11
Device Display Name: DGC Fibre Channel Disk (naa.60060160ec2427001dac9740a308df11)
Storage Array Type: VMW_SATP_ALUA_CX
Storage Array Type Device Config: {navireg=on, ipfilter=on}{implicit_support=on;explicit_ow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPat=0,numBytesPending=0}
Working Paths: vmhba1:C0:T0:L61
Notice the Path Selection Policy has now changed to Round Robin.
I’m good to go, right? Wrong. I struggled with this last bit for a while. Using ESXTOP and IOMETER, I could see that I/O was still only going down one path instead of two. Then I remembered something Duncan Epping had said to me in an earlier conversation a few days ago. He mentioned something about the array reporting optimal and non-optimal paths to the PSA. I printed out a copy of the Storage Path and Storage Plugin Management with esxcli document from VMware and took it to lunch with me. The answer was buried on page 88. The nmp roundrobin setting useANO is configured by default to 0 which means unoptimized paths reported by the array will not be included in Round Robin path selection unless optimized paths become unavailable. Remember I said early on that unoptimized and optimized paths reported by the array would be a key piece of information. We can see this in action by looking at the device list above. The very last line shows working paths, and only one path is listed for Round Robin use – the optimized path reported by the array. The fix here is to issue the following command, again on each host for all LUNs in the configuration:
#use non-optimal paths for Round Robin
esxcli nmp roundrobin setconfig –useANO 1 –device naa.50060160c4602f4a50060160c4602f4a
esxcli nmp roundrobin setconfig –useANO 1 –device naa.60060160ec242700be1a7ec7a208df11
esxcli nmp roundrobin setconfig –useANO 1 –device naa.60060160ec242700bf1a7ec7a208df11
esxcli nmp roundrobin setconfig –useANO 1 –device naa.60060160ec2427001cac9740a308df11
esxcli nmp roundrobin setconfig –useANO 1 –device naa.60060160ec2427001dac9740a308df11
Once again, running the command:
esxcli nmp device list
Now results in:
naa.60060160ec2427001dac9740a308df11
Device Display Name: DGC Fibre Channel Disk (naa.60060160ec2427001dac9740a308df11)
Storage Array Type: VMW_SATP_ALUA_CX
Storage Array Type Device Config: {navireg=on, ipfilter=on}{implicit_support=on;explicit_support=on;explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}
TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=1;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
Working Paths: vmhba0:C0:T0:L61, vmhba1:C0:T0:L61
Notice the change in useANO which now reflects a value of 1. In addition, I now have two Working Paths – an optimized path and an unoptimized path.
I fired up ESXTOP and IOMETER which now showed a flurry of I/O traversing both paths. I kid you not, it was a Clark Griswold moment when all the Christmas lights on the house finally worked.
So it took a while to figure this out but with some reading and the help of experts, I finally got it, and I was extremely jazzed. What would have helped was if VMware’s PSA was more plug and play with various array types. For instance, why can’t PSA recognize ALUA on the CLARiiON and automatically configure SATP for VMW_SATP_ALUA_CX? Why is a reboot required for an SATP change? PSA configuration in the vSphere client might have also been convenient but I recognize has diminishing returns or practical use with a large amount of hosts and/or LUNs to configure. Scripting and CLI is the way to go for consistency and automation reasons or how about PSA configuration via Host Profiles?
I felt a little betrayed and confused by the Navisphere GUI reflecting Failover Mode 1 after several attempts to change it to 4. I was looking at host connectivity status. Was I looking in the wrong place?
Lastly, end to end documentation on how to configure Round Robin would have helped a lot. EMC got me part of the way there with the CLARiiON/VMware Applied Technology Guide document, but left me hanging, making no mention of the PSA configuration needed. I’m getting that the end game for EMC multipathing today is PowerPath, which is fine – I’ll get to that, but I really wanted to do some testing with native Round Robin first, if for no other reason to establish a baseline to compare PowerPath to once I get there.
Thanks again to the people I leaned on to help me through this. It was the usual crew who can always be counted on.
The VMTN Storage Performance Thread is a collaboration of storage performance results on VMware virtual infrastructure provided by VMTN Community members around the world. The thread starts here, was locked due to length, and continues on in a new thread here. There’s even a Google Spreadsheet version, however, activity in that data repository appears to have diminished long ago. The spirit of the testing is outlined by thread creater and VMTN Virtuoso christianZ:
“My idea is to create an open thread with uniform tests whereby the results will be all inofficial and w/o any warranty. If anybody shouldn’t be agreed with some results then he can make own tests and presents his/her results too. I hope this way to classify the different systems and give a “neutral” performance comparison. Additionally I will mention that the performance [and cost] is one of many aspects to choose the right system.”
Testing standards are defined by christianZ so that results from each submission are consistent and comparable. A pre-defined template is used in conjunction with IOMETER to generate the disk I/O and capture the performance metrics. The test lab environment and the results are then appended to the thread discussion linked above. The performance metrics measured are:
Following are my results with the EMC Celerra NS-120 Unified Storage array:
SERVER TYPE: Windows Server 2003 R2 VM ON ESXi 4.0 U1
CPU TYPE / NUMBER: VCPU / 1 / 1GB Ram (thin provisioned)
HOST TYPE: HP DL385 G2, 16GB RAM; 2x QC AMD Opteron 2356 Barcelona
STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC Celerra NS-120 / 15x 146GB 15K 4Gb FC / RAID 5
SAN TYPE / HBAs: Emulex dual port 4Gb Fiber Channel, HP StorageWorks 2Gb SAN switch
OTHER: Disk.SchedNumReqOutstanding and HBA queue depth set to 64
Fibre Channel SAN Fabric Test
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 1.62 | 35,261.29 | 1,101.92 |
| Real Life – 60% Rand / 65% Read | 16.71 | 2,805.43 | 21.92 |
| Max Throughput – 50% Read | 5.93 | 10,028.25 | 313.38 |
| Random 8K – 70% Read | 11.08 | 3,700.69 | 28.91 |
swISCSI Test
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 17.52 | 3,426.00 | 107.06 |
| Real Life – 60% Rand / 65% Read | 14.33 | 3,584.53 | 28.00 |
| Max Throughput – 50% Read | 11.33 | 5,236.50 | 163.64 |
| Random 8K – 70% Read | 15.25 | 3,335.68 | 22.06 |
NFS Test
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 17.18 | 3,494.48 | 109.20 |
| Real Life – 60% Rand / 65% Read | 121.85 | 480.81 | 3.76 |
| Max Throughput – 50% Read | 12.77 | 4,718.29 | 147.45 |
| Random 8K – 70% Read | 123.41 | 478.17 | 3.74 |
Please read further below for futher NFS testing results after applying EMC Celerra best practices
Fibre Channel Summary
Not surprisingly, Celerra over SAN fabric beats the pants off of the shared storage solutions I’ve had in the lab previously, HP MSA1000 and Openfiler 2.2 swISCSI before that, in all four IOMETER categories. I was, however, pleasantly surprised to find that Celerra over fibre channel was one of the top performing configurations among a sea of HP EVA, Hitachi, NetApp, and EMC CX series frames.
swISCSI Summary
Celerra over swISCSIwas only slightly faster than the Openfiler 2.2 swISCSI on HP Proliant ML570 G2 hardware I had in the past on the Max Throughput-100%Read test. In the other three test categories, however, the Celerra left the Openfiler array in the dust.
NFS Summary
Moving on to Celerra over NFS, performance results were consistent with swISCSI in two test categories (Max Throughput-100%Read and Max Throughput-50%Read), but NFS performance numbers really dropped in the remaining two categories as compared to swISCSI (RealLife-60%Rand-65%Read and Random-8k-70%Read).
What’s worth noting is that both the iSCSI and NFS datastores are backed by the same logical Disk Group and physical disks on the Celerra. I did this purposely to compare the iSCSI and NFS protocols, with everything else being equal. The differences in two out of the four categories are obvious. The question came to mind: Does the performance difference come from the Celerra, the VMkernel, or a combination of both? Both iSCSI and NFS have evolved into viable protocols for production use in enterprise datacenters, therefore, I’m leaning AWAY from the theory that the performance degradation over NFS stems from the VMkernel. My initial conclusion here is that Celerra over NFS doesn’t perform as well with Random Read disk I/O patterns. I welcome your comments and experience here.
Please read further below for futher NFS testing results after applying EMC Celerra best practices
CIFS
Although I did not test CIFS, I would like to take a look at its performance. CIFS isn’t used directly by VMware virtual infrastructure, but it can be a handy protocol to leverage with NFS storage. File management (ie. .ISOs, templates, etc.) on ESX NFS volumes becomes easier and more mobile and less tools are required when the NFS volumes are presented as CIFS shares on a predominantly Windows client network. Providing adequate security through CIFS will be a must to protect the ESX datastore on NFS.
If you’re curious about storage array configuration and its impact on performance, cost, and availability, take a look at this RAID triangle which VMTN Master meistermn posted in one of the performance threads:
The Celerra stroage is currently carved out in the following way:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | ||
| DAE 2 | FC | FC | FC | FC | FC | FC | FC | FC | FC | FC | FC | FC | FC | FC | FC | DAE 2 |
| DAE1 | NAS | NAS | NAS | NAS | NAS | Spr | Spr | DAE 1 | ||||||||
| DAE 0 | Vlt | Vlt | Vlt | Vlt | Vlt | NAS | NAS | NAS | NAS | NAS | NAS | NAS | NAS | NAS | NAS | DAE 0 |
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 |
FC = fibre channel Disk Group
NAS = iSCSI/NFS Disk Groups
Spr = Hot Spare
Vlt = Celerra Valut drives
I’m very pleased with the Celerra NS-120. With the first batch of tests complete, I’m starting to formulate ideas on when, where, and how to use the various storage protocols with the Celerra. My goal is not to eliminate use of the slowest performing protocol in the lab. I want to work with each of them on a continual basis to test future design and integration with VMware virtual infrastructure.
Update 1/30/10: New NFS performance numbers. I’ve begun working with EMC vSpecialist to troubleshoot the performance descrepancies between swISCSI and NFS protocols. A few key things have been identified and a new set of performance metrics have been posted below after making some changes:
The new NFS performance tests are below, using some of the recommendations above:
New NFS Test After Enabling the NFS file system Uncached Write Mechanism
VMware ESX Using EMC Celerra Storage Systems pages 99-101
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 17.39 | 3,452.30 | 107.88 |
| Real Life – 60% Rand / 65% Read | 20.28 | 2,816.13 | 22.00 |
| Max Throughput – 50% Read | 19.43 | 3,051.72 | 95.37 |
| Random 8K – 70% Read | 19.21 | 2,878.05 | 22.48 |
New NFS Test After Configuring
NFS.SendBufferSize = 256 (this was set at the default of 264 which is not divisible by 32k)
NFS.ReceiveBufferSize = 128 (this was already at the default of 128)
VMware ESX Using EMC Celerra Storage Systems pages 73-74
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 17.41 | 3,449.05 | 107.78 |
| Real Life – 60% Rand / 65% Read | 20.41 | 2,807.66 | 21.93 |
| Max Throughput – 50% Read | 18.25 | 3,247.21 | 101.48 |
| Random 8K – 70% Read | 18.55 | 2,996.54 | 23.41 |
New NFS Test After Configuring IOMETER for 120 second Ramp Up Time
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 17.28 | 3,472.43 | 108.51 |
| Real Life – 60% Rand / 65% Read | 21.05 | 2,726.38 | 21.30 |
| Max Throughput – 50% Read | 17.73 | 3,338.72 | 104.34 |
| Random 8K – 70% Read | 17.70 | 3,091.17 | 24.15 |
Slight change
Due to the commentary received on the 120 second ramp up, I re-ran the swISCSI test to see if that changeded things much. To fairly compare protocol performance, the same parameters must be used across the board in the tests.
New swISCSI Test After Configuring IOMETER for 120 second Ramp Up Time
| Test Name | Avg. Response Time | Avg. I/O per Second | Avg. MB per Second |
| Max Throughput – 100% Read | 17.79 | 3,351.07 | 104.72 |
| Real Life – 60% Rand / 65% Read | 14.74 | 3,481.25 | 27.20 |
| Max Throughput – 50% Read | 12.17 | 4,707.39 | 147.11 |
| Random 8K – 70% Read | 15.02 | 3,403.39 | 26.59 |
swISCSI is still performing slightly better than NFS on the Random Reads, however, the margin is much closer
At this point I am content, stroke, happy, (borrowing UK terminology there) with NFS performance. I am now moving on to ALUA, Round Robin, and PowerPath/VE testing. I set up NPIV over the weekend with the Celerra as well – look for a blog post coming up on that.
Thank you EMC and to the folks who replied in the comments below with your help tackling best practices and NFS optimization/tuning!
I thought I’d post a lab update since John Troyer nudged me letting me know this week’s weekly podcast was focusing home labs for VCP and VCDX studies.
Read more here. Scroll down to the Lab Update section.
boche.net - VMware Virtualization Evangelist by Jason Boche is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.
© 2010 boche.net – VMware Virtualization Evangelist · Proudly powered by WordPress & Green Park 2 by Cordobo.
Valid XHTML 1.0 Transitional | Valid CSS 3
