Posts Tagged ‘ESXi’

Performance Overview charts fail with STATs Report Service internal error

May 11th, 2011

A few months ago I was troubleshooting a problem with the Overview charts in the Performance tab of the vSphere Client.  This was a vSphere 4.0 Update 1 environment but I believe the root cause will impact other vSphere versions as well.

Instead of displaying the dashboard of charts in the Overview display, an error was displayed:

STATs Report service internal error
or
STATs Report application initialization is not completed successfully

One unique aspect of this environment was that the vCenter database was hosted on a Microsoft SQL Server which used a port other than the default of TCP 1433.  VMware KB Article 1012812 identified this as the root cause of the issue.

To resolve the issue, I was required to stop the vCenter Server service and modify the statsreport.xml file located on the vCenter Server in the \Program Files\VMware\Infrastructure\tomcat\conf\Catalina\localhost\ directory by inserting the line in bold.  Note the italicized components will vary and are environment specific based on the SQL server name, database name, alternate TCP port in use, and authentication method (SQL/false or Windows integrated/true):

<Resource auth=”Container”
   name=”jdbc/StatsDS”
   type=”javax.sql.DataSource”
   factory=”org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory”
   initialSize=”3″
  maxActive=”10″
  maxIdle=”3″
  maxWait=”10000″
  defaultReadOnly=”true”
  defaultTransactionIsolation=”READ_COMMITTED”
  removeAbandoned=”true”
  removeAbandonedTimeout=”60″
  url=”jdbc:sqlserver://sqlservername:1601;instanceName=sqlservername;
     databaseName=sqldatabasename;integratedSecurity=false;”
/>

Don’t forget to restart the vCenter Server service after saving the statsreport.xml file.

VMware vSphere SiteSurvey Plug-in

May 10th, 2011

VMware SiteSurvey is a free add-on utility which analyzes vSphere ESX and ESXi hosts for VMware Fault Tolerance (FT) compatibility.  My good friend Eric Siebert wrote in depth about this piece of software and its capabilities just after the GA launch of VMware vSphere in 2009.

In June of 2010, VMware released SiteSurvey version 2.5.0.  What was unique about this particular release was that VMware transformed it from a standalone Windows application to a vSphere Client Plug-in.  Today, version 2.5.2 (released 12/10/10) of this SiteSurvey Plug-in is available as a free download from VMware’s site.

Installation of the plug-in is as simple as they come.  Exit the vSphere Client if it is currently running and launch the SiteSurvey-2.5.2.msi executable file.  SiteSurvey is a client side plug-in and as such needs to be installed on each machine which has a vSphere Client in order to use the plug-in.

Click Next:

SnagIt Capture

Accept the license agreement and click Next:

SnagIt Capture

Click Next:

SnagIt Capture

After the installation routine completes, click Close:

SnagIt Capture

Now open the vSphere Client and choose Plug-ins | Manage Plug-ins.  Note the new SiteSurvey Plugin and VMware’s inconsistent spelling of the Plug-in phrase:

SnagIt Capture

With the plug-in installed and enabled, you’ll now see a SiteSurvey tab in the cluster and host inventory views which will help you identify the FT capabilities of both hosts and virtual machines.  Remember, there is a lengthy list of requirements which must be met for hosts, VMs, clusters, and vCenter to enable FT.  Information about FT requirements can be found here, here, and here:

SnagIt Capture

StarWind/Mellanox Add Datacenter Storage Efficiency

May 7th, 2011

Press Release:

StarWind iSCSI SAN software coupled with Mellanox Ethernet-based high speed end-to-end infrastructure accelerates storage networking connectivity and lowers overall power consumption

StarWind SAN iSCSI software running over a Mellanox ConnectX®-2 40GE networking solution provides better performance, high availability (HA) and redundant iSCSI storage solutions at 40Gb/s bandwidth and high IOPS.

Burlington, Mass. – April 27, 2011StarWind Software, a global leader and a pioneer in SAN software for building iSCSI storage servers, today announced that StarWind SAN iSCSI software has achieved a record level of 27 Gb/s throughput and 350K IOPs running over a Mellanox ConnectX®-2 40GE networking solution.

StarWind has recently conducted the test under Mellanox Enterprise Datacenter’s initiative where the following testing configuration has been used: 3 servers of Zorro (HP DL380 G6 with 2*167GB disks, 24GB RAM, 8 cores); 6 Mellanox HCAs with 40Gb/s single port, 2 HCAs in each server; connected with 3 subnets, copper QSFP cables; OS: Windows 2008 Server R2.

Testing showed StarWind software has achieved record connectivity performance:

  • With the non-HA configuration (one node of the HA cluster and the client were performing I/O through the single 40Gb/s connection in both directions utilizing full-duplex Ethernet connection) the cluster got 25Gb/s of an iSCSI traffic (due to PCIe Gen 2 system limitations full 40Gb/s wire speed has not been reached). More than 300K IOPs at 25Gb/s have been achieved with 16 clients using StarWind iSCSI SAN software.
  • The full HA (two nodes: HA 1 and HA 2 processed requests served under Round-Robin) has achieved the same results as the non-HA configuration, but with more workers and deeper I/O queue.

In the near future 40Gb NIC will provide four times higher performance compared to 10Gb NIC for the same price.

“We thank Mellanox Technologies for helping StarWind iSCSI SAN software to continue setting new performance records. Just one year ago StarWind was used in the iSCSI performance tests by Microsoft and Intel, achieving a groundbreaking one million I/Os per second,” said StarWind’s CEO Artem Berman. “StarWind Technology Alliance Program builds relationships with best of breed technology companies to align technology, create leading solutions and help customers adopt our collective products.”

For more information on the test and performance benefits using StarWind iSCSI SAN software version 5.6 with Mellanox ConnectX-2 EN Adapters plus 10Gb/s or higher speeds (preferably 40Gb/s), for current and future x86 servers along with PCIe Gen2 and PCIe Gen 3-enabled systems, visit: http://www.mellanox.com/pdf/case_studies/CS_StarWind2.pdf

About Mellanox Technologies
Mellanox Technologies (NASDAQ:MLNX, TASE:MLNX) is a leading supplier of end-to-end InfiniBand and Ethernet connectivity solutions and services for servers and storage. Mellanox products optimize data center performance and deliver industry-leading bandwidth, scalability, power conservation and cost-effectiveness while converging multiple legacy network technologies into one future-proof architecture. The company offers innovative solutions that address a wide range of markets including HPC, enterprise, mega warehouse data centers, cloud computing, Internet and Web 2.0.

Founded in 1999, Mellanox Technologies is headquartered in Sunnyvale, California and Yokneam, Israel.

About StarWind Software Inc.
StarWind Software is a global leader in storage management and SAN software for small and midsize organizations. The flagship product of StarWind is SAN software that turns any industry-standard Windows Server into a fault-tolerant, fail-safe iSCSI SAN. StarWind iSCSI SAN is qualified for use with VMware, Hyper-V, XenServer, and also with Linux and Unix environments. StarWind Software is focused on providing small and midsize organizations with affordable, high availability storage technology that previously was only available in high-end storage hardware. Advanced enterprise-class features in StarWind include Automated HA Storage Node Failover and Failback (High Availability), Replication across a WAN, Snapshots, Thin Provisioning and Virtual Tape management.

Since 2003, StarWind has pioneered the iSCSI SAN software industry and is the solution of choice for over 30,000 customers worldwide in over 100 countries, from small and midsize companies to governments and Fortune 1000 companies.

For more information on StarWind Software Inc., visit: www.starwindsoftware.com

VMTurbo Free Performance and Efficiency Reporter

May 4th, 2011

Press Release:

VMTurbo Announces Availability of Free Performance and Efficiency Reporter

Reporting joins Monitoring application as a complimentary, useful productivity tool that provides valuable insight into virtual environments

Valhalla, NY, May 4, 2011 — VMTurbo, provider of software to analyze, optimize and control the virtualized data center, today announced free, immediate availability of VMTurbo Performance and Efficiency Reporter. Available for download at the VMTurbo site, VMTurbo Performance and Efficiency Reporter joins VMTurbo Monitoring as a complimentary, useful productivity tool that provides valuable insight into the virtual environment.

With its breakthrough real time workload management algorithm, VMTurbo assures the performance of the applications running in the virtualized environment while utilizing the underline infrastructure as efficiently as possible. VMTurbo “ties the viewing with the doing” by proactively providing corrective actions for any potential performance bottleneck as well as workload placement and configuration actions to maximize resources utilization. Furthermore, leveraging the same algorithm for planning, VMTurbo maximizes the ROI from the virtualized environment in real time and all the time.

“The real value of VMTurbo is its ability to automate and optimize the virtual environment in real time, so it makes sense to give away both our monitoring dashboard and performance reports, which are really just table setters for IT management,” said Shmuel Kliger, president and CEO, VMTurbo. “Now any organization can experience the benefits of VMTurbo in their own data center without having to spend a single dollar.”

The VMTurbo Performance and Efficiency Reporter package is a collection of reports that fall into two categories.

Performance Reports

  • Host/VM Utilization Heat Map contains an ordered ranking of utilization (UI) for both physical hosts as well as virtual machines. Raw data for each host and each VM also is provided. Useful metrics include the peak utilization. This data enables both an “at-a-glance” indicator of workload as well as useful information for provisioning.
  • Host Top CPU Ready Queue provides a detailed breakdown of the metric surrounding CPU ready states. By showing both the host total wait times (aggregated over a sample period) and the total times waited by virtual machines for each multiple of vCPUs, the report provides insights into how either a re-allocation of vCPUs or the re-location of a VM would impact this raw performance metric.
  • Storage Access IOPS delivers a raw ranked standing of data stores in the environment. By aggregating and displaying the raw average sustained IOPs to and from these datastores, users can evaluate the suitability of the datastore against the applied workload and potentially make a better allocation. Further, by looking at the total storage used (vs. capacity), users are better able to plan for anticipated growth.

Efficiency Reports

  • VM Over/Under Provisioning looks at the resources consumed (taking into account historical peaks) to make meaningful recommendations as to right-provisioning. This potentially frees up additional resources that could be re-allocated to improve performance or accommodate additional VMs.
  • Storage Wasted Allocations provides immediate visibility into data stored on the managed drives that is not associated with any VM. With this ranked data, users can quickly free up potentially vast amounts of unused storage at a considerable cost saving.
  • Storage Allocated to Dormant VMs both identifies dormant VMs and enables the reclamation of their disc space. This can yield significant cost savings through resource reclamation and the ability to subsequently host additional VMs on the same hardware.
  • VM Rightsizing Recommendation is based on configurable thresholds as well as analyses of specific time ranges for each VM to make intelligent configuration recommendations based on the actual resource demands of the VM. This ability to intelligently right size represents a significant efficiency improvement over other more wasteful allocation strategies.

Pricing and Availability

VMTurbo Performance and Efficiency Reporter is currently available for free download at http://www.vmturbo.com/downloads/performance-reporter/

Related Links

Find out more about the VMTurbo Performance and Efficiency Reporter at: http://www.vmturbo.com/products/performance-and-efficiency-reporter/

About VMTurbo

VMTurbo provides an integrated software suite for proactive and automated management of workload and resources in virtualized data centers. Only VMTurbo provides a holistic view of your virtual infrastructure as well as detailed action plans with respect to workload placement and resource allocation.  Our customers accomplish ever more, with less IT resources, by using our suite to analyze, optimize and control their virtual infrastructure. 

Product Review: Veeam Backup & Replication v5

April 21st, 2011

Do you like free?  Do you like backup and replication?  Do you like VMware?  If you answered “yes” to any of the three, then you might like this:  I wrote a product review on Veeam Backup and Replication 5 which discusses the following:

Pros and Cons of different approaches to data protection

An in-depth look at Veeam Backup & Replication v5

What Veeam Backup & Replication v5 is missing

Register for your free download:

Product Review: Veeam Backup & Replication v5



Sound familiar? It should.  This isn’t the first time I’ve written about Veeam.  Check out a few of my previous posts about Veeam Backup & Replication:

Veeam Backup & Replication 5.0

Gestalt IT Tech Field Day – Veeam

HDS and VAAI Integration

April 3rd, 2011

SnagIt CaptureOn day 1 of Hitachi Data Systems Geek Day 2.0, we met with Michael Heffernan, Global Product Manager – Virtualization.  You might know him as @virtualheff on Twitter.  I was pleased to listen to Heff as he discussed HDS integration with VMware vSphere vStorage API for Array Integration (VAAI for short and most easily pronounced “vee·double-ehh·eye”).  For those who aren’t aware, VMware introduced VAAI with the GA release of vSphere 4.1 on July 13th of last year.  In short, VAAI allows the burden of certain storage related tasks to be offloaded from the ESX/ESXi hypervisor to the storage array.  Generally speaking, the advantages touted are performance improvement of intrinsic tasks and increased scalability of the storage array. HDS is one of a few storage vendors who supported VAAI integration on the July launch date and in February of this year, they announced VAAI support with their VSP (see also Hu Yoshida’s writing on the announcement).

Heff started off with some virtualization in the datacenter background and IDC stats.  Here are a few that he shared with us:

  • Only 12.8% of all physical servers are virtualized in 2009
  • More than half of all workloads (51%) will be virtualized by the end of 2010
  • Two-thirds (69%) by 2013
  • VM densities continue to rise predictably, averaging:
    • 6 VMs per physical server in 2009
    • 8.5 VMs per physical server in 2013

A few time line infographics were also shown which tell a short story about VMware, HDS:

 SnagIt Capture   SnagIt Capture

VMware provides several integration points which storage vendors can take advantage of, VAAI being just one of them.  These integration points are use case specific and standardized by VMware.  As such, integration is developed in parallel by competing vendors and most often the resulting offerings from each look and feel similar.  Great minds in storage and virtualization think alike.

SnagIt Capture   SnagIt Capture

SnagIt CaptureHDS integrates with all three VAAI attach points VMware offers:

  1. Hardware Assisted Copy
  2. Hardware Assisted Zeroing
  3. Hardware Assisted Locking

Heff also used this opportunity to mention Hitachi Dynamic Provisioning (HDP) technology which is essentially HDS thin provisioning plus other lesser known benefits but has nothing more to do with VAAI than any other storage vendor which supports both VAAI and thin provisioning.  Others may disagree but I see no sustainable or realizable real world benefit with VAAI and thin provisioning at this time; the discussion is rather academic.

HDS went on to show VAAI benefits are real.  Tests show an 18% efficiency improvement in the block copy test on a 30GB virtual disk.  85% decrease in elapsed time to eager write zeros to a 30GB virtual disk.  The third VAAI benefit, hardware assisted locking, can be a little trickier to prove or require specific use cases.  Following are examples of VMFS operations that require locking metadata, and as a result a SCSI reservation which hardware assisted locking improves, per VMware KB Article: 1005009:

  • Creating a VMFS datastore
  • Expanding a VMFS datastore onto additional extents
  • Powering on a virtual machine
  • Acquiring a lock on a file
  • Creating or deleting a file
  • Creating a template
  • Deploying a virtual machine from a template
  • Creating a new virtual machine
  • Migrating a virtual machine with VMotion
  • Growing a file, for example, a Snapshot file or a thin provisioned Virtual Disk

Heff showcased the following hardware assisted locking results.  Up to 36% increase in performance and 75% reduction in lock conflicts for the power on/linked clone test:

SnagIt Capture   SnagIt Capture

SnagIt Capture

VAAI offloads some of the heavy lifting from the hypervisor to the back end storage array so it was appropriate for the discussion to ultimately lead to impact on the array.  This is where I currently feel the bigger benefit is: better scalability or more mileage out of the array.  HDS is also the second storage vendor I’ve heard say that block LUN size and number of VMs per LUN is no longer a constraint (from a performance standpoint, everything else being equal).  This point always interests me and is frankly a tough pill to swallow.  I wasn’t able to pin Heff down to more specific details nor have I seen actual numbers, case studies, or endorsements from any storage vendor’s customer environments.  To some degree, I think this design consideration is still going to be use case and environment dependent.  It will also continue to be influenced by other constraints such as replication.  It may become more of a reality when VMware expands VAAI integration beyond the original three features.  HDS did mention that in vSphere 5, VMware is adding two more VAAI features bringing the total to five assuming they are released.

HDS offers competitive storage solutions for the VMware use case and it is clear they are totally committed to the virtualization push from both a storage and compute perspective.  You can learn more about these solutions and stay in tune with their evolution at their VMware Solutions site.

Full Disclosure Statement: HDS Geek Day is a sponsored event. Although I receive no direct compensation and take personal leave to attend, all event expenses are paid by the sponsors. No editorial control is exerted over me and I write what I want, if I want, when I want, and how I want.

EMC Celerra BETA Patch Pumps Up the NFS Volume

March 21st, 2011

A while back, Chad Sakac of EMC announced on his blog that he is looking for customers to volunteer their storage arrays to run various performance tests in addition to a piece of NFS specific BETA code for DART.  Having installed the BETA code (which I’m told is basically a nas executable swap in), I proceeded to compare NFS performance results with baseline results I had captured pre-patch.  In most test cases, the improvements ranged from significant to over twice the performance gain.  Most of the performance gains appear to surround write I/O.

Following are the results comparing NFS performance with four different workload types before BETA patch and after BETA patch on a Celerra NS-120 with 15 x 15k spindles:

SnagIt Capture

Detailed supporting data.  Keep in mind the NFS patch is still BETA with no firm release date as of yet from EMC:

SnagIt Capture

This looks like great stuff from EMC and assuming the code reaches GA status, it would bolster the design choice of NFS in the datacenter.  Chad may still be looking test results for certain use cases.  If you’re interested in participating in the tests with your EMC array, please reach out to Chad using the comments section in Chad’s blog post linked above.