Posts Tagged ‘vSphere’

IBM x3850 NICs Lose Network Connectivity With ESXi 4.0 Update 1

June 11th, 2011

This is a heads up on an issue I ran into some time ago upgrading to VMware ESXi 4.0 Update 1 on an IBM System x3850.  Granted, it’s an aging hardware platform and fast becoming a dated issue, nonetheless this information may help someone out of a late night or weekend fiasco.

Shortly after the upgrade, VMs began experiencing intermittent losses in network connectivity.  Tied to the problem, the following error was revealed in the ESXi log files:

WARNING: LinNet: netdev_watchdog: NETDEV WATCHDOG: vmnic7: transmit timed out

The root cause turned out to be a known issue with the e1000e driver on ESXi 4.0u1 and the IBM x3850.  The issue is documented well in VMware KB Article 1010313 (Intel 82571 NICs intermittently lose connectivity with ESX 4.x).  The KB article was updated last April and appears to still be giving VMware fits as it has spread to vSphere 4.1.  According to the KB article:

This issue may occur if the Message Signaled Interrupt (MSI) mode is enabled for the e1000e driver and this mode is not supported in a server platform. This driver supports these three interrupt modes:

  • 0.Legacy
  • 1.MSI
  • 2.MSI-X

ESX 4.0 added support for Message Signaled Interrupts in network and storage drivers. The default interrupt mode for the e1000e driver under ESX 4.x is MSI (1).

The workaround according to the KB article is to configure the e1000e driver to use Legacy (0) Interrupt mode (thus disabling MSI mode) by performing the following:

  1. Open a console to the ESX or ESXi host.
  2. To configure the e1000e module option IntMode and use Legacy (0) interrupts for a 4-port NIC, run the command:
    • esxcfg-module -s IntMode=0,0,0,0 e1000e
    • Note: A mode number must be specified for each NIC port. In case of 2 quad port NICs, specify the mode 0 for all 8 ports with the command:esxcfg-module -s IntMode=0,0,0,0,0,0,0,0 e1000e
  3. On ESX host, run this command to rebuild initrd:
    • esxcfg-boot -b
    • Note: This step is not applicable to ESXi hosts.
  4. Reboot the ESX/ESXi host for the changes to take effect.

Co-scheduling Visualized

May 21st, 2011

I stumbled onto this time lapse video of 51 airplanes taking off (and others taxiing) at Boston’s Logan International Airport.  One thought immediately popped into my mind: co-scheduling, which is a function of The VMware vSphere CPU Scheduler.  The accelerated speed of the video really pronounced the importance of precision the scheduler is responsible for, which in this case is the air traffic controller (or controllers).

httpv://www.youtube.com/watch?v=3k-xG8XX1EM

How does this video relate to co-scheduling?

  • Imagine the planes represent CPU execution (or more accurately CPU execution requests).
  • Imagine the various runways & taxiways represent the number of vCPUs in a VM.

The scheduler is responsible for managing the traffic, making sure there’s a clear path for each plane to move forward and to be on time. 

  • With less runways and taxiways (vCPUs in a VM), scheduling complexity is reduced.
  • Adding runways and taxiways (vCPUs in a VM) increases scheduling complexity but with a limited number of planes (guest OS CPU execution requests), scheduling will still be manageable and planes will arrive on time.
  • Now add a significant number of planes (4 vCPU, 8 vCPU) to our multitude of criss/crossing runways and taxiways.  The precision required to avoid accidents and maintain fairness becomes extremely complex.  The result is high %RDY time for VMs on the host.

How do we deal with scheduling complexity?

  1. Right size VMs whether they are new builds or P2V.  A minimalist approach to resource guarantees is the best place to start when we’re working with consolidated infrastructure and shared resources.
  2. If you’ve already right sized VMs and you’re running into high %RDY times:
    • Balance workloads by mixing VMs having both lower and higher number of vCPUs on the same host/cluster
    • Add cores to the host/cluster by:
      • Scaling up (increasing the core count in the host)
      • Scaling out (increasing the number of hosts in the cluster)

(Video source: @GuyKawasaki‘s Holy Kaw!)

Cisco Discovery Protocol (CDP) Tag Team

May 15th, 2011

For this blog post, I collaborated with Dawn Theirl (@KokopeIIi on Twitter) who is a Network Engineer in the San Francisco Bay Area.  Dawn performs a  lot of hands-on work in her day to day role as a wired and wireless network guru.  We understand that CDP provides benefits for both the network and virtualization platform teams.  However, in larger or siloed environments, our two teams don’t necessarily know what the other is seeing in their dashboard.  Curiosity prevailed and here we are.  In this writing, Dawn and I will discuss CDP, its implementation, and what exactly is seen in each of our siloed roles using our respective management tools, as well as the benefits provided by both having and sharing this information..

CDP is a useful troubleshooting tool in networking…. When given an IP of a host that someone has questions about and tracing the IP and MAC from a distribution layer switch down to the access layer, CDP info can tell you what switch to look at next. It is also useful if you don’t have an accurate network map to get an idea of how a network is physically laid out by learning what devices are physically connected to each other.  CDP operates at Layer 2 (Datalink) of the OSI model.  CDP packets are non-routable.

By default, CDP is enabled (and advertising) on Cisco switches and routers.  CDP is enabled and effectively configured as listen on ESX(i) vSwitches.  The value added by CDP benefits VMware administrators.  Looking at the CDP properties of each vmnic from the vSphere Client, CDP information is provided.  The most useful information is highlighted in yellow.  The name of the switch which the vmnic is cabled to as well as the port number on the switch that the network cable is connected to.  In access port configurations where 802.1Q VLANs are enabled, the VLAN field will also contain useful information:

SnagIt Capture

From the Cisco switch point of view in the default configuration, we don’t see any information about the ESXi host or its vmnics.  This is because the vSwitch tied to the vmnic uplinks is in listen mode only (no advertising).  # show cdp neighbors is the command which would display information about other devices advertising information by way of CDP:

SnagIt Capture

So out of the box, ESXi is configured to pull CDP information about the upstream network and this is quite valuable to have for implementation and troubleshooting.  However, there is an additional configuration which can be made on the ESXi host which will allow it to provide its own intrinsic data to the Cisco switch via CDP and that is by enabling CDP advertising.  This information is useful for troubleshooting which benefits both the network and virtual infrastructure teams by providing a method for close collaboration.  Let’s make the additional configuration change and note the additional information which is exposed by the ESXi host.

At the ESXi host DCUI, we can examine the CDP status of a vSwitch by issuing the command # esxcfg-vswitch -b vSwitch0.  Shown here, vSwitch0 is in listen only mode:

SnagIt Capture

Now let’s change the CDP mode for vSwitch0 to both (meaning both listen and advertise) and then verify the configuration change:

5-15-2011 11-30-24 AM

At this point, both the Cisco switch and the ESXi host are listening and advertising which is mutually beneficial to the network and virtual infrastructure teams.  Nothing changes visibility wise on the ESXi side.  However, the network team is now able to receive and view CDP advertisements on their Cisco gear from the ESXi hosts.  Let’s take a look by issuing the > show cdp neighbors command on the Cisco switch.  Note a difference from when I ran this command earlier that we can view CDP neighbor information in either user or privileged mode on the switch.  With CDP advertisements enabled on the ESXi host, we’re able to see ESXi host information as well as the host vmnic uplinks and the respective ports they’re cabled to on the Cisco switch:

5-15-2011 11-42-58 AM

From the switch side I can see what ports the VMs are on. This can be useful as unless you put a description on a port with the host name every time something gets installed (and then moved), you don’t know what is connected on any given port without a lot effort to backtrack a mac address to a IP to a hostname.  Lots of information… you get the host name, what port it’s connected to on the switch and which nic the host is using for that connection. Very useful for troubleshooting when a systems admin is questioning if there are problems on the network when a particular host is having issues. Usually the most the sys admin can tell you is what network the host is on and the network admin has to trace the IP and then the MAC address to find what port the host is on. With the CDP exchange once you narrow down what switch the host is on just issuing the “show CDP neighbor” command will tell you what port to focus on. One interesting note is the Host advertises itself as a switch instead of a host.

> show cdp neighbors detail provides some additional information about the host such as the build number and CDP version.  This detail is not quite as valuable for troubleshooting but nonetheless could come in handy for either a large enterprise or a smaller environment with consolidated roles:

5-15-2011 11-43-56 AM

Looking at the [advertised] Cisco Discovery Protocol output from the VM, important information seen is the switch name, IP address, vlan and port the host is connected to. Other things I can see are that the port is set to full duplex, and that it’s a switch vs. a router (don’t laugh, I’ve seen a router with a blade with a small number of ports used for a very small office.)

With the implementation details and benefits out of the way, let’s focus a bit on CDP strategy.  There are a few approaches to CDP which can be evaluated from labor, change management, and security primitives:

  1. Infrastructure implementation with default configurations – No changes required at implementation time providing the easiest and fastest deployment of ESXi in addition to providing CDP listen mode benefits from the virtual platform point of view.  The virtual platform remains secure while upstream network information is advertised to neighbors.
  2. Disable CDP globally, enable only as needed for the short term – Requires disabling CDP at implementation time in addition to change management time spent temporarily enabling and disabling CDP later on to aid troubleshooting.  Most secure from the network and virtual platform standpoint.
  3. Enable bidirectional CDP globally, always on – Requires enabling CDP both (listen and advertise) at implementation time thereby providing comprehensive information for troubleshooting later on.  Least secure; both network and virtual platform information is exposed by CDP advertisements to neighbors.

I’ve worked with organizations who implement one, of or a combination of all three.  As with many design decisions, philosophy and justifications will vary.  A decision here could be made based on the size of the datacenter, distribution of roles, security approach, or the vertical which the business operates in (think regulatory compliance).  CDP is of course beneficial to network and virtual platform owners but it can also aid a hacker who has penetrated the environment thereby becoming a sharing recipient of the same network information.  Speaking for myself, I’ve gotten a lot of operational benefits while leveraging CDP for troubleshooting.  Network engineers often ask me to configure CDP for advertising on the host side.  What helps them ultimately helps me in a troubleshooting scenario and can ultimately shorten the time we spend focusing on an issue.  In customer facing or production environments, every minute of downtime costs and therefore counts.  My preference is to operate with CDP configured for listen on the host side.  This configuration provides the most bang for the buck as it the default out-of-box configuration on both the Cisco and VMware side.  In other words, if you do nothing at all, you can reap major benefits with the native configuration when it comes time to troubleshoot or provide capacity and/or SPOF planning for network resources.  That’s my preference.  That said, I get the security side of the discussion and of course I’m not opposed to disabling CDP when compelling requirements or constraints exist.

Aside from the design decisions above, I would be remiss if I did not also mention a potential stability issue (categorize as potential risk in your design) I came across from Cisco. When enabling CDP or leaving CDP enabled in an environment, there is a known CDP issue which should be taken into consideration because it can cause a disruption of the network.  CDP Can Consume All Router Memory.  When a large amount of CDP neighbor announcements are sent, it is possible to consume all memory of an available device. This causes a crash or other abnormal behavior. Refer to Cisco’s Response to the CDP Issue (Document ID: 13621) for more details.  This issue is quite old and may no longer be a threat with modern versions of IOS and NX-OS.

CDP is wonderful tool.  However, one obvious weakness in the heterogeneous datacenter is that it is vendor specific to Cisco switches and routers.  Other networking vendors don’t support CDP and therefore cannot integrate with it.  A newer and similar vendor neutral protocol called LLDP (Link Layer Discovery Protocol) appears to fill the need for the other vendors which choose support it.  At this time however VMware is not supporting LLDP though at least one source claims it is on the VMware roadmap which is a good thing.

In closing, I’d like to leave the audience with an Appendix style list of VMware and Cisco CDP commands, as well as a few links to additional Cisco resources on the web.  I would also like to thank Dawn for her contribution and eager willingness to collaborate with me on this article.

Update 11/17/11: Link Layer Discovery Protocol (LLDP) has been published

Appendix A: ESX(i) esxcfg-vswitch (or vicfg-vswitch) parameters:

-B or –set-cdp Set the CDP status for a given virtual switch. To set, pass one of “down”, “listen”, “advertise”, “both”.
-b or –get-cdp Print the current CDP setting for this switch.

Appendix B: Cisco switch commands (some require privileged mode):

cdp run Enables CDP globally (on by default).
cdp enable Enables CDP on an interface.
cdp advertise-v2 Enables CDP Version-2 advertising functionality on a device.
clear cdp counters Resets the traffic counters to zero.
clear cdp table Deletes the CDP table of information about neighbors.
debug cdp adjacency Monitors CDP neighbor information.
show cdp Displays global CDP information such as the interval between transmissions of CDP advertisements, the number of seconds the CDP advertisement is valid for a given port, and the version of the advertisement.
show cdp neighbors  Displays information about neighbors.
show cdp neighbors detail  Displays more detail about neighboring devices.
show cdp entry * Displays information about all devices.
show cdp interface [type number] Displays information about interfaces on which CDP is enabled.
show cdp traffic Displays CDP counters, including the number of packets sent and received and checksum errors.
cdp timer seconds Specifies frequency of transmission of CDP updates.
cdp holdtime seconds Specifies the amount of time a receiving device should hold the information sent by your device before discarding it.
no cdp run Turns off CDP globally.

Appendix C: Helpful CDP resources from Cisco and VMware:

Configuring Cisco Discovery Protocol (CDP)

Configuring Cisco Discovery Protocol on Cisco Routers and Switches Running Cisco IOS (Document ID: 43485)

Cisco Discovery Protocol (CDP) network information

Configuring the Cisco Discovery Protocol (CDP) with ESX

Performance Overview charts fail with STATs Report Service internal error

May 11th, 2011

A few months ago I was troubleshooting a problem with the Overview charts in the Performance tab of the vSphere Client.  This was a vSphere 4.0 Update 1 environment but I believe the root cause will impact other vSphere versions as well.

Instead of displaying the dashboard of charts in the Overview display, an error was displayed:

STATs Report service internal error
or
STATs Report application initialization is not completed successfully

One unique aspect of this environment was that the vCenter database was hosted on a Microsoft SQL Server which used a port other than the default of TCP 1433.  VMware KB Article 1012812 identified this as the root cause of the issue.

To resolve the issue, I was required to stop the vCenter Server service and modify the statsreport.xml file located on the vCenter Server in the \Program Files\VMware\Infrastructure\tomcat\conf\Catalina\localhost\ directory by inserting the line in bold.  Note the italicized components will vary and are environment specific based on the SQL server name, database name, alternate TCP port in use, and authentication method (SQL/false or Windows integrated/true):

   name=”jdbc/StatsDS”
   type=”javax.sql.DataSource”
   factory=”org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory”
   initialSize=”3″
  maxActive=”10″
  maxIdle=”3″
  maxWait=”10000″
  defaultReadOnly=”true”
  defaultTransactionIsolation=”READ_COMMITTED”
  removeAbandoned=”true”
  removeAbandonedTimeout=”60″
  url=”jdbc:sqlserver://sqlservername:1601;instanceName=sqlservername;
     databaseName=sqldatabasename;integratedSecurity=false;”
/>

Don’t forget to restart the vCenter Server service after saving the statsreport.xml file.

VMware vSphere SiteSurvey Plug-in

May 10th, 2011

VMware SiteSurvey is a free add-on utility which analyzes vSphere ESX and ESXi hosts for VMware Fault Tolerance (FT) compatibility.  My good friend Eric Siebert wrote in depth about this piece of software and its capabilities just after the GA launch of VMware vSphere in 2009.

In June of 2010, VMware released SiteSurvey version 2.5.0.  What was unique about this particular release was that VMware transformed it from a standalone Windows application to a vSphere Client Plug-in.  Today, version 2.5.2 (released 12/10/10) of this SiteSurvey Plug-in is available as a free download from VMware’s site.

Installation of the plug-in is as simple as they come.  Exit the vSphere Client if it is currently running and launch the SiteSurvey-2.5.2.msi executable file.  SiteSurvey is a client side plug-in and as such needs to be installed on each machine which has a vSphere Client in order to use the plug-in.

Click Next:

SnagIt Capture

Accept the license agreement and click Next:

SnagIt Capture

Click Next:

SnagIt Capture

After the installation routine completes, click Close:

SnagIt Capture

Now open the vSphere Client and choose Plug-ins | Manage Plug-ins.  Note the new SiteSurvey Plugin and VMware’s inconsistent spelling of the Plug-in phrase:

SnagIt Capture

With the plug-in installed and enabled, you’ll now see a SiteSurvey tab in the cluster and host inventory views which will help you identify the FT capabilities of both hosts and virtual machines.  Remember, there is a lengthy list of requirements which must be met for hosts, VMs, clusters, and vCenter to enable FT.  Information about FT requirements can be found here, here, and here:

SnagIt Capture

StarWind/Mellanox Add Datacenter Storage Efficiency

May 7th, 2011

Press Release:

StarWind iSCSI SAN software coupled with Mellanox Ethernet-based high speed end-to-end infrastructure accelerates storage networking connectivity and lowers overall power consumption

StarWind SAN iSCSI software running over a Mellanox ConnectX®-2 40GE networking solution provides better performance, high availability (HA) and redundant iSCSI storage solutions at 40Gb/s bandwidth and high IOPS.

Burlington, Mass. – April 27, 2011StarWind Software, a global leader and a pioneer in SAN software for building iSCSI storage servers, today announced that StarWind SAN iSCSI software has achieved a record level of 27 Gb/s throughput and 350K IOPs running over a Mellanox ConnectX®-2 40GE networking solution.

StarWind has recently conducted the test under Mellanox Enterprise Datacenter’s initiative where the following testing configuration has been used: 3 servers of Zorro (HP DL380 G6 with 2*167GB disks, 24GB RAM, 8 cores); 6 Mellanox HCAs with 40Gb/s single port, 2 HCAs in each server; connected with 3 subnets, copper QSFP cables; OS: Windows 2008 Server R2.

Testing showed StarWind software has achieved record connectivity performance:

  • With the non-HA configuration (one node of the HA cluster and the client were performing I/O through the single 40Gb/s connection in both directions utilizing full-duplex Ethernet connection) the cluster got 25Gb/s of an iSCSI traffic (due to PCIe Gen 2 system limitations full 40Gb/s wire speed has not been reached). More than 300K IOPs at 25Gb/s have been achieved with 16 clients using StarWind iSCSI SAN software.
  • The full HA (two nodes: HA 1 and HA 2 processed requests served under Round-Robin) has achieved the same results as the non-HA configuration, but with more workers and deeper I/O queue.

In the near future 40Gb NIC will provide four times higher performance compared to 10Gb NIC for the same price.

“We thank Mellanox Technologies for helping StarWind iSCSI SAN software to continue setting new performance records. Just one year ago StarWind was used in the iSCSI performance tests by Microsoft and Intel, achieving a groundbreaking one million I/Os per second,” said StarWind’s CEO Artem Berman. “StarWind Technology Alliance Program builds relationships with best of breed technology companies to align technology, create leading solutions and help customers adopt our collective products.”

For more information on the test and performance benefits using StarWind iSCSI SAN software version 5.6 with Mellanox ConnectX-2 EN Adapters plus 10Gb/s or higher speeds (preferably 40Gb/s), for current and future x86 servers along with PCIe Gen2 and PCIe Gen 3-enabled systems, visit: http://www.mellanox.com/pdf/case_studies/CS_StarWind2.pdf

About Mellanox Technologies
Mellanox Technologies (NASDAQ:MLNX, TASE:MLNX) is a leading supplier of end-to-end InfiniBand and Ethernet connectivity solutions and services for servers and storage. Mellanox products optimize data center performance and deliver industry-leading bandwidth, scalability, power conservation and cost-effectiveness while converging multiple legacy network technologies into one future-proof architecture. The company offers innovative solutions that address a wide range of markets including HPC, enterprise, mega warehouse data centers, cloud computing, Internet and Web 2.0.

Founded in 1999, Mellanox Technologies is headquartered in Sunnyvale, California and Yokneam, Israel.

About StarWind Software Inc.
StarWind Software is a global leader in storage management and SAN software for small and midsize organizations. The flagship product of StarWind is SAN software that turns any industry-standard Windows Server into a fault-tolerant, fail-safe iSCSI SAN. StarWind iSCSI SAN is qualified for use with VMware, Hyper-V, XenServer, and also with Linux and Unix environments. StarWind Software is focused on providing small and midsize organizations with affordable, high availability storage technology that previously was only available in high-end storage hardware. Advanced enterprise-class features in StarWind include Automated HA Storage Node Failover and Failback (High Availability), Replication across a WAN, Snapshots, Thin Provisioning and Virtual Tape management.

Since 2003, StarWind has pioneered the iSCSI SAN software industry and is the solution of choice for over 30,000 customers worldwide in over 100 countries, from small and midsize companies to governments and Fortune 1000 companies.

For more information on StarWind Software Inc., visit: www.starwindsoftware.com

VMTurbo Free Performance and Efficiency Reporter

May 4th, 2011

Press Release:

VMTurbo Announces Availability of Free Performance and Efficiency Reporter

Reporting joins Monitoring application as a complimentary, useful productivity tool that provides valuable insight into virtual environments

Valhalla, NY, May 4, 2011 — VMTurbo, provider of software to analyze, optimize and control the virtualized data center, today announced free, immediate availability of VMTurbo Performance and Efficiency Reporter. Available for download at the VMTurbo site, VMTurbo Performance and Efficiency Reporter joins VMTurbo Monitoring as a complimentary, useful productivity tool that provides valuable insight into the virtual environment.

With its breakthrough real time workload management algorithm, VMTurbo assures the performance of the applications running in the virtualized environment while utilizing the underline infrastructure as efficiently as possible. VMTurbo “ties the viewing with the doing” by proactively providing corrective actions for any potential performance bottleneck as well as workload placement and configuration actions to maximize resources utilization. Furthermore, leveraging the same algorithm for planning, VMTurbo maximizes the ROI from the virtualized environment in real time and all the time.

“The real value of VMTurbo is its ability to automate and optimize the virtual environment in real time, so it makes sense to give away both our monitoring dashboard and performance reports, which are really just table setters for IT management,” said Shmuel Kliger, president and CEO, VMTurbo. “Now any organization can experience the benefits of VMTurbo in their own data center without having to spend a single dollar.”

The VMTurbo Performance and Efficiency Reporter package is a collection of reports that fall into two categories.

Performance Reports

  • Host/VM Utilization Heat Map contains an ordered ranking of utilization (UI) for both physical hosts as well as virtual machines. Raw data for each host and each VM also is provided. Useful metrics include the peak utilization. This data enables both an “at-a-glance” indicator of workload as well as useful information for provisioning.
  • Host Top CPU Ready Queue provides a detailed breakdown of the metric surrounding CPU ready states. By showing both the host total wait times (aggregated over a sample period) and the total times waited by virtual machines for each multiple of vCPUs, the report provides insights into how either a re-allocation of vCPUs or the re-location of a VM would impact this raw performance metric.
  • Storage Access IOPS delivers a raw ranked standing of data stores in the environment. By aggregating and displaying the raw average sustained IOPs to and from these datastores, users can evaluate the suitability of the datastore against the applied workload and potentially make a better allocation. Further, by looking at the total storage used (vs. capacity), users are better able to plan for anticipated growth.

Efficiency Reports

  • VM Over/Under Provisioning looks at the resources consumed (taking into account historical peaks) to make meaningful recommendations as to right-provisioning. This potentially frees up additional resources that could be re-allocated to improve performance or accommodate additional VMs.
  • Storage Wasted Allocations provides immediate visibility into data stored on the managed drives that is not associated with any VM. With this ranked data, users can quickly free up potentially vast amounts of unused storage at a considerable cost saving.
  • Storage Allocated to Dormant VMs both identifies dormant VMs and enables the reclamation of their disc space. This can yield significant cost savings through resource reclamation and the ability to subsequently host additional VMs on the same hardware.
  • VM Rightsizing Recommendation is based on configurable thresholds as well as analyses of specific time ranges for each VM to make intelligent configuration recommendations based on the actual resource demands of the VM. This ability to intelligently right size represents a significant efficiency improvement over other more wasteful allocation strategies.

Pricing and Availability

VMTurbo Performance and Efficiency Reporter is currently available for free download at http://www.vmturbo.com/downloads/performance-reporter/

Related Links

Find out more about the VMTurbo Performance and Efficiency Reporter at: http://www.vmturbo.com/products/performance-and-efficiency-reporter/

About VMTurbo

VMTurbo provides an integrated software suite for proactive and automated management of workload and resources in virtualized data centers. Only VMTurbo provides a holistic view of your virtual infrastructure as well as detailed action plans with respect to workload placement and resource allocation.  Our customers accomplish ever more, with less IT resources, by using our suite to analyze, optimize and control their virtual infrastructure.