Posts Tagged ‘Networking’

vCenter MAC Address allocation and conflicts

November 3rd, 2010

Paul Davey of Xtravirt published a VMware networking article a few weeks ago called vCenter MAC Address allocation and conflicts.  The article describes the mechanism behind MAC address assignments in vCenter, and more specifically how conflicts are avoided:

When a vCenter server is installed a unique ID is generated. This ID is randomly generated and is in the range of 0 to 64. The ID gets used when generating MAC address and the UUIDS, or unique identifiers for virtual machines. You can see that if two vCenter servers had the same unique ID, a possibility exists that duplicate MAC addresses might get generated; cue packet loss, connectivity issues and your desk phone ringing a lot…

 SnagIt Capture

I receive email updates from Xtravirt at regular intervals – about one every week or two.  Each update contains new virtualization content as well as links to their most popular content.  I find the content very interesting and always look forward to opening new email from them.  I think this speaks volumes considering how much of a chore email can be at times.  If this sounds appealing to you, head to their site and look at the bottom of the page to sign up for the Xtravirt Newsletter.   No purchase or invitation necessary.

Updated 3/14/11: I thought this might also be helpful for this article.  VMware explains the automatic MAC address generation process as follows:

The VMware Universally Unique Identifier (UUID) generates MAC addresses that are checked for conflicts. The generated MAC addresses are created by using three parts: the VMware OUI, the SMBIOS UUID for the physical ESXi machine, and a hash based on the name of the entity that the MAC address is being generated for.

NFS and Name Resolution

June 11th, 2010

Sometimes I take things for granted.  For instance, the health and integrity of the lab environment.  Although it is “lab”, I do run some workloads which are key to keep online on a regular basis. Primarily the web server which this blog is served from, the email server which is where I do a lot of collaboration, and the Active Directory Domain Controllers/DNS Servers which provide the authentication mechanisms, mailbox access, external host name resolution to fetch resources on the internet, and internal host name resolution.

The workloads and infrastructure in my lab are 100% virtualized.  The only “physical” items I have are type 1 hypervisor hosts, storage, and network.  By this point I’ll assume most are familiar with the benefits of consolidation.  The downside is that when the wheels come off in a highly consolidated environment, the impacts can be severe as they fan out and tip over down stream dependencies like dominos.

A few weeks ago I had decided to recarve the EMC Celerra fibre channel SAN storage.  The VMs which were running on the EMC fibre channel block storage were all moved to NFS on the NetApp filer.  Then last week, the Gb switch which supports all the infrastructure died.  Yes it was a single point of failure – it’s a lab.  The timing for that to happen couldn’t have been worse since all lab workloads were running on NFS storage.  All VMs had lost their virtual storage and the NFS connections on the ESX(i) hosts eventually timed out.

The network switch was replaced later that day and since all VMs were down and NFS storage had disconnected, I took the opportunity to gracefully reboot the ESX(i) hosts; good time for a fresh start.  Not surprised, I had to use the vSphere Client to connect to each host by IP address since at that point I had no functional DNS name resolution in the lab whatsoever. When the hosts came back online, I was about to begin powering up VMs, but instead I encountered a situation which I hadn’t planned for – all the VMs were grayed out, esentially disconnected.  I discovered the cause of this was that after the host reboot, the NFS storage hadn’t come back online – both NetApp and EMC Celerra – on both hosts.  There’s no way both storage cabinets and/or both hosts were having a problem at the same time so I assumed it was a network or cabling problem. With the NFS mounts in the vSphere client staring back at me in their disconnected state, it dawned on me – lack of DNS name resolution was preventing the hosts from connecting to the storage.  The hosts could not resolve the FQDN name of the EMC Celerra or the NetApp filer storage.  I modified /etc/hosts on each ESX(i) host, adding the TCP/IP address and FQDN for the NetApp filer and Celerra Data Movers.  Shortly after I was back in business.

What did I learn?  Not much.  It was more a reiteration of important design considerations which I was already aware of:

  1. 100% virtualization/consolidation is great – when it works.  The web of upstream/downstream dependencies makes it a pain when something breaks.  Consolidated dependencies which you might consider leaving physical or placing in a separate failure domain:
    • vCenter Management
    • Update Manager
    • SQL/Oracle back ends
    • Name Resolution (DNS/WINS)
    • DHCP
    • Routing
    • Active Directory/FSMO Roles/LDAP/Authentication/Certification Authorities
    • Mail
    • Internet connectivity
  2. Hardware redundancy is always key but expensive.  Perform a risk assessment and make a decision based on the cost effectiveness.
  3. When available, diversify virtualized workload locations to reduce failure domain, particularly to split workloads which provide redundant infrastructure support such as Active Directory Domain Controllers, DNS servers.  This can mean placing workloads on separate hosts, separate clusters, separate datastores, separate storage units, maybe even separate networks depending on the environment.
  4. Static entires in /etc/hosts isn’t a bad idea as a fallback if you plan on using NFS in an environment with unreliable DNS but I think the better point to discuss is the risk and pain which will be realized in deploying virtual infrastructure in an unreliable environment. Garbage In – Garbage Out.  I’m not a big fan of using IP addresses to mount NFS storage unless the environment is small enough.

Win A Free VMworld Pass From

June 6th, 2010

6-6-2010 12-31-26 PMThe economy has been rough.  Individuals and businesses have felt the impacts in various ways.  Reduction of income or revenue.  Increased operational expenses.  Reduction in valuation of homes or assets.  Downsizing of staff.  The slashing of budgets, including training, conferences, and travel.  Those who are in verticals which tail economic trends by a year or two will begin feeling the impacts soon.

As a reader of this blog, you already know VMworld 2010 in San Francisco is just a few months away.  If you’re like me, you’re wondering “How am I going to get there this year?”  Due to the reasons I’ve outlined above, details are sketchy on whether or not you’ll get to go.  Management says “Ask again in August, we’ll let you know.”  It doesn’t sound promising.  Or maybe you’ve already been told “It’s just too expensive given the economy, sorry.” would like to help.  If you can get yourself to the door of the Moscone Center, will get you in.  This is a $1,895 value if you were to purchase a conference pass at the door.  There is no purchase necessary for this contest other than your own T&E (transportation, hotel, van down by the river, etc.)  On Friday June 18th, 2010, one random and lucky winner who has followed the contest rules completely (detailed below) will be revealed.

The intent here is not to save a company money.  Rather, to make the difference between someone going to VMworld versus not going.  Therefore, I would appreciate it if entries would be limited to those who do not already have budget approval for the VMworld conference pass.  At the same time, should you win, you owe it to yourself and the other contestants to follow through and attend the conference.  It would be a shame for the pass to go to waste.  Perhaps another blogger or vendor would like to co-sponsor airfare or hotel for the winner.  Consider this an open invitation for co-sponsorship.

Be sure to read the VMworld 2010 FAQ so that you thoroughly understand the conference logistics, ensuring you are an eligible candidate to attend.

Update 6/6/10: I’m happy to announce that Gestalt IT has graciously offered to pay for the airfare.  In addition to the VMworld conference pass, Gestalt IT will provide the winner with round trip airfare, up to $500.  All we ask in return is that the recipient provide a post-VMworld write-up of what they learned from attending the conference.  This could be a written document, a blog post, a video, you choose.  Thank you Gestalt IT for your donation!

Contest Rules:

  1. Post one comment/reply and only one comment/reply to this blog article below.
    • Include your first and last name.
    • Provide a valid email address when completing the comment form.
    • Include a short bio about yourself and how you use VMware currently or how you would like to leverage VMware products.
    • Include three (3) things you are looking to gain from attending VMworld 2010 (ie. Why do you want to go?)
    • Contest entry must be recieved by noon CST Thursday June 17th, 2010.
  2. One (1) random winner will be chosen Thursday evening June 17th, 2010.
    • Winner will be contacted via email address provided above.
    • Winner will recieve a VMworld 2010 San Francisco conference pass.
    • Winner will receive airfare up to $500 from Gestalt IT.
    • Winner will provide a post-VMworld write-up of what they learned from attending the conference to Gestalt IT.
  3. Contest results will posted Friday June 18th, 2010.
  4. The conference pass is non-transferrable and non-refundable.
  5. Hotel, meals, and other expenses are not covered by
  6. No purchase necessary.

Good Luck!

Update 6/17/10: 


A name has been randomly drawn and we have a winner! Congratulations to contest winner Greg Stuart who will be receiving a VMworld 2010 conference pass and round trip airfare (up to $500).  Greg’s winning entry and BIO is listed below:

I currently work for an organization that has begun to leverage VMware more and more. I’m new to virtualization and would like to gain a better knowledge of the VMware products, attend some hands on sessions and come back with solutions that I can employ in our environment. The ability to discuss scenarios and solutions with vendors in person would be awesome.

I’m pleased with the outcome of the contest.  Greg is new to virtualization and I think there is a lot of valuable information he will be able to pick up at VMworld.  Better yet, VMworld is a 4 day event this year – Monday thru Thursday instead of 3 days as it was prior years.  This affords Greg the opportunity to take in a whole extra day of content.

Thank you to all who participated in the contest including Gestalt IT for contributed the round trip airfare.  Although there could ultimately be only one grand prize contest winner, my hope is that you all will make the show this year somehow.  There are nearly 90 comments/replies to this post explaining the values which VMworld can provide. Much of this content could be borrowed to write or improve your own compelling justification, hopefully earning you a trip to VMworld.

Everyone have a great weekend!

Minneapolis VMUG / Veterans Sock Drive Benefit Friday

May 18th, 2010

Friendly reminder that from 1-5pm this Friday, the Minneapolis VMware User Group will hold a quarterly meeting in Bloomington.  I expect a lot of VDI content.  The meeting details can be found here.

Hilton Minneapolis
Bloomington Ballroom Foyer A
3900 American Boulevard West
Bloomington, MN 55437

In addition…

VMware is proud to work with the Minnesota Assistance Council For Veterans (MACV) by supporting a one day SOCK DRIVE BLITZ at the VMware User Group Meeting (VMUG). 

On Friday 21 May 2010 MACV and VMware will be collecting packages of new white cotton socks (primarily men’s sizes) as part of the MACV Veterans Standdown. The socks collected will then be distributed to homeless veterans in need at the Various standdown events across Minnesota throughtout the year. 


Items Needed for donations:
Packages of new white cotton socks
(primarily men’s sizes)

Minnesota Assistance Council for Veterans (MACV) is a 501(c) 3 non-profit organization assisting veterans (and their families) in crisis throughout Minnesota for over 18 years including those who are experiencing homelessness. MACV is the only organization of its kind in the State of Minnesota completely dedicated to serving the needs of veterans, and is nationally recognized for its model and success.

VMkernel Networks, Jumbo Frames, and ESXi 4

February 12th, 2010

Question:  Can I implement jumbo frames on ESXi 4 Update 1 VMkernel networks?

Answer:  Who in the hell knows?

You see, the ESXi 4.0 Update 1 Configuration Guide states on page 54:

“Jumbo frames are not supported for VMkernel networking interfaces in ESXi.”

Duncan Epping of Yellow Bricks also reports:

“Jumbo frames are not supported for VMkernel networking interfaces in ESXi. (page 54)”

One month after the release of ESXi 4 Update 1, Charu Chaubal of VMware posted on the ESXi Chronicles blog:

“I am happy to say that this is merely an error in the documentation. In fact, ESXi 4.0 DOES support Jumbo Frames on VMkernel networking interfaces. The correction will hopefully appear in a new release of the documentation, but in the meantime, go ahead and configure Jumbo frames for your ESXi 4.0 hosts.”

Shortly after, Duncan Epping of Yellow Bricks confirms Charu Chaubal’s report that jumbo frames are supported on ESXi VMkernel networks.

Now, nearly two months after Charu’s clarification and three months after the release of ESXi 4 Update 1, the documentation remains dubious on page 54 stating that jumbo frames are not supported on ESXi 4 VMkernel networks which is a direct contradition to a VMware ESXi blog.

I opened a Business Critical Support SR with VMware on the question.  I was told by VMware BCS that jumbo frames are NOT supported on ESXi 4 Update 1 VMkernel networks and a reference was made to the documentatation on page 54. 

Our dedicated VMware onsite Engineer escalated and I was then told ESXi 4 Update 1 DOES support jumbo frames on VMkernel networks, making reference to Charu’s article.

Hey VMware, which is it?  If this is a documentation mistake, why are you dragging your feet in getting the documentation updated two months after a VMware employee discovers the error and blogs it?  Waiting for the next release of ESXi?  Unacceptable!  You update the public documentation as soon as you discover the error and be damned sure your BCS support Engineers know the right answer!  Do you know how much companies pay for BCS?  You owe your customers the correct answer.  If misinformation comes as a result of a known documentation error, SHAME ON YOU!  Architecture and design decisions are being made daily on this information or misinformation, which ever it may be.

Update 2/23/10:  Toby Kraft (@vmwarewriter on Twitter) will be updating the documentation by next week.  Thank you Toby!

Update 3/1/10:  VMware has updated their documentation to reflect currently supported configurations.  Thank you VMware (and Toby)!

Virtualizing vCenter With vDS Catch-22

October 9th, 2009

I’ve typically been a fan of virtualizing the vCenter management server in most situations. VMware vCenter and Update Manager both make fine virtualization candidates as long as the underlying infrastructure for vCenter stays up. Loss of vCenter in a blackout situation can make things a bit of a hassle, but one can work through it with the right combination of patience and knowledge.

A few nights ago I had decided to migrate my vCenter VM to my vSphere virtual infrastructure. Because my vCenter VM was on a standalone VMware Server 2.0 box, I had to shut down the vCenter VM in order to cold migrate it to one of the ESX4 hosts directly, transfer the files to the SAN, upgrade virtual hardware, etc. Once the files were migrated to the vSphere infrastructure, it was time to configure the VM for the correct network and power it up. This is where I ran into the problem.

vCenter was shut down and unavailable, therefore, I had connected my vSphere client directly to the ESX4 host in which I transferred the VM to. When trying to configure the vCenter VM to use the vNetwork Distributed Switch (vDS) port group I had set up for all VM traffic, it was unavailable in the dropdown list of networks. The vCenter server was powered down and thus the vDS Control Plane was unavailable, eliminating my view of vDS networks.

This is a dilemma. Without a network connection, the vCenter server will not be able to communicate with the back end SQL database on a different box running SQL. This will cause the vCenter server services to not start and thus I’ll never have visibility to the vDS. Fortunately I have a fairly flat network in the lab with just a few subnets. I was able to create a temporary vSwitch and port group locally on the ESX4 host which would grant the vCenter VM the network connectivity it needed so I could then modify the network, changing from a local to a vDS port group on the fly.

Once the vCenter server was back up, I further realized that vDS port groups are still unable to be seen when the vSphere client is connected directly to an ESX4 host. The ability configure a VM to utilize vDS networking requires both that the vCenter server be functional, as well as a vSphere client connected to said vCenter server and not a managed host.

The situation I explained above is the catch-22 – the temporary inability to configure VMs for vDS networking while the vCenter server is unavailable. One might call my situation a convergence of circumstances, but with an existing virtualized vCenter server that you’re looking to migrate to a vDS integrated vSphere infrastructure, the scenario is very real. I’d like to note all VMs that had been running on a vDS port continued to run without a network outage as the vDS Data Plane is maintained on each host and remained in tact.

Lab Manager 4 and vDS

September 19th, 2009

VMware Lab Manager 4 enables new functionality in that fenced configurations can now span ESX(i) hosts by leveraging vNetwork Distributed Switch (vDS) technology which is a new feature in VMware vSphere. Before getting overly excited, remember that vDS is a VMware Enterprise Plus feature only and it’s only found in vSphere. Without vSphere and VMware’s top tier license, vDS cannot be implemented and thus you wouldn’t be able to enable fenced Lab Manager 4 configurations to span hosts.

Host Spanning is enabled by default when a Lab Manager 4 host is prepared as indicated by the green check marks below:

When Host Spanning is enabled, an unmanageable Lab Manager service VM is pinned to each host where Host Spanning is enabled. This Lab Manager service VM cannot be powered down, suspended, VMotioned, etc.:

One ill side effect of this new Host Spanning technology is that an ESX(i) host will not enter maintenance mode while Host Spanning is enabled. For those new to Lab Manager 4, the cause may not be so obvious and it can lead to much frustration. An unmanageable Lab Manager service VM is pinned to each host where Host Spanning is enabled and a running VM will prevent a host from entering maintenance mode. Maintenance mode will hang at the infamous 2% complete status:

The resolution is to first cancel the maintenance mode request. Then, manually disable host spanning in the Lab Manager host configuration property sheet by unchecking the box. Notice the highlighted message in pink telling us that Host Spanning must be disabled in order for the host to enter standby or maintenance mode. Unpreparing the host will also accomplish the goal of removing the service VM but this is much more drastic and should only be done if no other Lab Manager VMs are running on the host:

After reconfiguring the Lab Manager 4 host as described above, vSphere Client Recent Tasks shows the service VM is powered off and then removed by the Lab Manager service account:

At this time, invoke the maintenance mode request and the host will now be able to migrate all VMs off and successfully enter maintenance mode.

While Lab Manager 4 Host Spanning is a step in the right direction for more flexible load distribution across hosts in a Lab Manager 4 cluster, I find the process for entering maintenance mode counter intuitive, cumbersome, and at the beginning when I didn’t know what was going on, frustrating. Unsuccessful maintenance mode attempts have always been somewhat mysterious in the past because vCenter Server doesn’t give us much information to pinpoint the problem as far as what’s preventing the maintenance mode. This situation now adds another element to the complexity. VMware should have enough intelligence to disable Host Spanning for us in the event of a maintenance mode request, or at the very least, tell us to shut it off since it is conveniently and secretly enabled by default during host preparation. Of course, all of this information is available in the Lab Manager documentation, but who reads that, right? 🙂