Comments on: Jumbo Frames Comparison Testing with IP Storage and vMotion

By: Michael Webster

Michael Webster — Thu, 10 Jan 2013 18:19:59 +0000

Remember when virtualizing that it’s not just about increase in network performance but also a decrease in CPU utilization that can be a benefit of Jumbo Frames. Depending on your environment this may have a more important benefit than the 1% difference in performance. With 10G infrastructure my testing found more like a 10% difference, which is worth while gaining, and this magnifies the more NIC’s you have on your hosts. But I wouldn’t normally bother with Jumbo Frames unless on 10G networking or higher as the benefit is not there for the additional effort.

By: Robert

Robert — Wed, 28 Nov 2012 23:21:38 +0000

Thanks Jason for the extensive research.

I read your article because I was challenged recently on the reason I failed to implement jumbo frames on a 2-way replicated vmware solution running LH P4000 VSA’s.

I really thought it was a huge issue. Thank goodness it is not.

I feel personally I would not implement jumbo frames unless I was dealing with a 10Gig Core.

Thanks so much for your article.

By: Abhinav

Abhinav — Fri, 09 Nov 2012 11:31:22 +0000

Frankly, I was surprised to see the results in this post and decided to verify it.

My tests also showed 0-3% of performance enhancement using Jumbo Frames.

Here is what I did:
vmk1: 9000 MTU
vSwitch: 9000 MTU
Cisco 3750 : 9000 MTU
NetApp Filer: 9000 MTU.

VM size: 10.46 GB MTU 9000 MTU 1500 Difference
vMotion (F1 to F4) 6 mins 29 secs 6 mins 35 secs
vMotion (F1 to F4) 6 mins 33 secs 6 mins 35 secs
vMotion (F4 to F1) 6 mins 31 secs 6 mins 33 secs
vMotion (F4 to F1) 6 mins 31 secs 6 mins 32 secs
Average (secs) 391 393.75 1%

Offline Migration (F1 to F4) 6 mins 27 secs 6 mins 33 secs
Offline Migration (F1 to F4) 6 mins 29 secs 6 mins 34 secs
Offline Migration (F4 to F1) 6 mins 28 secs 6 mins 30 secs
Offline Migration (F4 to F1) 6 mins 27 secs 6 mins 33 secs
Average (secs) 387.25 392.5 1%

Cloning (F1 to F4) 6 mins 31 secs 6 mins 31 secs 0%

DataSize: 102 MB (in Seconds) (in Seconds)
Copy-Paste 1st (after reboot) 13 14
Copy-Paste 1st (after reboot) 13 13
Copy-Paste 2nd (after 1st copy) 4 4
Copy-Paste 2nd (after 1st copy) 4 4
Average 8.5 8.75 3%

VM Boot time 13 13 0%

NOTE:
F1 = NetApp Filer 1
F4 = NetApp Filer 4

By: Andre

Andre — Mon, 05 Nov 2012 21:10:47 +0000

I’d like to rehash this post. Especially now since vSphere storage vmotions automagically across L2 networks.

We have 2 management nics. One for Management Traffic only(vmk0), and one for vMotion traffic only(vmk1).
I would like to know 2 things.

1) Since management traffic runs over SSL. Does vmk0 management traffic fragment when MTU is set at 1500, should I set it at 1600
2) Does Storage vMotion traffic use Jumbo Frames, and if so, should I set my vmk1 nic to 9000MTU

By: Jumbo Frames on vSphere 5 Update 1 « Long White Virtual Clouds

Jumbo Frames on vSphere 5 Update 1 « Long White Virtual Clouds — Mon, 24 Sep 2012 22:19:13 +0000

[...] The original inspiration to write this article came from reading Jason Boche’s article Jumbo Frames Comparison Testing with IP Storage and vMotion. [...]

By: Erik

Erik — Sat, 22 Sep 2012 23:03:32 +0000

Any idea if testing 4k as an MTU size would make any difference? I’ve read some materials that indicate 4k performed better than 1500 and 9000 in some cases.

By: Joe Antkowiak

Joe Antkowiak — Tue, 29 May 2012 17:21:54 +0000

Also… depending on the type of switch (the type and class of the switch really matters a lot here) you can gain some performance for iscsi, both 1500 and 9000 mtu, if you are able to put your iscsi traffic into the priority queue/low latency queue on the switches. you will lower your latency by doing this.

typically this is used for voice traffic, but if you’ve correctly separated your voice and storage traffic, this shouldn’t be an issue.

By: Michael Webster

Michael Webster — Mon, 16 Apr 2012 13:33:45 +0000

My testing on 10G shows a much bigger difference between Jumbo and non-Jumbo. It’s still definitely worth considering.

By: 8026mn

8026mn — Fri, 13 Apr 2012 18:17:11 +0000

Good to have data to show if it makes a difference or not. I always wondered about the utopia that is Jumbo Frames, but sounds like KISS applies again. Unless there is a requirement to get every bit of performance that you can, otherwise its a nominal change.

By: gchapman

gchapman — Sun, 19 Feb 2012 15:31:48 +0000

Great testing and very detailed. Perhaps you can run the test with the IO Analyzer that vmware provided in a fling earlier this year. There are roughly 50 different IO workloads you can simulate with a canned centos vm host.

By: Jumbo Frames on vSphere 5 « Long White Virtual Clouds

Jumbo Frames on vSphere 5 « Long White Virtual Clouds — Sun, 19 Feb 2012 12:05:52 +0000

[...] read a great blog post a while ago from Jason Boche titled Jumbo Frames Comparison Testing with IP Storage and vMotion. The results of the tests showed at best marginal gains to be had from using Jumbo Frames with [...]

By: jason

jason — Wed, 11 Jan 2012 14:16:08 +0000

@JBosman: IIRC there is only 1 setting on the 3COM switch I tested with – Jumbo Frames enabled or disabled. MTU was configured for 9000 on all endpoint interfaces during the tests.

By: JBosman

JBosman — Wed, 11 Jan 2012 08:28:11 +0000

Hello Jason,

Someone pointed me to this article and after reading it i was wondering what MTU size was set on the 3COM switch.

Thanks!

By: Open Tabs 12/13

Open Tabs 12/13 — Tue, 13 Dec 2011 18:10:11 +0000

[...] Frames & vSphere IO performance – http://www.boche.net/blog/index.php/2011/01/24/jumbo-frames-comparison-testing-with-ip-storage-and-v…Well worth a read if you’ll be working with Jumbo Frames or IP storage in your [...]

By: Michael Webster

Michael Webster — Sun, 04 Dec 2011 21:33:20 +0000

Hi Jason, Testing I’ve done on 10G links shows massive improvement just by using Jumbo Frames. But granted it probably won’t show much improvement on a 1Gb/s network as most everything can saturate it. vMotion for example will top out at 800 – 900MB/s with 1500 MTU and will easily exceed 1250MB/s with 9000MTU on ESXi 5 single NIC tests. With dual 10G test I topped out at around 1600MB/s because I was CPU constrained on my servers. Also the theoretical throughput limit for 1Gb/s is 125MB/s, which you should be able to easily hit with Jumbo turned on when doing vMotion.