Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Author Topic: To Jumbo or not to Jumbo, the MTU dilemma  (Read 622 times)

Alex Atkin UK

  • Kitizen
  • ****
  • Posts: 1166
    • My Broadband History
To Jumbo or not to Jumbo, the MTU dilemma
« on: March 30, 2020, 07:59:07 PM »

Jumbo frames seem to be very polarising, some say you should use them, some say you absolutely shouldn't.  So I figured I would test my own network.

The host is an Intel i5 8600k (with two cores running Folding at Home) with an Intel X520-DA1 connected to the switch over an SFP+ 10Gbit DAC.

i5 4690 running Folding at Home on all cores with an ASUS XG-C100C 10Gbit (10G/N-BASE-T) NIC
1500 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  5]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec                 
[  5]   2.00-3.00   sec  1.10 GBytes  9.41 Gbits/sec                 
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  5]   4.00-5.00   sec  1.09 GBytes  9.36 Gbits/sec                 
[  5]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec                 
[  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec                 
[  5]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  5]   9.00-10.00  sec  1.09 GBytes  9.41 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver

3000 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.13 GBytes  9.69 Gbits/sec                 
[  5]   1.00-2.00   sec  1.13 GBytes  9.69 Gbits/sec                 
[  5]   2.00-3.00   sec  1.13 GBytes  9.70 Gbits/sec                 
[  5]   3.00-4.00   sec  1.13 GBytes  9.70 Gbits/sec                 
[  5]   4.00-5.00   sec  1.13 GBytes  9.70 Gbits/sec                 
[  5]   5.00-6.00   sec  1.12 GBytes  9.59 Gbits/sec                 
[  5]   6.00-7.00   sec  1.13 GBytes  9.69 Gbits/sec                 
[  5]   7.00-8.00   sec  1.13 GBytes  9.70 Gbits/sec                 
[  5]   8.00-9.00   sec  1.13 GBytes  9.69 Gbits/sec                 
[  5]   9.00-10.00  sec  1.12 GBytes  9.63 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.3 GBytes  9.68 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.3 GBytes  9.68 Gbits/sec                  receiver

6000 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.14 GBytes  9.77 Gbits/sec                 
[  5]   1.00-2.00   sec  1.15 GBytes  9.85 Gbits/sec                 
[  5]   2.00-3.00   sec  1.15 GBytes  9.85 Gbits/sec                 
[  5]   3.00-4.00   sec  1.14 GBytes  9.81 Gbits/sec                 
[  5]   4.00-5.00   sec  1.15 GBytes  9.85 Gbits/sec                 
[  5]   5.00-6.00   sec  1.15 GBytes  9.85 Gbits/sec                 
[  5]   6.00-7.00   sec  1.15 GBytes  9.85 Gbits/sec                 
[  5]   7.00-8.00   sec  1.14 GBytes  9.82 Gbits/sec                 
[  5]   8.00-9.00   sec  1.15 GBytes  9.85 Gbits/sec                 
[  5]   9.00-10.00  sec  1.14 GBytes  9.83 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.4 GBytes  9.83 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.4 GBytes  9.83 Gbits/sec                  receiver

9000 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.13 GBytes  9.72 Gbits/sec                 
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec                 
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec                 
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec                 
[  5]   4.00-5.00   sec  1.15 GBytes  9.87 Gbits/sec                 
[  5]   5.00-6.00   sec  1.15 GBytes  9.88 Gbits/sec                 
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec                 
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec                 
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec                 
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.88 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  receiver

Intel i9 9900k using an Aquantia AQtion AQC111C 5G USB adapter (bottlenecked by USB 3.1 Gen 1)
1500 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   407 MBytes  3.41 Gbits/sec                 
[  5]   1.00-2.00   sec   416 MBytes  3.49 Gbits/sec                 
[  5]   2.00-3.00   sec   416 MBytes  3.49 Gbits/sec                 
[  5]   3.00-4.00   sec   416 MBytes  3.49 Gbits/sec                 
[  5]   4.00-5.00   sec   417 MBytes  3.49 Gbits/sec                 
[  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec                 
[  5]   6.00-7.00   sec   416 MBytes  3.49 Gbits/sec                 
[  5]   7.00-8.00   sec   417 MBytes  3.50 Gbits/sec                 
[  5]   8.00-9.00   sec   416 MBytes  3.49 Gbits/sec                 
[  5]   9.00-10.00  sec   416 MBytes  3.49 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec   31             sender
[  5]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec                  receiver

3000 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   427 MBytes  3.58 Gbits/sec                 
[  5]   1.00-2.00   sec   428 MBytes  3.59 Gbits/sec                 
[  5]   2.00-3.00   sec   428 MBytes  3.59 Gbits/sec                 
[  5]   3.00-4.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   4.00-5.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   5.00-6.00   sec   422 MBytes  3.54 Gbits/sec                 
[  5]   6.00-7.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   7.00-8.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   8.00-9.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   9.00-10.00  sec   429 MBytes  3.60 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.18 GBytes  3.59 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.18 GBytes  3.59 Gbits/sec                  receiver

6000 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   1.00-2.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   2.00-3.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   3.00-4.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   4.00-5.00   sec   428 MBytes  3.59 Gbits/sec                 
[  5]   5.00-6.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   6.00-7.00   sec   429 MBytes  3.60 Gbits/sec                 
[  5]   7.00-8.00   sec   430 MBytes  3.60 Gbits/sec                 
[  5]   8.00-9.00   sec   430 MBytes  3.60 Gbits/sec                 
[  5]   9.00-10.00  sec   427 MBytes  3.58 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.19 GBytes  3.60 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.19 GBytes  3.60 Gbits/sec                  receiver

9000 MTU:
Code: [Select]
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   423 MBytes  3.55 Gbits/sec                 
[  5]   1.00-2.00   sec   423 MBytes  3.55 Gbits/sec                 
[  5]   2.00-3.00   sec   423 MBytes  3.55 Gbits/sec                 
[  5]   3.00-4.00   sec   424 MBytes  3.55 Gbits/sec                 
[  5]   4.00-5.00   sec   424 MBytes  3.55 Gbits/sec                 
[  5]   5.00-6.00   sec   424 MBytes  3.55 Gbits/sec                 
[  5]   6.00-7.00   sec   423 MBytes  3.55 Gbits/sec                 
[  5]   7.00-8.00   sec   423 MBytes  3.55 Gbits/sec                 
[  5]   8.00-9.00   sec   423 MBytes  3.55 Gbits/sec                 
[  5]   9.00-10.00  sec   423 MBytes  3.55 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.14 GBytes  3.55 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.14 GBytes  3.55 Gbits/sec                  receiver

These are purely downstream tests with flow control enabled on the switches as I had serious issues between different speed clients without it.

Its curious how the USB adapter ALWAYS struggles on the standard MTU, it always ends up with some retries which makes me wonder if that is some sort of USB bottleneck in itself.

The interesting thing about these results is the 10Gbit client is actually connecting to my second switch, so passes over copper 10G into that switch, copper 10G to the main switch (using a cheap 7m flat suposedly CAT6 cable) then DAC 10G to the server.  So its nice to see that path working without any issues.
« Last Edit: March 30, 2020, 08:06:59 PM by Alex Atkin UK »
Logged
Exchange: INTAKE (ECI DSLAM) Router: pfSense (i5-7200U) ISP/Modems: Zen (Home Hub 5A running OpenWrt) + Plusnet (VMG-3925-B10B) WiFi: Ubiquiti nanoHD

neil

  • Reg Member
  • ***
  • Posts: 194
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #1 on: March 30, 2020, 09:47:42 PM »

Hi,

Can you please explain what these tests show?
Logged

Alex Atkin UK

  • Kitizen
  • ****
  • Posts: 1166
    • My Broadband History
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #2 on: March 30, 2020, 10:57:51 PM »

Firstly to avoid any potential confusion, this has nothing to do with broadband, this is purely about improving the data transfer speeds over the LAN.

The basic principle is that each packet of network data is normally 1500 bytes, but on really fast networks this consumes a lot of CPU time so increasing it can improve data transfer speeds.

That said in my case, there is no shortage of CPU power so I'm kinda curious where the improvement comes from.
Logged
Exchange: INTAKE (ECI DSLAM) Router: pfSense (i5-7200U) ISP/Modems: Zen (Home Hub 5A running OpenWrt) + Plusnet (VMG-3925-B10B) WiFi: Ubiquiti nanoHD

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 6203
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #3 on: March 30, 2020, 11:10:05 PM »

I would expect jumbo frames would help you more on a very weak cpu, as its about the ability to process PPS.  A fast cpu can do enough PPS to max out the link without jumbo frames.
Logged
AAISP - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

Alex Atkin UK

  • Kitizen
  • ****
  • Posts: 1166
    • My Broadband History
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #4 on: March 31, 2020, 01:26:00 AM »

I would expect jumbo frames would help you more on a very weak cpu, as its about the ability to process PPS.  A fast cpu can do enough PPS to max out the link without jumbo frames.

Indeed, but its curious how it scales in my testing as neither of those clients have weak CPUs.

The USB is particularly curious, I can only guess its something to do with how the packet processing works on that adapter.
Logged
Exchange: INTAKE (ECI DSLAM) Router: pfSense (i5-7200U) ISP/Modems: Zen (Home Hub 5A running OpenWrt) + Plusnet (VMG-3925-B10B) WiFi: Ubiquiti nanoHD

CarlT

  • Kitizen
  • ****
  • Posts: 1589
  • Next generation network design and deployment
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #5 on: March 31, 2020, 12:33:02 PM »

Just do it. On reasonably capable NICs and switching hardware it's a non-issue and has been for a while.
Logged
WiFi: Nighthawk® AX12 RAX120 - 5Gb uplink
Routing: Ubiquiti UDM Pro - 10Gb uplink
Switching: 2 * Mikrotik CRS305-1G-4S-IN, 10Gb uplinks, various cheap and cheerful
ONT: Huawei 1+1
Exchange: Wakefield
ISP: BT Full Fibre 900. Yes, BT, I know.

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 8682
  • Retd sw dev; A&A; 4 × 7km ADSL2; IPv6; Firebrick
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #6 on: March 31, 2020, 02:19:41 PM »

The larger MTU means less wastage due to headers, there’s that as well as the PPS figure.
Logged

Alex Atkin UK

  • Kitizen
  • ****
  • Posts: 1166
    • My Broadband History
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #7 on: March 31, 2020, 07:28:01 PM »

Just do it. On reasonably capable NICs and switching hardware it's a non-issue and has been for a while.

Indeed, and MTU discovery seems to do its job.  The whole idea of having to set the whole network to the same value seems to be redundant.

The Server happily sits with a 9000 MTU, the 10Gbit client also at 9000, the 5Gbit at 6000 and the rest of the network still at 1500.

Considering the server is then routed over a VPN via pfSense, its not like I wouldn't have noticed MTU issues by now.

The most frustrating aspect is Windows 10, why they have to make changing the MTU so awkward I do not know.  I can set it in the driver for the 5G adapter but for some bizarre reason they only have three pre-defined settings, none of which are optimal from my testing above.  Of course I could also set it from the command line, but Windows has a nasty habit of re-detecting as a new network for no apparent reason (possibly pfSense updates throwing it off?) which then means its not using the old settings.
Logged
Exchange: INTAKE (ECI DSLAM) Router: pfSense (i5-7200U) ISP/Modems: Zen (Home Hub 5A running OpenWrt) + Plusnet (VMG-3925-B10B) WiFi: Ubiquiti nanoHD

CarlT

  • Kitizen
  • ****
  • Posts: 1589
  • Next generation network design and deployment
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #8 on: March 31, 2020, 07:42:50 PM »

TCP will be okay by default. UDP can be fragmented and reassembled or use ICMP to find the correct MTU.
Logged
WiFi: Nighthawk® AX12 RAX120 - 5Gb uplink
Routing: Ubiquiti UDM Pro - 10Gb uplink
Switching: 2 * Mikrotik CRS305-1G-4S-IN, 10Gb uplinks, various cheap and cheerful
ONT: Huawei 1+1
Exchange: Wakefield
ISP: BT Full Fibre 900. Yes, BT, I know.

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 6203
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #9 on: March 31, 2020, 09:13:42 PM »

Just do it. On reasonably capable NICs and switching hardware it's a non-issue and has been for a while.

If I am using a NIC for both WAN and LAN, e.g. my desktop, and enable 9000 byte frames, would this send out 9000 byte frames for internet traffic (which my router would then have to reduce to 1500byte segments), or is software intelligent enough to automatically size packets for LAN and WAN by itself?

I am curious, for this reason I have always kept all my LAN on 1500 bytes.
Logged
AAISP - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 8682
  • Retd sw dev; A&A; 4 × 7km ADSL2; IPv6; Firebrick
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #10 on: March 31, 2020, 09:23:51 PM »

Very good question. I don’t know but I’m assuming that software has to be intelligent and do the right thing.
Logged

CarlT

  • Kitizen
  • ****
  • Posts: 1589
  • Next generation network design and deployment
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #11 on: March 31, 2020, 11:56:34 PM »

No.

Your kit will offer an MSS of 8960, the remote end will come back with 1460 and you use the smallest of the two.
Logged
WiFi: Nighthawk® AX12 RAX120 - 5Gb uplink
Routing: Ubiquiti UDM Pro - 10Gb uplink
Switching: 2 * Mikrotik CRS305-1G-4S-IN, 10Gb uplinks, various cheap and cheerful
ONT: Huawei 1+1
Exchange: Wakefield
ISP: BT Full Fibre 900. Yes, BT, I know.

aesmith

  • Reg Member
  • ***
  • Posts: 997
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #12 on: April 01, 2020, 02:46:04 PM »

Very good question. I don’t know but I’m assuming that software has to be intelligent and do the right thing.
Maybe it would be more accurate to say software has to be "not stupid".  We have a fault report, which as usual had been qualified as a network issue, which was due to a server sending out jumbo frames with DF set.
Logged

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 29459
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: To Jumbo or not to Jumbo, the MTU dilemma
« Reply #13 on: April 01, 2020, 08:43:21 PM »

. . . which was due to a server sending out jumbo frames with DF set.

A rather big Oops!  :doh:
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.