Kitz Forum
Computers & Hardware => Networking => Topic started by: Alex Atkin UK on March 30, 2020, 07:59:07 PM
-
Jumbo frames seem to be very polarising, some say you should use them, some say you absolutely shouldn't. So I figured I would test my own network.
The host is an Intel i5 8600k (with two cores running Folding at Home) with an Intel X520-DA1 connected to the switch over an SFP+ 10Gbit DAC.
i5 4690 running Folding at Home on all cores with an ASUS XG-C100C 10Gbit (10G/N-BASE-T) NIC
1500 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec
[ 5] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 5] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 4.00-5.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec
[ 5] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec
[ 5] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 5] 9.00-10.00 sec 1.09 GBytes 9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver
3000 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.13 GBytes 9.69 Gbits/sec
[ 5] 1.00-2.00 sec 1.13 GBytes 9.69 Gbits/sec
[ 5] 2.00-3.00 sec 1.13 GBytes 9.70 Gbits/sec
[ 5] 3.00-4.00 sec 1.13 GBytes 9.70 Gbits/sec
[ 5] 4.00-5.00 sec 1.13 GBytes 9.70 Gbits/sec
[ 5] 5.00-6.00 sec 1.12 GBytes 9.59 Gbits/sec
[ 5] 6.00-7.00 sec 1.13 GBytes 9.69 Gbits/sec
[ 5] 7.00-8.00 sec 1.13 GBytes 9.70 Gbits/sec
[ 5] 8.00-9.00 sec 1.13 GBytes 9.69 Gbits/sec
[ 5] 9.00-10.00 sec 1.12 GBytes 9.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.3 GBytes 9.68 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.3 GBytes 9.68 Gbits/sec receiver
6000 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.14 GBytes 9.77 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.85 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.85 Gbits/sec
[ 5] 3.00-4.00 sec 1.14 GBytes 9.81 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.85 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.85 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.85 Gbits/sec
[ 5] 7.00-8.00 sec 1.14 GBytes 9.82 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.85 Gbits/sec
[ 5] 9.00-10.00 sec 1.14 GBytes 9.83 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.4 GBytes 9.83 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.4 GBytes 9.83 Gbits/sec receiver
9000 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.13 GBytes 9.72 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.88 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec receiver
Intel i9 9900k using an Aquantia AQtion AQC111C 5G USB adapter (bottlenecked by USB 3.1 Gen 1)
1500 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 407 MBytes 3.41 Gbits/sec
[ 5] 1.00-2.00 sec 416 MBytes 3.49 Gbits/sec
[ 5] 2.00-3.00 sec 416 MBytes 3.49 Gbits/sec
[ 5] 3.00-4.00 sec 416 MBytes 3.49 Gbits/sec
[ 5] 4.00-5.00 sec 417 MBytes 3.49 Gbits/sec
[ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec
[ 5] 6.00-7.00 sec 416 MBytes 3.49 Gbits/sec
[ 5] 7.00-8.00 sec 417 MBytes 3.50 Gbits/sec
[ 5] 8.00-9.00 sec 416 MBytes 3.49 Gbits/sec
[ 5] 9.00-10.00 sec 416 MBytes 3.49 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec 31 sender
[ 5] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec receiver
3000 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 427 MBytes 3.58 Gbits/sec
[ 5] 1.00-2.00 sec 428 MBytes 3.59 Gbits/sec
[ 5] 2.00-3.00 sec 428 MBytes 3.59 Gbits/sec
[ 5] 3.00-4.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 4.00-5.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 5.00-6.00 sec 422 MBytes 3.54 Gbits/sec
[ 5] 6.00-7.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 7.00-8.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 8.00-9.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 9.00-10.00 sec 429 MBytes 3.60 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.18 GBytes 3.59 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.18 GBytes 3.59 Gbits/sec receiver
6000 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 1.00-2.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 2.00-3.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 3.00-4.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 4.00-5.00 sec 428 MBytes 3.59 Gbits/sec
[ 5] 5.00-6.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 6.00-7.00 sec 429 MBytes 3.60 Gbits/sec
[ 5] 7.00-8.00 sec 430 MBytes 3.60 Gbits/sec
[ 5] 8.00-9.00 sec 430 MBytes 3.60 Gbits/sec
[ 5] 9.00-10.00 sec 427 MBytes 3.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.19 GBytes 3.60 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.19 GBytes 3.60 Gbits/sec receiver
9000 MTU:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 1.00-2.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 2.00-3.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 3.00-4.00 sec 424 MBytes 3.55 Gbits/sec
[ 5] 4.00-5.00 sec 424 MBytes 3.55 Gbits/sec
[ 5] 5.00-6.00 sec 424 MBytes 3.55 Gbits/sec
[ 5] 6.00-7.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 7.00-8.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 8.00-9.00 sec 423 MBytes 3.55 Gbits/sec
[ 5] 9.00-10.00 sec 423 MBytes 3.55 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.14 GBytes 3.55 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.14 GBytes 3.55 Gbits/sec receiver
These are purely downstream tests with flow control enabled on the switches as I had serious issues between different speed clients without it.
Its curious how the USB adapter ALWAYS struggles on the standard MTU, it always ends up with some retries which makes me wonder if that is some sort of USB bottleneck in itself.
The interesting thing about these results is the 10Gbit client is actually connecting to my second switch, so passes over copper 10G into that switch, copper 10G to the main switch (using a cheap 7m flat suposedly CAT6 cable) then DAC 10G to the server. So its nice to see that path working without any issues.
-
Hi,
Can you please explain what these tests show?
-
Firstly to avoid any potential confusion, this has nothing to do with broadband, this is purely about improving the data transfer speeds over the LAN.
The basic principle is that each packet of network data is normally 1500 bytes, but on really fast networks this consumes a lot of CPU time so increasing it can improve data transfer speeds.
That said in my case, there is no shortage of CPU power so I'm kinda curious where the improvement comes from.
-
I would expect jumbo frames would help you more on a very weak cpu, as its about the ability to process PPS. A fast cpu can do enough PPS to max out the link without jumbo frames.
-
I would expect jumbo frames would help you more on a very weak cpu, as its about the ability to process PPS. A fast cpu can do enough PPS to max out the link without jumbo frames.
Indeed, but its curious how it scales in my testing as neither of those clients have weak CPUs.
The USB is particularly curious, I can only guess its something to do with how the packet processing works on that adapter.
-
Just do it. On reasonably capable NICs and switching hardware it's a non-issue and has been for a while.
-
The larger MTU means less wastage due to headers, there’s that as well as the PPS figure.
-
Just do it. On reasonably capable NICs and switching hardware it's a non-issue and has been for a while.
Indeed, and MTU discovery seems to do its job. The whole idea of having to set the whole network to the same value seems to be redundant.
The Server happily sits with a 9000 MTU, the 10Gbit client also at 9000, the 5Gbit at 6000 and the rest of the network still at 1500.
Considering the server is then routed over a VPN via pfSense, its not like I wouldn't have noticed MTU issues by now.
The most frustrating aspect is Windows 10, why they have to make changing the MTU so awkward I do not know. I can set it in the driver for the 5G adapter but for some bizarre reason they only have three pre-defined settings, none of which are optimal from my testing above. Of course I could also set it from the command line, but Windows has a nasty habit of re-detecting as a new network for no apparent reason (possibly pfSense updates throwing it off?) which then means its not using the old settings.
-
TCP will be okay by default. UDP can be fragmented and reassembled or use ICMP to find the correct MTU.
-
Just do it. On reasonably capable NICs and switching hardware it's a non-issue and has been for a while.
If I am using a NIC for both WAN and LAN, e.g. my desktop, and enable 9000 byte frames, would this send out 9000 byte frames for internet traffic (which my router would then have to reduce to 1500byte segments), or is software intelligent enough to automatically size packets for LAN and WAN by itself?
I am curious, for this reason I have always kept all my LAN on 1500 bytes.
-
Very good question. I don’t know but I’m assuming that software has to be intelligent and do the right thing.
-
No.
Your kit will offer an MSS of 8960, the remote end will come back with 1460 and you use the smallest of the two.
-
Very good question. I don’t know but I’m assuming that software has to be intelligent and do the right thing.
Maybe it would be more accurate to say software has to be "not stupid". We have a fault report, which as usual had been qualified as a network issue, which was due to a server sending out jumbo frames with DF set.
-
. . . which was due to a server sending out jumbo frames with DF set.
A rather big Oops! :doh: