Kitz Forum

Computers & Hardware => Networking => Topic started by: vdsl on November 07, 2020, 01:00:39 PM

Title: 10Gb on a "budget"
Post by: vdsl on November 07, 2020, 01:00:39 PM
Toying with the idea of moving part of my LAN to 10Gb ethernet. Initially would just be looking to upgrade the link between my home server and desktop PC.

I've run Cat 6 throughout the house already. It's not explicitly marked as Cat 6A, but is rated at 500 MHz rather than 250MHz.

Core switch is a Zyxel GS1920-24HP, which doesn't support 10Gb or SFP+. So I'd need a 10Gb switch between the server and PC, and then uplink would be 1Gb to the Zyxel.

The MikroTik CRS305-1G-4S+IN (https://www.broadbandbuyer.com/products/36054-mikrotik-crs305-1g-4s-in/) (~£115) looks to be one of the cheapest 10Gb switches I can find, and reviews pretty well. Bonus, it can run on PoE from the uplink port.

To convert the SFP+ ports to RJ45 and reuse the existing cabling, I'm thinking of buying a pair of ipolex 10Gb SFP+ RJ45 Module (https://www.amazon.co.uk/ipolex-Transceiver-10GBase-T-SFP-10G-T-S-Supermicro/dp/B078WLTX26/)s (~£50 each).

The ASUS XG-C100C (https://www.amazon.co.uk/ASUS-XG-C100C-PCI-Network-Interface/dp/B071JR2ZW8/) (~£85 each) looks like the cheapest way to get 10GBASE-T on the server and PC. I can't find any combination of SFP+ PCI-e cards and SFP+ modules that come close to that price.

All in, this comes to the guts of £400. Does this seem like a reasonable approach and kit selection?
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 07, 2020, 02:30:51 PM
Funnily enough I've been looking into this recently as well, considering a mix of 2.5Gbps and 10 Gbps to keep costs down - I have a server and two PC's I'd like to connect.

Going SFP+ to RJ45 gets rather expensive, so I was considering a more expensive switch in the first place, but these don't have as many 10Gbps ports, but would allow me to run 1 x 10Gbps with 2 x 2.5Gbps with the future option of 2 x 10Gbps and 1 x 5Gbps if going for the Netgear switches below.

Switches I have on my list so far.
            
£138 ZyXel XGS1010-12   8 x 1Gb, 2 x 2.5Gb and 2 x 10Gb SFP+ unmanaged   
£188 Zyxel XGS1210-12   8 x 1Gb, 2 x 2.5Gb and 2 x 10Gb SFP+ Managed
£208 Netgear MS510TX    4 x 1G, 2 x 1G/2.5G, 2 x 1G/2.5G/5G, 1 x 1G/2.5G/5G/10G, 1 SFP+ 10G Managed
£302 MS510TXPP-100EUS    As above but with POE+               

For connection to the sever (which is next to the switch) I'm considering going with SFP + DAC cable (https://www.amazon.co.uk/10Gtek-10Gb-SFP-Cable-SFP-H10GB-CU2M/dp/B01DCZCP1G), and an SFP+ card (https://www.ebay.co.uk/itm/INTEL-82599ES-Chip-X520-DA1-E10G41BTDA-10G-Dual-Port-SFP-Ethernet-Server-Adapter/153110592016). Both my PC's are on CAT5e, cables are 12 & 13 meters long so should be good for 10Gbps, but will likely run 2.5Gbps to start with.

So I'll be very interested in what suggestion others have as well.

ETA. That card I linked to on eBay is a fake https://sviko.com/t/how-to-distinguish-10-gigabit-network-card-intel-x520-da2-from-chinese-fakes/37
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 07, 2020, 03:15:08 PM
If initially all you want is your home server to your desktop, then the cheapest solution is to simply add a second NIC to both and wire them directly, putting them on a different IP range to the LAN (assign the IP addresses manually as static IPs) and making sure you use that IP for the network shares on your desktop.  This also has the bonus in that you can use Jumbo Frames on that network, while not screwing up the LAN.

I have to use two NICs on my desktop at the moment because its ITX so I can't add a 10Gbit NIC.  So I use the onboard Gigabit for the LAN and 5Gbit for my NAS VLAN.  It basically works the same way even though I only have a single NIC in the NAS (the smart-managed switch effectively makes this behave like two different NICs via VLANs).
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 09, 2020, 10:57:42 AM
Over the weekend two Asus XG-C100C cards accidentally got purchased at an average price of £62.50 each, bit more than I'd like to have paid, especially for secondhand items but there you accidents happen  ;D
Title: Re: 10Gb on a "budget"
Post by: Robbie on November 09, 2020, 12:15:26 PM
Budget 10GbE for me was initially just a Netgear GS110EMX, which was enough to link my main desktop to my main NAS.  This was on the longest run of Cat5e I have at around 27m from my study to network cupboard, but I had zero issues running 10GbE over it.

This was replaced a little later when I bought an ex-press demo UniFi XG-6POE for a very friendly price.  It now links all the 10GbE clients, server and NASs around the house as well as providing a 10GbE SFP+ link to my 48-port switch, which covers all the other 100Mbps & 1GbE stuff.

I only use Cat5e at home as it is far easier to work with and the short distances and low interference levels ensure a good clean 10GbE link.

From the top - 2 routers, 10 GbE switch, patch-panel, 48-port switch with servers and NASs below:

(https://www.disco3.co.uk/gallery/albums/userpics/10700/normal_9-ER3-ER4-UniFi_Switches-Patch_Panel-Mac_IMG_9933_copy.JPG)
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 09, 2020, 01:42:20 PM
Very nice, that would be ideal for my needs, but I just can't justify spending that much, don't seem to be any cheap second hand ones going through ebay either.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 10, 2020, 08:55:49 PM
I've just ordered a second hand Netgear XS508M for 181 Euro (https://www.ebay.co.uk/itm/193738176368) 4 x RJ45 at 1G/2.5G/5G/10G and 10Gb SFP+ unmanaged. It would have been nice to have a managed switch, but I really don't need it.

I narrowly missed out on a used Intel X520-SR1 (variation of the X520-DA1) yesterday for £40, there was four left out of 24, wasn't sure if it would work with a DAC, seller replied back this morning saying it would but they had all sold  :no:

Only need the SFP+ card for my server now and that will be me done once it all arrives.

Just realised after all these years that the exchange rate the eBay use and Paypal use are different. eBay uses the actual exchange rate to give an approximate UK price, Paypal use their rather poor exchange rate that ends up costing quite a bit more than eBays approximate price. Always knew Paypals rate was poor, but never realised eBay used the actual exchange rate.
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 11, 2020, 02:26:57 AM
I've just ordered a second hand Netgear XS508M for 181 Euro (https://www.ebay.co.uk/itm/193738176368) 4 x RJ45 at 1G/2.5G/5G/10G and 10Gb SFP+ unmanaged. It would have been nice to have a managed switch, but I really don't need it.

Have you tried Gigabit to 10Gigabit?  I got something like 200Mbit until I enabled flow control on my switch, which obviously is not an option for an unmanaged switch so would presumably have to be forced on the NICs instead?

The price is that unit new is hilarious, it costs more than the MS510TX smart-managed 8+2 port switch.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 11, 2020, 06:24:33 AM
No I haven't tried 10Gbps yet, the MX510TX only has two 10G capable ports, where as the XS508M has four RJ45 10G ports and one 10G SFP+ ports, there is clearly a big premium for 10g RJ45 ports whichever way you go and very few second hand options, even if you find a second-hand switch they are usually 2/3 of new price or more.
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 11, 2020, 03:49:44 PM
Good point, I clearly was half asleep when I posted.  They obviously expect all ports to be 10Gbit populated which presumably would avoid my issue.

Still, I'm curious to know if unmanaged switches have the same problem my smart managed switch did when moving data between Gigabit and 10Gigbit as it was utterly useless.  Its particularly strange as it was happening across TCP which should have detected the dropped frames and slowed down accordingly.

All the information I found suggested you shouldn't need flow control in 99% of cases, but I found the opposite.  ???
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 11, 2020, 07:35:04 PM
We'll have to wait a week or two to find out, delivery of the switch is estimated between Wednesday 18 Nov - Tuesday 24 Nov from Poland. There was five listed, now there's only one left, selling out very quickly. I've ordered the card for my server as well, a Supermicro AOC-STGN-I2S dual SFP+ card for £60, didn't need dual port, but was getting impatient. Just got to order some CAT 6 patch cables now and that should hopefully be everything I need.

Hopefully all this is proving somewhat useful to the OP as I seem to have hijacked their thread.
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 11, 2020, 11:21:17 PM
Hopefully all this is proving somewhat useful to the OP as I seem to have hijacked their thread.

All useful, I'm waiting eagerly for the review on the Asus XG-C100C cards  :) £62.50 each seems like a bargain, they seem to hold their price very well and I'm struggling to find much under £74!
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 12, 2020, 04:22:52 AM
All useful, I'm waiting eagerly for the review on the Asus XG-C100C cards  :) £62.50 each seems like a bargain, they seem to hold their price very well and I'm struggling to find much under £74!

Darn it, I completely missed that.

It was what I had in my server before I switched it out for Intel when I upgraded the switch (so I could use the SFP+ port), it was fine there.  Also used it for a short while in Windows before I went back to my ITX board because the one I upgraded to had coil whine.

I will be using it again as soon as I can re-build my gaming PC with a new Ryzen.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 12, 2020, 06:21:22 AM
All useful, I'm waiting eagerly for the review on the Asus XG-C100C cards  :) £62.50 each seems like a bargain, they seem to hold their price very well and I'm struggling to find much under £74!

Set up an eBay search for XG-C100C with email notification, set the price range to say £20 to £65, then you'll get notified when one pops up. Mind you looking at the completed sales I seem to have struck lucky and got the only two recent  UK one's under £65.

It would also be worth setting up a search for QXG-10G1T, which is the Qnap card, there's also another but can't think of what that one is.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 12, 2020, 08:49:57 PM
My first Asus card arrived today, along with the DAC cable, would of thought the other card would be here as well, hopefully that will arrive tomorrow.


There is this Asus card on eBay (http://www.ebay.co.uk/itm/284049470050), might be worth an offer and mentioning the other two that recently sold for £60 & £65 including delivery ;)
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 13, 2020, 10:19:36 AM
Came across another 10Gbps RJ45 card today, no cheaper than other options though.

Syba 1 Port 10 Gigabit Ethernet Network Card - PCIe x4 10Gb 10GBASE-T NIC AQTION AQC107-10Gbps Ethernet PCI-Express x4 Adapter, SD-PEX24055
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 14, 2020, 10:04:08 AM
I am watching this thread with great interest and some jealousy; but almost all the hosts I have now are wireless-only so the speed bottleneck within the LAN lies there, in the WLSN physical layer, not at my HP switch, which is only gigabit. The Raspberry Pis and the Apple TV are wireful now but limited to gigabit too. My Firebrick FB2900 router has one SFP port in it; canít imagine the router is very fast though as it doesnít have lots of very expensive custom hardware routing, as far as I know anyway. And all the ethernet copper ports of the FB2900 are gigabit-only, which makes sense as itís insane to use it as a switch.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 14, 2020, 01:09:57 PM
My second XG-C100C finally arrived this morning, much to the annoyance of my oldest daughter, the postwomen delivered it at 07:15 and the rest of us slept through the door bell ringing  :) Lucky she heard it really and the postwomen waited a little while. The other one was also delivered at 07:15 the other day, but I was literally about to leave for work, don't think I've ever had post so early.

I've just run some very basic tests - see attached screenshot.

PC 1 has a 512GB Samsung 950 Pro NVME, PC 2 has a 1TB Sabrent Rocket Gen 4 NVME.

I copied 9.5GB of data using Teracopy, with the main file being 9.48GB in size.

At 1Gbps connection via my switch it averaged 112MB/s and took 1 minute 27 seconds.
At 10Gbps with the two PC's connected by a Cat5e patch cable it averaged 396MB/s and took 24 seconds.
At 10Gbps with the two PC's connected via my patch panel it averaged 314MB/s and took 30 seconds.

I was expecting far faster transfers via 10Gbps, the drives are up to it, and the connection was at 10Gbps, could it be errors?

The current patch cables are basic Cat5e cables, I do have some decent Cat6a cables coming. The third test was running through 25 meters of Cat5e a short patch cable at the patch panel, and two cat5e cables to each PC, one being the flat variety. Once fully setup PC 1 will be running over 12 meters of Cat5e, PC 2 over 13 meters approximately using Cat6a patch cables.

Once I get my switch, new cat6a cables and the other card I can try some tests from the server.

I've also discovered that Asus do a SFP+ card XG-C100F (https://www.asus.com/Networking-IoT-Servers/Wired-Networking/All-series/XG-C100F/)
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 14, 2020, 01:15:35 PM
Maybe TCP is just too slow. Or maybe the disk controllers are a bottleneck. Might be worth trying MTU=9000 ? If your kit all supports jumbo frames, that is. Do you have L2 flow control turned on?
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 14, 2020, 03:00:39 PM
Worth using iperf3 to check bandwidth between the two machines (i.e. removing drives / file copy protocols from the equation)?
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 14, 2020, 03:58:01 PM
@Weaver The SSD's are fine I believe (see attached), no idea about the rest (I'll look into it further when I have the full setup). I was reading from the Samsung to the Sabrent.
@vdsl I have Lan Speed Test setup on the server, so once I've got the switch and the server setup I'll try that. I'll also give Iperf3 a go, tried it once before and didn't get on with it (actually might be confusing it with something else though), not sure why because the video I just watched looks really simple.

Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 14, 2020, 05:36:15 PM
Maybe TCP is just too slow. Or maybe the disk controllers are a bottleneck. Might be worth trying MTU=9000 ? If your kit all supports jumbo frames, that is. Do you have L2 flow control turned on?

Strange thing I found is an MTU of 6000 performs better than 9000, but that might be due to my desktop being a 5Gbit USB adapter.  It does pull 430MB/s from my NAS, which is pretty good for a USB 3.0 adapter.

I'm not entirely convinced the MTU makes much difference in the great scheme of things though, its mostly intended to reduce CPU overhead from the NIC, when I think its the network file transfer protocols that have the most overhead.
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 14, 2020, 09:10:21 PM
I think TCP is rubbish in the supercomputer environment although hardware assist can help. Jumbo frames capability is all very well but the software has to use the MTU not just ignore it I suppose. But I think supercomputer LANs use jumbo frames for a reason and giving TCP an easier time is a clear win, fewer packets lost. Youíre probably right about network file transfer protocols. IPv6 over the LAN with a jumbo MTU might be very fast because the IPv6 headers are designed to be fast with their qword alignment of fields which suits modern 64-bit machines well (four 128-bit fetches to get the whole thing) and the simple alignment suits hardware processing too. With a large MTU the byte overhead disadvantage of IPv6 due to the greater header size diminishes from (60-40)/1500 to (60-40)/9000 say. That would be an interesting experiment, but it could be ruined by differences in hardware support for TCP+IPv6 vs TCP+IPv4, and an iperf test with UDP mode would also be interesting as then there would only be hardware support for IP to worry about, even though thatís very much enough. I would have thought that most modern cards that have hardware acceleration for IPv4 would have it for IPv6 too now? Can check ?
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 14, 2020, 11:18:21 PM
Couple of other thoughts:

* What happens if you start multiple file copy operations in parallel? (or use iperf3 -P option to use parallel streams)?
* Have you tried copying the file via Windows Explorer or command line (e.g. xcopy, robocopy) to rule out Teracopy?
* The Samsung SSD sequential write at 737 MB/s is 5.8 Gbps - not in the picture yet if you're constrained to 396 MB/s (3.1 Gbps), but may come up as you progress.
* Are both machines SMB 3 capable (i.e. Windows 8+ or Server 2012+, unsure on Linux)?
* I assume neither machine is bottlenecked for CPU / RAM?
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 14, 2020, 11:58:24 PM
Good point about the multiple streams idea; and also do the UDP iperf thing as I said before because that almost removes all the software apart from the two operating systems. Confession:  :-[  I once did a UDP iperf over the internet and immediately regretted it; sorry, world!  :-[ :-[ ???  I was (literally) paying money for every dropped packet too.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 15, 2020, 12:59:19 AM
Couple of other thoughts:

1 What happens if you start multiple file copy operations in parallel? (or use iperf3 -P option to use parallel streams)?
2 Have you tried copying the file via Windows Explorer or command line (e.g. xcopy, robocopy) to rule out Teracopy?
3 The Samsung SSD sequential write at 737 MB/s is 5.8 Gbps - not in the picture yet if you're constrained to 396 MB/s (3.1 Gbps), but may come up as you progress.
4 Are both machines SMB 3 capable (i.e. Windows 8+ or Server 2012+, unsure on Linux)?
5 I assume neither machine is bottlenecked for CPU / RAM?

1 Haven't tried at the moment, may do in the morning if I get time
2 No, but Teracopy is generally considered far better than explorer
4 Both Windows 10 64bit Prof.
5 That'll be a no, the one with the Samsung has 32GB DDR4 2666 quad channel  and a 5820k at 4.5Ghz, the other is a 5900x with 32GB DDR4 at 3800Mhz  ;D
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 15, 2020, 01:25:10 AM
Oh crap, did we forget the obligatory "try from a Linux Live USB stick"?  ::)
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 15, 2020, 07:22:05 AM
What is the performance of SMB3 like compared to the old SMB2 ?
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 15, 2020, 09:54:36 AM
Oh crap, did we forget the obligatory "try from a Linux Live USB stick"?  ::)

Good jobs it was left out, although I have used a Linux live boot USB stick on a few occasions recently I can well imagine having to use some weird seemingly totally random hieroglyphics to run a Iperf speed test across the network  ;)
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 15, 2020, 11:27:40 AM
What is the performance of SMB3 like compared to the old SMB2 ?

I don't have numbers, but SMB3 added multichannel support (i.e. multiple concurrent TCP streams and multiple processing threads). I also believe SMB3 is supposed to be a more "chunky" vs "chatty" protocol.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn610980(v=ws.11)#example-configurations

Apparently the CopyFileEx API in Windows was reworked to take advantage of SMB3 features to improve throughput - would be interesting to know whether Teracopy a) uses this API, or b) have applied similar changes themselves.

Also, a registry setting that may be of interest for SMB tuning is Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\DisableLargeMtu - this is defaulted to 0 in Windows 8 only, the docs are a bit ambiguous on Windows 10 behaviour, but I suspect it may be defaulted to 1 (limiting the SMB payload to 64 KB chunks).

https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/

Disclaimer - this is all research, I don't have my 10 Gb LAN yet to test! :)
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 15, 2020, 12:45:58 PM
@vdsl I'll leave you to test things like that  ;)

Results of running Iperf3 are much better.

First set is via my 1Gbps switch and the hardwired Cat5e network cables.

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 31478 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   112 MBytes   938 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  4]   2.00-3.00   sec   112 MBytes   944 Mbits/sec
[  4]   3.00-4.00   sec   112 MBytes   943 Mbits/sec
[  4]   4.00-5.00   sec   112 MBytes   943 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   947 Mbits/sec
[  4]   6.00-7.00   sec   109 MBytes   917 Mbits/sec
[  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  4]   8.00-9.00   sec   110 MBytes   925 Mbits/sec
[  4]   9.00-10.00  sec   110 MBytes   923 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  receiver

iperf Done.

Second set is via a patch cable between the two PC's

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 22514 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1009 MBytes  8.47 Gbits/sec
[  4]   1.00-2.00   sec  1008 MBytes  8.46 Gbits/sec
[  4]   2.00-3.00   sec  1005 MBytes  8.43 Gbits/sec
[  4]   3.00-4.00   sec  1014 MBytes  8.50 Gbits/sec
[  4]   4.00-5.00   sec  1017 MBytes  8.53 Gbits/sec
[  4]   5.00-6.00   sec  1017 MBytes  8.53 Gbits/sec
[  4]   6.00-7.00   sec  1016 MBytes  8.53 Gbits/sec
[  4]   7.00-8.00   sec  1010 MBytes  8.47 Gbits/sec
[  4]   8.00-9.00   sec  1007 MBytes  8.45 Gbits/sec
[  4]   9.00-10.00  sec  1020 MBytes  8.55 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  9.89 GBytes  8.49 Gbits/sec                  sender
[  4]   0.00-10.00  sec  9.89 GBytes  8.49 Gbits/sec                  receiver

iperf Done.

Last set is via a short patch cable to link the ports on the patch panel - so roughly 25 meters of cat5e

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 32015 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   997 MBytes  8.36 Gbits/sec
[  4]   1.00-2.00   sec  1010 MBytes  8.47 Gbits/sec
[  4]   2.00-3.00   sec  1010 MBytes  8.47 Gbits/sec
[  4]   3.00-4.00   sec  1000 MBytes  8.39 Gbits/sec
[  4]   4.00-5.00   sec  1002 MBytes  8.41 Gbits/sec
[  4]   5.00-6.00   sec   981 MBytes  8.23 Gbits/sec
[  4]   6.00-7.00   sec   990 MBytes  8.30 Gbits/sec
[  4]   7.00-8.00   sec  1003 MBytes  8.41 Gbits/sec
[  4]   8.00-9.00   sec  1013 MBytes  8.49 Gbits/sec
[  4]   9.00-10.00  sec  1010 MBytes  8.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  9.78 GBytes  8.40 Gbits/sec                  sender
[  4]   0.00-10.00  sec  9.78 GBytes  8.40 Gbits/sec                  receiver

iperf Done.

Overall a lot better than yesterdays basic test, but ultimately it will be read/write speeds of my server that will be the limiting factor.

Almost forgot, attached a screenshot of Task Manager showing 1159.4Mbps whilst running Iperf3 at 10Gbps but shouldn't it be MB/s ? Definitely something not right there, if I run Iperf3 across the 1Gbps network Task Manager reports just under 900Mbps  ???
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 15, 2020, 04:58:49 PM
Nice, good to see that the TCP throughput is much closer to 10 Gbps! :)

Did you try iperf3 with additional streams (e.g. -P 2, -P 5)? Would be interesting to see if it can be pushed closer to the theoretical max or if you're hitting cable / card limits.

Not sure what's going on with Task Manager there, I'd agree it would seem to be showing the wrong unit.

@vdsl I'll leave you to test things like that  ;)

I'll report back when I manage to bag a couple of NICs for a decent price! :)
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 15, 2020, 06:35:12 PM
Did you try iperf3 with additional streams (e.g. -P 2, -P 5)? Would be interesting to see if it can be pushed closer to the theoretical max or if you're hitting cable / card limits.

No I didn't, it's not something I'm familiar with so wasn't aware?


Quote
I'll report back when I manage to bag a couple of NICs for a decent price! :)

Can you use SFP+ cards and DAC cables?
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 15, 2020, 06:41:52 PM
I would have tested mine but I, umm, can't remember where I put it after taking it out of my desktop when I returned my motherboard.  :-[
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 15, 2020, 07:00:10 PM
No I didn't, it's not something I'm familiar with so wasn't aware?

Fair enough, you can use "-P" or "--parallel" from the client, e.g.

Code: [Select]
iperf3 -c <host or IP> -P 2

Note that it's case sensitive, and "-p" means something else entirely (sets the port number IIRC).

I'd try a couple of values, e.g. 2, 5, 10 and see what (if any) effect it has.

Can you use SFP+ cards and DAC cables?

Haven't been able to find an SFP+ option that comes close to the price of the Asus 10GBASE-T cards. It's still a possibility at this stage. Long term I'm planning to move the server into a different room, hoping to reuse my existing cat 6 cables. So I'd probably be looking at 10GBASE-T SFP+ modules anyway, which seem to be about £50 each and only advertise working over 30m of cat 6 (10GbE spec suggests 55m should be possible). I think 30m would be okay, but maybe cutting a little close for my longer runs (if just patching together at the panel).
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 15, 2020, 07:11:43 PM
Definitely try UDP mode, as I said earlier.
Title: Re: 10Gb on a "budget"
Post by: vdsl on November 15, 2020, 07:16:10 PM
For UDP mode, use:

Code: [Select]
iperf3 -c <host or IP> -u -b 10G

(-u is UDP, -b is the bandwidth limit, which for UDP, defaults to 1 Mbps)
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 15, 2020, 08:56:37 PM
Haven't been able to find an SFP+ option that comes close to the price of the Asus 10GBASE-T cards. It's still a possibility at this stage. Long term I'm planning to move the server into a different room, hoping to reuse my existing cat 6 cables. So I'd probably be looking at 10GBASE-T SFP+ modules anyway, which seem to be about £50 each and only advertise working over 30m of cat 6 (10GbE spec suggests 55m should be possible). I think 30m would be okay, but maybe cutting a little close for my longer runs (if just patching together at the panel).

If you don't mind going second hand then there a quite a few options for SFP+, I missed out on an Intel X520-SR1 for £40, which appears to be identical to an X520-DA1 because I wasn't sure if it would work with a DAC cable, but by all accounts it would. Though if you're going to be moving the server and already have CAT6 installed then that would be easier, luckily for me my server is right next to all my main networking gear. Apparently https://www.fs.com/uk/ is pretty good for cables and transceivers.

Thanks for the info on Iperf3, I'll give that a go later in the week when the other stuff has arrived.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 18, 2020, 09:13:41 PM
My switch turned up today, to all intents and purposes it looks brand new, it came with new cables, new pack of ears, new instructions, and proper foam packaging, just wasn't in a Netgear box.

Need to tidy up the cabling, I bought a splitter cable for the two switches, but can't unplug the other switch to change it over at the moment or I'll get complaints.

(https://i.postimg.cc/L5YXf0Qh/2020-11-18-20-19-09-Copy.jpg) (https://postimg.cc/SY4kbVhp)


Some quick speed tests as requested:

This one is rather slow??

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10 -u -b 10G
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 63054 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   158 MBytes  1.32 Gbits/sec  20201
[  4]   1.00-2.00   sec   161 MBytes  1.35 Gbits/sec  20604
[  4]   2.00-3.00   sec   161 MBytes  1.35 Gbits/sec  20623
[  4]   3.00-4.00   sec   164 MBytes  1.38 Gbits/sec  21039
[  4]   4.00-5.00   sec   164 MBytes  1.38 Gbits/sec  20987
[  4]   5.00-6.00   sec   165 MBytes  1.38 Gbits/sec  21059
[  4]   6.00-7.00   sec   159 MBytes  1.33 Gbits/sec  20318
[  4]   7.00-8.00   sec   165 MBytes  1.38 Gbits/sec  21107
[  4]   8.00-9.00   sec   164 MBytes  1.38 Gbits/sec  20996
[  4]   9.00-10.00  sec   162 MBytes  1.36 Gbits/sec  20716
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.58 GBytes  1.36 Gbits/sec  0.002 ms  0/207649 (0%)
[  4] Sent 207649 datagrams

iperf Done.

Oddly running through the switch and with CAT6A patch cables its a bit slower than before, mind you one of the PC's updated to Window 10 20H2 the other day.

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 8979 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   949 MBytes  7.96 Gbits/sec
[  4]   1.00-2.00   sec   909 MBytes  7.62 Gbits/sec
[  4]   2.00-3.00   sec   872 MBytes  7.31 Gbits/sec
[  4]   3.00-4.00   sec   943 MBytes  7.91 Gbits/sec
[  4]   4.00-5.00   sec   940 MBytes  7.88 Gbits/sec
[  4]   5.00-6.00   sec   929 MBytes  7.79 Gbits/sec
[  4]   6.00-7.00   sec   904 MBytes  7.58 Gbits/sec
[  4]   7.00-8.00   sec   932 MBytes  7.82 Gbits/sec
[  4]   8.00-9.00   sec   900 MBytes  7.55 Gbits/sec
[  4]   9.00-10.00  sec   934 MBytes  7.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  9.00 GBytes  7.73 Gbits/sec                  sender
[  4]   0.00-10.00  sec  9.00 GBytes  7.73 Gbits/sec                  receiver

iperf Done.

Much closer to 10Gbps in parallel though, although the second time I ran it was 7.5Gbps

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10 -P 2
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 9120 connected to 192.168.0.10 port 5201
[  6] local 192.168.0.11 port 9121 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   562 MBytes  4.71 Gbits/sec
[  6]   0.00-1.00   sec   549 MBytes  4.60 Gbits/sec
[SUM]   0.00-1.00   sec  1.08 GBytes  9.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec   563 MBytes  4.72 Gbits/sec
[  6]   1.00-2.00   sec   554 MBytes  4.65 Gbits/sec
[SUM]   1.00-2.00   sec  1.09 GBytes  9.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec   560 MBytes  4.70 Gbits/sec
[  6]   2.00-3.00   sec   554 MBytes  4.65 Gbits/sec
[SUM]   2.00-3.00   sec  1.09 GBytes  9.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   600 MBytes  5.04 Gbits/sec
[  6]   3.00-4.00   sec   502 MBytes  4.21 Gbits/sec
[SUM]   3.00-4.00   sec  1.08 GBytes  9.25 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec   634 MBytes  5.32 Gbits/sec
[  6]   4.00-5.00   sec   449 MBytes  3.77 Gbits/sec
[SUM]   4.00-5.00   sec  1.06 GBytes  9.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec   536 MBytes  4.50 Gbits/sec
[  6]   5.00-6.00   sec   548 MBytes  4.60 Gbits/sec
[SUM]   5.00-6.00   sec  1.06 GBytes  9.10 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec   522 MBytes  4.38 Gbits/sec
[  6]   6.00-7.00   sec   508 MBytes  4.26 Gbits/sec
[SUM]   6.00-7.00   sec  1.01 GBytes  8.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec   590 MBytes  4.95 Gbits/sec
[  6]   7.00-8.00   sec   514 MBytes  4.31 Gbits/sec
[SUM]   7.00-8.00   sec  1.08 GBytes  9.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   555 MBytes  4.65 Gbits/sec
[  6]   8.00-9.00   sec   558 MBytes  4.68 Gbits/sec
[SUM]   8.00-9.00   sec  1.09 GBytes  9.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec   564 MBytes  4.73 Gbits/sec
[  6]   9.00-10.00  sec   553 MBytes  4.64 Gbits/sec
[SUM]   9.00-10.00  sec  1.09 GBytes  9.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  5.55 GBytes  4.77 Gbits/sec                  sender
[  4]   0.00-10.00  sec  5.55 GBytes  4.77 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  5.16 GBytes  4.44 Gbits/sec                  sender
[  6]   0.00-10.00  sec  5.16 GBytes  4.44 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.7 GBytes  9.21 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  10.7 GBytes  9.21 Gbits/sec                  receiver

iperf Done.

The PC with 20H2 version of Windows actually reports the network speed correctly.

Just waiting on the card for the server now, Royal Mail 2nd class seems to take for ever now :-(
Title: Re: 10Gb on a "budget"
Post by: Weaver on November 18, 2020, 10:23:18 PM
Is that an ONT, Ronski?

And a small NAS or Intel NUC or similar?

And Iím wondering what the brightly coloured thing on the left is.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 18, 2020, 10:45:11 PM
The brightly coloured thing is  a Dyson cordless vacuum. The black object bottom centre is my UPS, on the left end of the shelf is my pfSense router (a Qotom QOTOM-Q190G4-S02 Mini PC) , other end of the shelf is the Virgin SH3 and my server below, which has a quad satellite tuner for TV recording. I still have a redundant HG612 screwed to the wall, hopefully I'll never need that again.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 20, 2020, 09:12:35 PM
Finally received my SFP+ card today, it wasn't supposed to include any transceivers, I guess someone hadn't noticed the two that were plugged in to it. Both Intel units, a FTLX8571D3BCV-IT and a AFBR-703SDZ-IN2, I may just have to get a cable and see if they work.

(https://i.postimg.cc/r0mdKtj3/2020-11-20-18-44-09-Copy.jpg) (https://postimg.cc/r0mdKtj3)

and the card which is a Supermicro AOC-STGN-I2S Rev 1

(https://i.postimg.cc/LgP81k1C/2020-11-20-20-55-08-Copy.jpg) (https://postimg.cc/LgP81k1C)

Tomorrow I'll install it and see what happens.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 21, 2020, 04:52:47 PM
I installed the card this morning and it worked without installing any drivers, but Iperf3 reported only about 2Gbps transfer speeds.

Went through some fun and games trying to update the drivers, but got there in the end, but that didn't seem to make much difference, no idea why I'm not seeing the speeds I did the other day between the two PC's with the Asus cards either.

From the X99 system to the server I get:

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.20
Connecting to host 192.168.0.20, port 5201
[  4] local 192.168.0.11 port 2154 connected to 192.168.0.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   319 MBytes  2.68 Gbits/sec
[  4]   1.00-2.00   sec   350 MBytes  2.93 Gbits/sec
[  4]   2.00-3.00   sec   362 MBytes  3.03 Gbits/sec
[  4]   3.00-4.00   sec   349 MBytes  2.93 Gbits/sec
[  4]   4.00-5.00   sec   378 MBytes  3.17 Gbits/sec
[  4]   5.00-6.00   sec   357 MBytes  2.99 Gbits/sec
[  4]   6.00-7.00   sec   360 MBytes  3.02 Gbits/sec
[  4]   7.00-8.00   sec   369 MBytes  3.09 Gbits/sec
[  4]   8.00-9.00   sec   377 MBytes  3.16 Gbits/sec
[  4]   9.00-10.00  sec   367 MBytes  3.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.50 GBytes  3.01 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.50 GBytes  3.01 Gbits/sec                  receiver

iperf Done.

The X570 pc to server achieves more

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.20
Connecting to host 192.168.0.20, port 5201
[  4] local 192.168.0.10 port 61192 connected to 192.168.0.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   447 MBytes  3.75 Gbits/sec
[  4]   1.00-2.00   sec   562 MBytes  4.71 Gbits/sec
[  4]   2.00-3.00   sec   559 MBytes  4.69 Gbits/sec
[  4]   3.00-4.00   sec   569 MBytes  4.77 Gbits/sec
[  4]   4.00-5.00   sec   584 MBytes  4.90 Gbits/sec
[  4]   5.00-6.00   sec   582 MBytes  4.88 Gbits/sec
[  4]   6.00-7.00   sec   574 MBytes  4.82 Gbits/sec
[  4]   7.00-8.00   sec   559 MBytes  4.69 Gbits/sec
[  4]   8.00-9.00   sec   556 MBytes  4.66 Gbits/sec
[  4]   9.00-10.00  sec   576 MBytes  4.83 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  5.44 GBytes  4.67 Gbits/sec                  sender
[  4]   0.00-10.00  sec  5.44 GBytes  4.67 Gbits/sec                  receiver

iperf Done.

and X99 to X570 system

Code: [Select]
C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  4] local 192.168.0.11 port 33823 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   624 MBytes  5.23 Gbits/sec
[  4]   1.00-2.00   sec   611 MBytes  5.13 Gbits/sec
[  4]   2.00-3.00   sec   657 MBytes  5.51 Gbits/sec
[  4]   3.00-4.00   sec   657 MBytes  5.52 Gbits/sec
[  4]   4.00-5.00   sec   648 MBytes  5.43 Gbits/sec
[  4]   5.00-6.00   sec   657 MBytes  5.51 Gbits/sec
[  4]   6.00-7.00   sec   657 MBytes  5.51 Gbits/sec
[  4]   7.00-8.00   sec   675 MBytes  5.67 Gbits/sec
[  4]   8.00-9.00   sec   668 MBytes  5.60 Gbits/sec
[  4]   9.00-10.00  sec   680 MBytes  5.70 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  6.38 GBytes  5.48 Gbits/sec                  sender
[  4]   0.00-10.00  sec  6.38 GBytes  5.48 Gbits/sec                  receiver

iperf Done.

When I run Iperf3 in parallel it does perform better, but still not as high as it did the other day.
 
File transfers to/from the server do seem roughly 3 times faster though.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 22, 2020, 11:14:36 AM
I found this guide last night for updating the firmware on the Asus cards, I've updated one, but haven't done the other yet, I'll also install the Marvell drivers later. There is also some tips for improving the through put as well, which I'll give a try later.

https://www.rashedtalukder.com/updating-asus-xg-c100cs-aquantia-aqc107-firmware-and-drivers/

This thread also looks useful, lists some newer firmware and drivers https://www.overclock.net/threads/aquantia-10g-lan-firmware-update-aqc107-3-1-90.1735798/
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 22, 2020, 07:29:29 PM
I don't recall having any issues once flow control was enabled, but maybe this firmware actually fixes the bug that caused me to need flow control in the first place?
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 22, 2020, 10:15:58 PM
Was that on Linux or Windows Alex?
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 28, 2020, 06:08:04 PM
I received my LC to LC OM3 fibre cable today, and the transceivers work, so I'm now running fibre between the server and switch.

I've moved the cards around, both in my X99 system and the server as the cards were in slots not able to fully utilise maximum speed, but it made no difference to my Iperf3 results. I  think it's a Windows thing, and may be also Iperf3, I found some talk of it having some problems on Windows. At some point I'll try Linux Live USB stick on the two desktop systems.

The motherboard on the server only has PCIe 2 slots (it is 10 years old), but as far as I can work out the x8 slot the NIC is in should be plenty fast enough. I'm considering changing the motherboard, and at some point installing Windows 10 Pro on there instead of WHS2011 which is now out of support.
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 28, 2020, 09:13:22 PM
Was that on Linux or Windows Alex?

I think I tested on both but to be safe I will assume Linux, the NAS I was testing against would always be Linux though.

https://forum.kitz.co.uk/index.php/topic,24581.msg413332.html
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 28, 2020, 10:51:35 PM
I'm now hitting around 9Gbps between the two Windows 10 machines, but getting substantially lower to the server, between 4 to 6Gbps. This is with Jumbo Packets of 9014 bytes on all three machines.

The server is about 10 years old, running WHS2011 on an i3 540 CPU, but I don't see any increase in CPU usage during the tests. The NIC is a Supermicro AOC-STGN-I2S SFP+ which uses an Intel 82599 chipset. It's plugged into the x16 socket, but is a x8 card, socket is PCIe v2.

Code: [Select]
Microsoft Windows [Version 10.0.19042.630]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.0.11 port 29224 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   998 MBytes  8.37 Gbits/sec
[  5]   1.00-2.00   sec  1.06 GBytes  9.12 Gbits/sec
[  5]   2.00-3.00   sec  1.00 GBytes  8.62 Gbits/sec
[  5]   3.00-4.00   sec   980 MBytes  8.22 Gbits/sec
[  5]   4.00-5.00   sec  1.05 GBytes  9.03 Gbits/sec
[  5]   5.00-6.00   sec  1.08 GBytes  9.25 Gbits/sec
[  5]   6.00-7.00   sec  1.09 GBytes  9.36 Gbits/sec
[  5]   7.00-8.00   sec  1.09 GBytes  9.32 Gbits/sec
[  5]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec
[  5]   9.00-10.00  sec  1.04 GBytes  8.95 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.4 GBytes  8.96 Gbits/sec                  sender
[  5]   0.00-10.05  sec  10.4 GBytes  8.91 Gbits/sec                  receiver

iperf Done.

C:\Temp>iperf3 -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.0.11 port 29228 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.02 GBytes  8.78 Gbits/sec
[  5]   1.00-2.00   sec  1.07 GBytes  9.19 Gbits/sec
[  5]   2.00-3.00   sec  1.06 GBytes  9.10 Gbits/sec
[  5]   3.00-4.00   sec  1.02 GBytes  8.75 Gbits/sec
[  5]   4.00-5.00   sec  1.12 GBytes  9.60 Gbits/sec
[  5]   5.00-6.00   sec  1.07 GBytes  9.15 Gbits/sec
[  5]   6.00-7.00   sec  1018 MBytes  8.54 Gbits/sec
[  5]   7.00-8.00   sec  1.02 GBytes  8.76 Gbits/sec
[  5]   8.00-9.00   sec  1.09 GBytes  9.32 Gbits/sec
[  5]   9.00-10.00  sec  1.11 GBytes  9.57 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.6 GBytes  9.08 Gbits/sec                  sender
[  5]   0.00-10.04  sec  10.6 GBytes  9.04 Gbits/sec                  receiver

iperf Done.

C:\Temp>iperf3 -c 192.168.0.10 -P2
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.0.11 port 29230 connected to 192.168.0.10 port 5201
[  7] local 192.168.0.11 port 29231 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   569 MBytes  4.76 Gbits/sec
[  7]   0.00-1.00   sec   574 MBytes  4.80 Gbits/sec
[SUM]   0.00-1.00   sec  1.12 GBytes  9.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   558 MBytes  4.69 Gbits/sec
[  7]   1.00-2.00   sec   570 MBytes  4.79 Gbits/sec
[SUM]   1.00-2.00   sec  1.10 GBytes  9.48 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   576 MBytes  4.83 Gbits/sec
[  7]   2.00-3.00   sec   583 MBytes  4.89 Gbits/sec
[SUM]   2.00-3.00   sec  1.13 GBytes  9.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   545 MBytes  4.58 Gbits/sec
[  7]   3.00-4.00   sec   546 MBytes  4.59 Gbits/sec
[SUM]   3.00-4.00   sec  1.07 GBytes  9.16 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   575 MBytes  4.82 Gbits/sec
[  7]   4.00-5.00   sec   571 MBytes  4.78 Gbits/sec
[SUM]   4.00-5.00   sec  1.12 GBytes  9.60 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   553 MBytes  4.64 Gbits/sec
[  7]   5.00-6.00   sec   585 MBytes  4.90 Gbits/sec
[SUM]   5.00-6.00   sec  1.11 GBytes  9.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   544 MBytes  4.56 Gbits/sec
[  7]   6.00-7.00   sec   552 MBytes  4.63 Gbits/sec
[SUM]   6.00-7.00   sec  1.07 GBytes  9.20 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   543 MBytes  4.56 Gbits/sec
[  7]   7.00-8.00   sec   537 MBytes  4.51 Gbits/sec
[SUM]   7.00-8.00   sec  1.05 GBytes  9.07 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   519 MBytes  4.35 Gbits/sec
[  7]   8.00-9.00   sec   484 MBytes  4.07 Gbits/sec
[SUM]   8.00-9.00   sec  1004 MBytes  8.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   551 MBytes  4.62 Gbits/sec
[  7]   9.00-10.00  sec   541 MBytes  4.54 Gbits/sec
[SUM]   9.00-10.00  sec  1.07 GBytes  9.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  5.40 GBytes  4.64 Gbits/sec                  sender
[  5]   0.00-10.04  sec  5.40 GBytes  4.62 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  5.41 GBytes  4.65 Gbits/sec                  sender
[  7]   0.00-10.04  sec  5.41 GBytes  4.63 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.8 GBytes  9.29 Gbits/sec                  sender
[SUM]   0.00-10.04  sec  10.8 GBytes  9.25 Gbits/sec                  receiver

iperf Done.

C:\Temp>iperf3 -c 192.168.0.20
Connecting to host 192.168.0.20, port 5201
[  5] local 192.168.0.11 port 29256 connected to 192.168.0.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   338 MBytes  2.83 Gbits/sec
[  5]   1.00-2.00   sec   414 MBytes  3.48 Gbits/sec
[  5]   2.00-3.00   sec   409 MBytes  3.43 Gbits/sec
[  5]   3.00-4.00   sec   419 MBytes  3.52 Gbits/sec
[  5]   4.00-5.00   sec   403 MBytes  3.38 Gbits/sec
[  5]   5.00-6.00   sec   426 MBytes  3.58 Gbits/sec
[  5]   6.00-7.00   sec   356 MBytes  2.99 Gbits/sec
[  5]   7.00-8.00   sec   376 MBytes  3.15 Gbits/sec
[  5]   8.00-9.00   sec   384 MBytes  3.22 Gbits/sec
[  5]   9.00-10.00  sec   408 MBytes  3.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.84 GBytes  3.30 Gbits/sec                  sender
[  5]   0.00-10.03  sec  3.84 GBytes  3.29 Gbits/sec                  receiver

iperf Done.

C:\Temp>iperf3 -c 192.168.0.20 -P4
Connecting to host 192.168.0.20, port 5201
[  5] local 192.168.0.11 port 29258 connected to 192.168.0.20 port 5201
[  7] local 192.168.0.11 port 29259 connected to 192.168.0.20 port 5201
[  9] local 192.168.0.11 port 29260 connected to 192.168.0.20 port 5201
[ 11] local 192.168.0.11 port 29261 connected to 192.168.0.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   156 MBytes  1.31 Gbits/sec
[  7]   0.00-1.00   sec   152 MBytes  1.28 Gbits/sec
[  9]   0.00-1.00   sec   140 MBytes  1.18 Gbits/sec
[ 11]   0.00-1.00   sec   137 MBytes  1.15 Gbits/sec
[SUM]   0.00-1.00   sec   586 MBytes  4.92 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   194 MBytes  1.63 Gbits/sec
[  7]   1.00-2.00   sec   185 MBytes  1.55 Gbits/sec
[  9]   1.00-2.00   sec   163 MBytes  1.37 Gbits/sec
[ 11]   1.00-2.00   sec   159 MBytes  1.33 Gbits/sec
[SUM]   1.00-2.00   sec   701 MBytes  5.88 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   199 MBytes  1.67 Gbits/sec
[  7]   2.00-3.00   sec   184 MBytes  1.55 Gbits/sec
[  9]   2.00-3.00   sec   163 MBytes  1.37 Gbits/sec
[ 11]   2.00-3.00   sec   161 MBytes  1.35 Gbits/sec
[SUM]   2.00-3.00   sec   708 MBytes  5.94 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   207 MBytes  1.73 Gbits/sec
[  7]   3.00-4.00   sec   186 MBytes  1.56 Gbits/sec
[  9]   3.00-4.00   sec   163 MBytes  1.37 Gbits/sec
[ 11]   3.00-4.00   sec   153 MBytes  1.29 Gbits/sec
[SUM]   3.00-4.00   sec   709 MBytes  5.95 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   187 MBytes  1.57 Gbits/sec
[  7]   4.00-5.00   sec   183 MBytes  1.54 Gbits/sec
[  9]   4.00-5.00   sec   126 MBytes  1.06 Gbits/sec
[ 11]   4.00-5.00   sec   123 MBytes  1.03 Gbits/sec
[SUM]   4.00-5.00   sec   620 MBytes  5.20 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   218 MBytes  1.83 Gbits/sec
[  7]   5.00-6.00   sec   200 MBytes  1.68 Gbits/sec
[  9]   5.00-6.00   sec   160 MBytes  1.34 Gbits/sec
[ 11]   5.00-6.00   sec   154 MBytes  1.29 Gbits/sec
[SUM]   5.00-6.00   sec   732 MBytes  6.14 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   200 MBytes  1.68 Gbits/sec
[  7]   6.00-7.00   sec   172 MBytes  1.44 Gbits/sec
[  9]   6.00-7.00   sec   165 MBytes  1.38 Gbits/sec
[ 11]   6.00-7.00   sec   157 MBytes  1.31 Gbits/sec
[SUM]   6.00-7.00   sec   694 MBytes  5.82 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   196 MBytes  1.65 Gbits/sec
[  7]   7.00-8.00   sec   181 MBytes  1.52 Gbits/sec
[  9]   7.00-8.00   sec   160 MBytes  1.34 Gbits/sec
[ 11]   7.00-8.00   sec   157 MBytes  1.31 Gbits/sec
[SUM]   7.00-8.00   sec   694 MBytes  5.82 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   211 MBytes  1.77 Gbits/sec
[  7]   8.00-9.00   sec   192 MBytes  1.61 Gbits/sec
[  9]   8.00-9.00   sec   154 MBytes  1.29 Gbits/sec
[ 11]   8.00-9.00   sec   150 MBytes  1.26 Gbits/sec
[SUM]   8.00-9.00   sec   708 MBytes  5.94 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   213 MBytes  1.79 Gbits/sec
[  7]   9.00-10.00  sec   192 MBytes  1.61 Gbits/sec
[  9]   9.00-10.00  sec   167 MBytes  1.40 Gbits/sec
[ 11]   9.00-10.00  sec   164 MBytes  1.37 Gbits/sec
[SUM]   9.00-10.00  sec   736 MBytes  6.16 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.94 GBytes  1.66 Gbits/sec                  sender
[  5]   0.00-10.01  sec  1.94 GBytes  1.66 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.78 GBytes  1.53 Gbits/sec                  sender
[  7]   0.00-10.01  sec  1.78 GBytes  1.53 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.53 GBytes  1.31 Gbits/sec                  sender
[  9]   0.00-10.01  sec  1.52 GBytes  1.31 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec                  sender
[ 11]   0.00-10.01  sec  1.48 GBytes  1.27 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  6.73 GBytes  5.78 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  6.73 GBytes  5.77 Gbits/sec                  receiver

iperf Done.

C:\Temp>iperf3 -c 192.168.0.20 -P8
Connecting to host 192.168.0.20, port 5201
[  5] local 192.168.0.11 port 29266 connected to 192.168.0.20 port 5201
[  7] local 192.168.0.11 port 29267 connected to 192.168.0.20 port 5201
[  9] local 192.168.0.11 port 29268 connected to 192.168.0.20 port 5201
[ 11] local 192.168.0.11 port 29269 connected to 192.168.0.20 port 5201
[ 13] local 192.168.0.11 port 29270 connected to 192.168.0.20 port 5201
[ 15] local 192.168.0.11 port 29271 connected to 192.168.0.20 port 5201
[ 17] local 192.168.0.11 port 29272 connected to 192.168.0.20 port 5201
[ 19] local 192.168.0.11 port 29273 connected to 192.168.0.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  68.4 MBytes   573 Mbits/sec
[  7]   0.00-1.00   sec  66.6 MBytes   559 Mbits/sec
[  9]   0.00-1.00   sec   132 MBytes  1.10 Gbits/sec
[ 11]   0.00-1.00   sec   143 MBytes  1.20 Gbits/sec
[ 13]   0.00-1.00   sec  66.5 MBytes   558 Mbits/sec
[ 15]   0.00-1.00   sec  62.8 MBytes   526 Mbits/sec
[ 17]   0.00-1.00   sec  63.8 MBytes   535 Mbits/sec
[ 19]   0.00-1.00   sec  60.9 MBytes   510 Mbits/sec
[SUM]   0.00-1.00   sec   663 MBytes  5.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  76.9 MBytes   645 Mbits/sec
[  7]   1.00-2.00   sec  74.0 MBytes   621 Mbits/sec
[  9]   1.00-2.00   sec   153 MBytes  1.28 Gbits/sec
[ 11]   1.00-2.00   sec   146 MBytes  1.22 Gbits/sec
[ 13]   1.00-2.00   sec  85.0 MBytes   713 Mbits/sec
[ 15]   1.00-2.00   sec  73.0 MBytes   613 Mbits/sec
[ 17]   1.00-2.00   sec  75.1 MBytes   630 Mbits/sec
[ 19]   1.00-2.00   sec  75.4 MBytes   632 Mbits/sec
[SUM]   1.00-2.00   sec   758 MBytes  6.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  83.5 MBytes   700 Mbits/sec
[  7]   2.00-3.00   sec  79.5 MBytes   667 Mbits/sec
[  9]   2.00-3.00   sec   146 MBytes  1.22 Gbits/sec
[ 11]   2.00-3.00   sec   129 MBytes  1.08 Gbits/sec
[ 13]   2.00-3.00   sec  81.6 MBytes   685 Mbits/sec
[ 15]   2.00-3.00   sec  80.2 MBytes   673 Mbits/sec
[ 17]   2.00-3.00   sec  78.0 MBytes   654 Mbits/sec
[ 19]   2.00-3.00   sec  82.5 MBytes   692 Mbits/sec
[SUM]   2.00-3.00   sec   760 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  78.0 MBytes   654 Mbits/sec
[  7]   3.00-4.00   sec  75.4 MBytes   632 Mbits/sec
[  9]   3.00-4.00   sec   145 MBytes  1.22 Gbits/sec
[ 11]   3.00-4.00   sec   134 MBytes  1.13 Gbits/sec
[ 13]   3.00-4.00   sec  76.9 MBytes   645 Mbits/sec
[ 15]   3.00-4.00   sec  72.8 MBytes   610 Mbits/sec
[ 17]   3.00-4.00   sec  74.2 MBytes   623 Mbits/sec
[ 19]   3.00-4.00   sec  69.9 MBytes   586 Mbits/sec
[SUM]   3.00-4.00   sec   727 MBytes  6.10 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  78.1 MBytes   655 Mbits/sec
[  7]   4.00-5.00   sec  63.5 MBytes   533 Mbits/sec
[  9]   4.00-5.00   sec   163 MBytes  1.37 Gbits/sec
[ 11]   4.00-5.00   sec   148 MBytes  1.24 Gbits/sec
[ 13]   4.00-5.00   sec  80.6 MBytes   676 Mbits/sec
[ 15]   4.00-5.00   sec  74.4 MBytes   624 Mbits/sec
[ 17]   4.00-5.00   sec  76.8 MBytes   644 Mbits/sec
[ 19]   4.00-5.00   sec  74.2 MBytes   623 Mbits/sec
[SUM]   4.00-5.00   sec   759 MBytes  6.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  78.8 MBytes   659 Mbits/sec
[  7]   5.00-6.00   sec  74.8 MBytes   625 Mbits/sec
[  9]   5.00-6.00   sec   143 MBytes  1.19 Gbits/sec
[ 11]   5.00-6.00   sec   135 MBytes  1.13 Gbits/sec
[ 13]   5.00-6.00   sec  79.8 MBytes   667 Mbits/sec
[ 15]   5.00-6.00   sec  73.0 MBytes   611 Mbits/sec
[ 17]   5.00-6.00   sec  76.8 MBytes   642 Mbits/sec
[ 19]   5.00-6.00   sec  74.1 MBytes   620 Mbits/sec
[SUM]   5.00-6.00   sec   734 MBytes  6.14 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  74.9 MBytes   630 Mbits/sec
[  7]   6.00-7.00   sec  66.0 MBytes   555 Mbits/sec
[  9]   6.00-7.00   sec   157 MBytes  1.32 Gbits/sec
[ 11]   6.00-7.00   sec   141 MBytes  1.19 Gbits/sec
[ 13]   6.00-7.00   sec  74.6 MBytes   628 Mbits/sec
[ 15]   6.00-7.00   sec  68.1 MBytes   573 Mbits/sec
[ 17]   6.00-7.00   sec  71.6 MBytes   603 Mbits/sec
[ 19]   6.00-7.00   sec  70.2 MBytes   591 Mbits/sec
[SUM]   6.00-7.00   sec   723 MBytes  6.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  77.9 MBytes   653 Mbits/sec
[  7]   7.00-8.00   sec  76.6 MBytes   642 Mbits/sec
[  9]   7.00-8.00   sec   152 MBytes  1.27 Gbits/sec
[ 11]   7.00-8.00   sec   144 MBytes  1.21 Gbits/sec
[ 13]   7.00-8.00   sec  79.4 MBytes   666 Mbits/sec
[ 15]   7.00-8.00   sec  73.9 MBytes   619 Mbits/sec
[ 17]   7.00-8.00   sec  75.6 MBytes   634 Mbits/sec
[ 19]   7.00-8.00   sec  73.6 MBytes   617 Mbits/sec
[SUM]   7.00-8.00   sec   753 MBytes  6.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  82.5 MBytes   692 Mbits/sec
[  7]   8.00-9.00   sec  83.4 MBytes   700 Mbits/sec
[  9]   8.00-9.00   sec   144 MBytes  1.21 Gbits/sec
[ 11]   8.00-9.00   sec   146 MBytes  1.22 Gbits/sec
[ 13]   8.00-9.00   sec  79.6 MBytes   668 Mbits/sec
[ 15]   8.00-9.00   sec  72.9 MBytes   611 Mbits/sec
[ 17]   8.00-9.00   sec  72.1 MBytes   605 Mbits/sec
[ 19]   8.00-9.00   sec  75.0 MBytes   629 Mbits/sec
[SUM]   8.00-9.00   sec   755 MBytes  6.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  81.6 MBytes   684 Mbits/sec
[  7]   9.00-10.00  sec  77.6 MBytes   650 Mbits/sec
[  9]   9.00-10.00  sec   153 MBytes  1.28 Gbits/sec
[ 11]   9.00-10.00  sec   150 MBytes  1.26 Gbits/sec
[ 13]   9.00-10.00  sec  82.1 MBytes   688 Mbits/sec
[ 15]   9.00-10.00  sec  73.8 MBytes   618 Mbits/sec
[ 17]   9.00-10.00  sec  79.1 MBytes   663 Mbits/sec
[ 19]   9.00-10.00  sec  74.1 MBytes   621 Mbits/sec
[SUM]   9.00-10.00  sec   771 MBytes  6.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   780 MBytes   655 Mbits/sec                  sender
[  5]   0.00-10.01  sec   780 MBytes   654 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   737 MBytes   618 Mbits/sec                  sender
[  7]   0.00-10.01  sec   737 MBytes   618 Mbits/sec                  receiver
[  9]   0.00-10.00  sec  1.45 GBytes  1.25 Gbits/sec                  sender
[  9]   0.00-10.01  sec  1.45 GBytes  1.25 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  sender
[ 11]   0.00-10.01  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec   786 MBytes   659 Mbits/sec                  sender
[ 13]   0.00-10.01  sec   786 MBytes   659 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   725 MBytes   608 Mbits/sec                  sender
[ 15]   0.00-10.01  sec   724 MBytes   607 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec   743 MBytes   623 Mbits/sec                  sender
[ 17]   0.00-10.01  sec   743 MBytes   623 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   730 MBytes   612 Mbits/sec                  sender
[ 19]   0.00-10.01  sec   730 MBytes   612 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  7.23 GBytes  6.21 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  7.23 GBytes  6.20 Gbits/sec                  receiver

iperf Done.

C:\Temp>
Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 28, 2020, 11:16:17 PM
I'd definitely try booting the server from a Linux USB stick, just to confirm its not a hardware problem.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 29, 2020, 11:28:22 AM
I've now tested with Linux Mint on the server and get slightly better results, but still no where near what it should be.

Code: [Select]
Iperf3 Windows 10 to Linux Mint on my server.

Microsoft Windows [Version 10.0.19042.630]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Temp>iperf3 -c 192.168.0.209
Connecting to host 192.168.0.209, port 5201
[  5] local 192.168.0.11 port 9274 connected to 192.168.0.209 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   530 MBytes  4.44 Gbits/sec
[  5]   1.00-2.00   sec   442 MBytes  3.71 Gbits/sec
[  5]   2.00-3.00   sec   458 MBytes  3.84 Gbits/sec
[  5]   3.00-4.00   sec   484 MBytes  4.06 Gbits/sec
[  5]   4.00-5.00   sec   544 MBytes  4.57 Gbits/sec
[  5]   5.00-6.00   sec   448 MBytes  3.76 Gbits/sec
[  5]   6.00-7.00   sec   401 MBytes  3.36 Gbits/sec
[  5]   7.00-8.01   sec   427 MBytes  3.56 Gbits/sec
[  5]   8.01-9.00   sec   446 MBytes  3.77 Gbits/sec
[  5]   9.00-10.00  sec   473 MBytes  3.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.54 GBytes  3.90 Gbits/sec                  sender
[  5]   0.00-10.05  sec  4.54 GBytes  3.88 Gbits/sec                  receiver

iperf Done.

C:\Temp>iperf3 -c 192.168.0.209 -P2
Connecting to host 192.168.0.209, port 5201
[  5] local 192.168.0.11 port 9276 connected to 192.168.0.209 port 5201
[  7] local 192.168.0.11 port 9277 connected to 192.168.0.209 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   335 MBytes  2.79 Gbits/sec
[  7]   0.00-1.01   sec   335 MBytes  2.79 Gbits/sec
[SUM]   0.00-1.01   sec   670 MBytes  5.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.01-2.00   sec   338 MBytes  2.85 Gbits/sec
[  7]   1.01-2.00   sec   337 MBytes  2.84 Gbits/sec
[SUM]   1.01-2.00   sec   674 MBytes  5.70 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   334 MBytes  2.80 Gbits/sec
[  7]   2.00-3.00   sec   297 MBytes  2.49 Gbits/sec
[SUM]   2.00-3.00   sec   631 MBytes  5.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   332 MBytes  2.79 Gbits/sec
[  7]   3.00-4.00   sec   326 MBytes  2.74 Gbits/sec
[SUM]   3.00-4.00   sec   658 MBytes  5.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   301 MBytes  2.52 Gbits/sec
[  7]   4.00-5.00   sec   299 MBytes  2.51 Gbits/sec
[SUM]   4.00-5.00   sec   600 MBytes  5.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   336 MBytes  2.81 Gbits/sec
[  7]   5.00-6.00   sec   331 MBytes  2.78 Gbits/sec
[SUM]   5.00-6.00   sec   667 MBytes  5.59 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   345 MBytes  2.89 Gbits/sec
[  7]   6.00-7.00   sec   341 MBytes  2.86 Gbits/sec
[SUM]   6.00-7.00   sec   686 MBytes  5.75 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   296 MBytes  2.49 Gbits/sec
[  7]   7.00-8.00   sec   292 MBytes  2.45 Gbits/sec
[SUM]   7.00-8.00   sec   589 MBytes  4.94 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   330 MBytes  2.77 Gbits/sec
[  7]   8.00-9.00   sec   328 MBytes  2.75 Gbits/sec
[SUM]   8.00-9.00   sec   658 MBytes  5.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   348 MBytes  2.92 Gbits/sec
[  7]   9.00-10.00  sec   338 MBytes  2.83 Gbits/sec
[SUM]   9.00-10.00  sec   686 MBytes  5.74 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.22 GBytes  2.76 Gbits/sec                  sender
[  5]   0.00-10.05  sec  3.22 GBytes  2.75 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  3.15 GBytes  2.70 Gbits/sec                  sender
[  7]   0.00-10.05  sec  3.15 GBytes  2.69 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  6.37 GBytes  5.47 Gbits/sec                  sender
[SUM]   0.00-10.05  sec  6.36 GBytes  5.44 Gbits/sec                  receiver

iperf Done.

C:\Temp>
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 29, 2020, 12:43:36 PM
Actually this is all getting rather confusing, Iperf3 from the X99 to the server generally seems to be slower than from the X570 PC, and I've just discovered if I do -R -P2 from the X570 I can hit an average of 8.4Gbps and even 9.37Gbps, then another time it will barely manage 4.16Gbps, most odd.
Title: Re: 10Gb on a "budget"
Post by: Ronski on November 29, 2020, 05:36:43 PM
I run a more windows optimised test, which equates to around 9Gbps  ;D

Code: [Select]
C:\NTttcp>ntttcp.exe -s -m 8,*,192.168.0.10 -l 128k -a 2 -t 15

Copyright Version 5.33

Network activity progressing...



Thread  Time(s) Throughput(KB/s) Avg B / Compl

======  ======= ================ =============

     0   14.999       144496.033    131072.000

     1   15.000       144989.867    131072.000

     2   14.999       144922.728    131072.000

     3   14.999       144956.864    131072.000

     4   15.000       145058.133    131072.000

     5   14.999       144538.703    131072.000

     6   14.999       145221.415    131072.000

     7   14.999       144547.236    131072.000



#####  Totals:  #####



   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)

================ =========== ============== ================

    16972.750000      15.000       1456.187         1131.517



Throughput(Buffers/s) Cycles/Byte       Buffers

===================== =========== =============

             9052.133       2.438    135782.000



DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)

============= ============= =============== ==============

    16160.867         2.489       21234.200          1.894



Packets Sent Packets Received Retransmits Errors Avg. CPU %

============ ================ =========== ====== ==========

    12221792           603255          16      8     23.582


C:\NTttcp>

Title: Re: 10Gb on a "budget"
Post by: Alex Atkin UK on November 29, 2020, 07:30:26 PM
iperf3 can be weird sometimes, I have seen it report lower than actual file transfers before.