Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 [2] 3 4

Author Topic: 10Gb on a "budget"  (Read 5934 times)

Ronski

  • Helpful
  • Kitizen
  • *
  • Posts: 4300
Re: 10Gb on a "budget"
« Reply #15 on: November 13, 2020, 10:19:36 AM »

Came across another 10Gbps RJ45 card today, no cheaper than other options though.

Syba 1 Port 10 Gigabit Ethernet Network Card - PCIe x4 10Gb 10GBASE-T NIC AQTION AQC107-10Gbps Ethernet PCI-Express x4 Adapter, SD-PEX24055
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: 10Gb on a "budget"
« Reply #16 on: November 14, 2020, 10:04:08 AM »

I am watching this thread with great interest and some jealousy; but almost all the hosts I have now are wireless-only so the speed bottleneck within the LAN lies there, in the WLSN physical layer, not at my HP switch, which is only gigabit. The Raspberry Pis and the Apple TV are wireful now but limited to gigabit too. My Firebrick FB2900 router has one SFP port in it; can’t imagine the router is very fast though as it doesn’t have lots of very expensive custom hardware routing, as far as I know anyway. And all the ethernet copper ports of the FB2900 are gigabit-only, which makes sense as it’s insane to use it as a switch.
Logged

Ronski

  • Helpful
  • Kitizen
  • *
  • Posts: 4300
Re: 10Gb on a "budget"
« Reply #17 on: November 14, 2020, 01:09:57 PM »

My second XG-C100C finally arrived this morning, much to the annoyance of my oldest daughter, the postwomen delivered it at 07:15 and the rest of us slept through the door bell ringing  :) Lucky she heard it really and the postwomen waited a little while. The other one was also delivered at 07:15 the other day, but I was literally about to leave for work, don't think I've ever had post so early.

I've just run some very basic tests - see attached screenshot.

PC 1 has a 512GB Samsung 950 Pro NVME, PC 2 has a 1TB Sabrent Rocket Gen 4 NVME.

I copied 9.5GB of data using Teracopy, with the main file being 9.48GB in size.

At 1Gbps connection via my switch it averaged 112MB/s and took 1 minute 27 seconds.
At 10Gbps with the two PC's connected by a Cat5e patch cable it averaged 396MB/s and took 24 seconds.
At 10Gbps with the two PC's connected via my patch panel it averaged 314MB/s and took 30 seconds.

I was expecting far faster transfers via 10Gbps, the drives are up to it, and the connection was at 10Gbps, could it be errors?

The current patch cables are basic Cat5e cables, I do have some decent Cat6a cables coming. The third test was running through 25 meters of Cat5e a short patch cable at the patch panel, and two cat5e cables to each PC, one being the flat variety. Once fully setup PC 1 will be running over 12 meters of Cat5e, PC 2 over 13 meters approximately using Cat6a patch cables.

Once I get my switch, new cat6a cables and the other card I can try some tests from the server.

I've also discovered that Asus do a SFP+ card XG-C100F
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: 10Gb on a "budget"
« Reply #18 on: November 14, 2020, 01:15:35 PM »

Maybe TCP is just too slow. Or maybe the disk controllers are a bottleneck. Might be worth trying MTU=9000 ? If your kit all supports jumbo frames, that is. Do you have L2 flow control turned on?
Logged

vdsl

  • Member
  • **
  • Posts: 17
Re: 10Gb on a "budget"
« Reply #19 on: November 14, 2020, 03:00:39 PM »

Worth using iperf3 to check bandwidth between the two machines (i.e. removing drives / file copy protocols from the equation)?
Logged

Ronski

  • Helpful
  • Kitizen
  • *
  • Posts: 4300
Re: 10Gb on a "budget"
« Reply #20 on: November 14, 2020, 03:58:01 PM »

@Weaver The SSD's are fine I believe (see attached), no idea about the rest (I'll look into it further when I have the full setup). I was reading from the Samsung to the Sabrent.
@vdsl I have Lan Speed Test setup on the server, so once I've got the switch and the server setup I'll try that. I'll also give Iperf3 a go, tried it once before and didn't get on with it (actually might be confusing it with something else though), not sure why because the video I just watched looks really simple.

« Last Edit: November 14, 2020, 04:02:44 PM by Ronski »
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

Alex Atkin UK

  • Addicted Kitizen
  • *****
  • Posts: 5261
    • Thinkbroadband Quality Monitors
Re: 10Gb on a "budget"
« Reply #21 on: November 14, 2020, 05:36:15 PM »

Maybe TCP is just too slow. Or maybe the disk controllers are a bottleneck. Might be worth trying MTU=9000 ? If your kit all supports jumbo frames, that is. Do you have L2 flow control turned on?

Strange thing I found is an MTU of 6000 performs better than 9000, but that might be due to my desktop being a 5Gbit USB adapter.  It does pull 430MB/s from my NAS, which is pretty good for a USB 3.0 adapter.

I'm not entirely convinced the MTU makes much difference in the great scheme of things though, its mostly intended to reduce CPU overhead from the NIC, when I think its the network file transfer protocols that have the most overhead.
Logged
Broadband: Zen Full Fibre 900 + Three 5G Routers: pfSense (Intel N100) + Huawei CPE Pro 2 H122-373 WiFi: Zyxel NWA210AX
Switches: Netgear MS510TXUP, Netgear MS510TXPP, Netgear GS110EMX My Broadband History & Ping Monitors

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: 10Gb on a "budget"
« Reply #22 on: November 14, 2020, 09:10:21 PM »

I think TCP is rubbish in the supercomputer environment although hardware assist can help. Jumbo frames capability is all very well but the software has to use the MTU not just ignore it I suppose. But I think supercomputer LANs use jumbo frames for a reason and giving TCP an easier time is a clear win, fewer packets lost. You’re probably right about network file transfer protocols. IPv6 over the LAN with a jumbo MTU might be very fast because the IPv6 headers are designed to be fast with their qword alignment of fields which suits modern 64-bit machines well (four 128-bit fetches to get the whole thing) and the simple alignment suits hardware processing too. With a large MTU the byte overhead disadvantage of IPv6 due to the greater header size diminishes from (60-40)/1500 to (60-40)/9000 say. That would be an interesting experiment, but it could be ruined by differences in hardware support for TCP+IPv6 vs TCP+IPv4, and an iperf test with UDP mode would also be interesting as then there would only be hardware support for IP to worry about, even though that’s very much enough. I would have thought that most modern cards that have hardware acceleration for IPv4 would have it for IPv6 too now? Can check ?
Logged

vdsl

  • Member
  • **
  • Posts: 17
Re: 10Gb on a "budget"
« Reply #23 on: November 14, 2020, 11:18:21 PM »

Couple of other thoughts:

* What happens if you start multiple file copy operations in parallel? (or use iperf3 -P option to use parallel streams)?
* Have you tried copying the file via Windows Explorer or command line (e.g. xcopy, robocopy) to rule out Teracopy?
* The Samsung SSD sequential write at 737 MB/s is 5.8 Gbps - not in the picture yet if you're constrained to 396 MB/s (3.1 Gbps), but may come up as you progress.
* Are both machines SMB 3 capable (i.e. Windows 8+ or Server 2012+, unsure on Linux)?
* I assume neither machine is bottlenecked for CPU / RAM?
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: 10Gb on a "budget"
« Reply #24 on: November 14, 2020, 11:58:24 PM »

Good point about the multiple streams idea; and also do the UDP iperf thing as I said before because that almost removes all the software apart from the two operating systems. Confession:  :-[  I once did a UDP iperf over the internet and immediately regretted it; sorry, world!  :-[ :-[ ???  I was (literally) paying money for every dropped packet too.
Logged

Ronski

  • Helpful
  • Kitizen
  • *
  • Posts: 4300
Re: 10Gb on a "budget"
« Reply #25 on: November 15, 2020, 12:59:19 AM »

Couple of other thoughts:

1 What happens if you start multiple file copy operations in parallel? (or use iperf3 -P option to use parallel streams)?
2 Have you tried copying the file via Windows Explorer or command line (e.g. xcopy, robocopy) to rule out Teracopy?
3 The Samsung SSD sequential write at 737 MB/s is 5.8 Gbps - not in the picture yet if you're constrained to 396 MB/s (3.1 Gbps), but may come up as you progress.
4 Are both machines SMB 3 capable (i.e. Windows 8+ or Server 2012+, unsure on Linux)?
5 I assume neither machine is bottlenecked for CPU / RAM?

1 Haven't tried at the moment, may do in the morning if I get time
2 No, but Teracopy is generally considered far better than explorer
4 Both Windows 10 64bit Prof.
5 That'll be a no, the one with the Samsung has 32GB DDR4 2666 quad channel  and a 5820k at 4.5Ghz, the other is a 5900x with 32GB DDR4 at 3800Mhz  ;D
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

Alex Atkin UK

  • Addicted Kitizen
  • *****
  • Posts: 5261
    • Thinkbroadband Quality Monitors
Re: 10Gb on a "budget"
« Reply #26 on: November 15, 2020, 01:25:10 AM »

Oh crap, did we forget the obligatory "try from a Linux Live USB stick"?  ::)
Logged
Broadband: Zen Full Fibre 900 + Three 5G Routers: pfSense (Intel N100) + Huawei CPE Pro 2 H122-373 WiFi: Zyxel NWA210AX
Switches: Netgear MS510TXUP, Netgear MS510TXPP, Netgear GS110EMX My Broadband History & Ping Monitors

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: 10Gb on a "budget"
« Reply #27 on: November 15, 2020, 07:22:05 AM »

What is the performance of SMB3 like compared to the old SMB2 ?
Logged

Ronski

  • Helpful
  • Kitizen
  • *
  • Posts: 4300
Re: 10Gb on a "budget"
« Reply #28 on: November 15, 2020, 09:54:36 AM »

Oh crap, did we forget the obligatory "try from a Linux Live USB stick"?  ::)

Good jobs it was left out, although I have used a Linux live boot USB stick on a few occasions recently I can well imagine having to use some weird seemingly totally random hieroglyphics to run a Iperf speed test across the network  ;)
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

vdsl

  • Member
  • **
  • Posts: 17
Re: 10Gb on a "budget"
« Reply #29 on: November 15, 2020, 11:27:40 AM »

What is the performance of SMB3 like compared to the old SMB2 ?

I don't have numbers, but SMB3 added multichannel support (i.e. multiple concurrent TCP streams and multiple processing threads). I also believe SMB3 is supposed to be a more "chunky" vs "chatty" protocol.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn610980(v=ws.11)#example-configurations

Apparently the CopyFileEx API in Windows was reworked to take advantage of SMB3 features to improve throughput - would be interesting to know whether Teracopy a) uses this API, or b) have applied similar changes themselves.

Also, a registry setting that may be of interest for SMB tuning is Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\DisableLargeMtu - this is defaulted to 0 in Windows 8 only, the docs are a bit ambiguous on Windows 10 behaviour, but I suspect it may be defaulted to 1 (limiting the SMB payload to 64 KB chunks).

https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/

Disclaimer - this is all research, I don't have my 10 Gb LAN yet to test! :)
Logged
Pages: 1 [2] 3 4
 

anything