Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Ixel]

Ixel still over 100ms here, they seem to have transit issues.

I've just tried again and can confirm that I'm now getting this too with them sadly. The problem appears to become noticeable from the 14th hop (10.1.10.19).

Code: [Select]

  8     6 ms     5 ms     5 ms  no-ptr.local [10.1.10.148]
  9     5 ms     5 ms     5 ms  no-ptr.local [10.1.10.69]
 10     4 ms     5 ms     5 ms  no-ptr.local [10.106.66.255]
 .....
 14   100 ms    95 ms   116 ms  no-ptr.local [10.1.10.19]
 15    59 ms    62 ms    67 ms  no-ptr.local [10.1.10.80]
 16   106 ms   106 ms   106 ms  5.180.211.133


Code: [Select]

IP Address:       5.180.211.133
Preferred Route:  UK Dedicated Servers

Upstream Name           Packets Recvd   Avg Latency (ms)        Jitter (ms)
UK Dedicated Servers    59              92                      17
Misaka Networks         58              87                      22
Mythic Beasts           58              92                      17
The Constant Company    59              92                      17

Last Queried: 2022-11-05 18:47:12

They are definitely having some kind of a problem.

[XGS_Is_On]

I wonder how the Mikrotik devices fair with PPPoE added in the mix for a WAN interface?  Maybe they're better at allowing you to allocate things to particular cores perhaps.

I'm not sure whether any of these ARM devices are really powerful enough if you end up doing much in SW (as opposed to offloaded).  As a noddy test I just ran iperf3 as a server on the Dream Machine, and connected to it over the localhost interface from itself.  The max throughput was 12Gb/sec.  Have you tried that out of interest on your Mikrotik?

The CCR2004 doesn't have any offload, it's all CPU. PPPoE seems to have some impact though I couldn't be specific. It's not huge.

I haven't done a loopback iPerf test. I have run iPerf through the CCR at 15 Gbit/s with a fair amount of the CPU spent on I/O wait. It is good for 25 Gbit/s under certain circumstances.

The CCR is now sitting in a corner waiting to go on Fleabay. It's been supplanted by their CCR2116. This beast contains an AL73400 16 core, 2 GHz CPU and a Marvell Aldrin switch chip with some L3 HW offload functionality too. It laughs at what I ask it to do, so I'm hoping to find more for it soon. Screenshot of this endlessly abused bit of kit attached. Busy, busy, busy.

[Chrysalis]

looks real busy.

[Alex Atkin UK]

I wonder if it can run Folding at Home?

[XGS_Is_On]

I wonder if it can run Folding at Home?

Certainly can. Supports containers.

[Chrysalis]

Just had to reboot the server I had tinkered with to get high single threaded gigabit speeds from Germany, Some kernel OOM issues allocating too much memory to TCP buffers. :p

I think I found the cuplrit though, tinkered with one too many things.  Might be a kernel bug though as the memory was stuck allocated even with no open TCP sockets.

[bogof]

The CCR2004 doesn't have any offload, it's all CPU. PPPoE seems to have some impact though I couldn't be specific. It's not huge.

I haven't done a loopback iPerf test. I have run iPerf through the CCR at 15 Gbit/s with a fair amount of the CPU spent on I/O wait. It is good for 25 Gbit/s under certain circumstances.

The CCR is now sitting in a corner waiting to go on Fleabay. It's been supplanted by their CCR2116. This beast contains an AL73400 16 core, 2 GHz CPU and a Marvell Aldrin switch chip with some L3 HW offload functionality too. It laughs at what I ask it to do, so I'm hoping to find more for it soon. Screenshot of this endlessly abused bit of kit attached. Busy, busy, busy.

Be interesting to know what the loopback iperf3 looks like on either of those. 

--

I just built a new router box using a Lenovo M720q Tiny 1L PC and an Intel i340-t4 quad gigabit card; its loopback iperf3 under OpenWRT is 54Gbits/sec - the CPU shouldn't ever become a bottleneck for those adapters! It is languishing around a percent or so CPU usage, only spiking a bit when running a speedtest which does PPPoE, but still no where near maxing out any of the six cores - whereas the Unifi box would be at 100% with gigabit PPPoE.

Thanks to having a separate /29 block via AAISP, I've been able to make the Lenovo box do the PPPoE to AAISP via one of the gigabit ports, and just present an IP connection with a static address from the /29 block to the Unifi on another port, without having to do any double NAT.  This seems to work nicely, it has allowed me a bit more control over the connection than the Unifi box gives me, and reduced the load on it thanks to not having to do gigabit PPPoE anymore. 

This seems like quite an acceptable solution, as I do like the Ubiquiti management side, it's just a bit of a shame the boxes are a bit underwhelming for fat PPPoE connections.

[XGS_Is_On]

Be interesting to know what the loopback iperf3 looks like on either of those. 

Attached from the 2116 using TCP. I up the thread count a bit it still maxes at 70 but averages over 69. Fnarf.

[bogof]

Phwoaarh would you get a load of that throughput...

[bogof]

Though multiple threads on this intel box still has it beat; close to 160Gbits / sec...
Of course, it has naff-all IO though, so kind of useless.  It does show how these ARM boxes can still lag behind in raw grunt against even fairly pedestrian PC HW.

[XGS_Is_On]

Not surprised. The CPU is there to ensure the box can saturate all its ports if L3HW acceleration isn't being used, and can converge a full BGP table quickly.

EDIT: Ah I didn't mention, the number I gave was 70 each way. The Intel still has it beaten of course but I guess not surprising. The ARM CPU is low power, low cost and is 2018 vintage. It's the same CPU in Mikrotik's 2216 router.

[bogof]

Just got a new FTTP 1000/115 connection with Unchained ISP.
Very impressed with single thread performance.

A handful of tests show I'm getting close to line rate single threads to London and Manchester Clouvider hosts, and ~800Mbps to Germany, which in all cases seem to beat the AAISP line I still have running.  Tests done at 11pm on a Saturday night.

Unchained:

Code: [Select]

root@OpenWrt:~# iperf3 -c man.speedtest.clouvider.net -R -p 5208 -t 30 -i 30
Connecting to host man.speedtest.clouvider.net, port 5208
Reverse mode, remote host man.speedtest.clouvider.net is sending
[  5] local 185.250.11.xx port 34260 connected to 103.214.44.130 port 5208
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  3.21 GBytes   919 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.04  sec  3.21 GBytes   918 Mbits/sec   68             sender
[  5]   0.00-30.00  sec  3.21 GBytes   919 Mbits/sec                  receiver

iperf Done.
root@OpenWrt:~# iperf3 -c lon.speedtest.clouvider.net -R -p 5208 -t 30 -i 30
Connecting to host lon.speedtest.clouvider.net, port 5208
Reverse mode, remote host lon.speedtest.clouvider.net is sending
[  5] local 185.250.11.xx port 33474 connected to 5.180.211.133 port 5208
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  3.24 GBytes   929 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.04  sec  3.25 GBytes   928 Mbits/sec   73             sender
[  5]   0.00-30.00  sec  3.24 GBytes   929 Mbits/sec                  receiver

iperf Done.
root@OpenWrt:~# iperf3 -c speedtest.wtnet.de -p 5209 -R -t 30 -i 30
Connecting to host speedtest.wtnet.de, port 5209
Reverse mode, remote host speedtest.wtnet.de is sending
[  5] local 185.250.11.xx port 38750 connected to 213.209.106.95 port 5209
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  2.79 GBytes   800 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.05  sec  2.82 GBytes   805 Mbits/sec  153             sender
[  5]   0.00-30.00  sec  2.79 GBytes   800 Mbits/sec                  receiver

iperf Done.

AAISP:

Code: [Select]

root@OpenWrt:~# iperf3 -c man.speedtest.clouvider.net -R -p 5208 -t 30 -i 30
Connecting to host man.speedtest.clouvider.net, port 5208
Reverse mode, remote host man.speedtest.clouvider.net is sending
[  5] local 81.2.116.xx port 48290 connected to 103.214.44.130 port 5208
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  2.25 GBytes   645 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.04  sec  2.25 GBytes   645 Mbits/sec  373             sender
[  5]   0.00-30.00  sec  2.25 GBytes   645 Mbits/sec                  receiver

iperf Done.
root@OpenWrt:~# iperf3 -c lon.speedtest.clouvider.net -R -p 5208 -t 30 -i 30
Connecting to host lon.speedtest.clouvider.net, port 5208
Reverse mode, remote host lon.speedtest.clouvider.net is sending
[  5] local 81.2.116.xx port 35468 connected to 5.180.211.133 port 5208
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  2.43 GBytes   697 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.04  sec  2.44 GBytes   697 Mbits/sec  604             sender
[  5]   0.00-30.00  sec  2.43 GBytes   697 Mbits/sec                  receiver

iperf Done.
root@OpenWrt:~# iperf3 -c speedtest.wtnet.de -p 5209 -R -t 30 -i 30
Connecting to host speedtest.wtnet.de, port 5209
Reverse mode, remote host speedtest.wtnet.de is sending
[  5] local 81.2.116.xx port 56904 connected to 213.209.106.95 port 5209
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  1.65 GBytes   472 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.04  sec  1.67 GBytes   478 Mbits/sec  1806             sender
[  5]   0.00-30.00  sec  1.65 GBytes   472 Mbits/sec                  receiver

iperf Done.