Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Chrysalis]

Hi guys, an extension to the discussion on the Zen thread, too early to point fingers anywhere, but I have my story to tell.

Due to some talk talk issues (which AAISP did resolve with talk talk after I put in migration order) I was perhaps been impatient and got myself switched from TTw to BTw.

During the first evening I noticed issues streaming youtube videos, so did a few speedtests.

This evening everything was slow I tested, including the AAISP libre speedtest.  I couldnt be bothered so went to sleep and resumed testing early morning before ringing AAISP.

During the morning AAISP's speedtester was fast again, thinkbroadband had pretty bad single threaded still which surprised me as I was expecting it to be fast outside of peak, and multi threaded was line rate.

I rang AAISP and had to explain it wasnt me loading my network or wifi, after that was asked to run AAISP's speedtest and fast.com speedtest, they did ask me to drop to 2-4 threads on fast.com, but I told them I was going down to 1 and tested it.  At single threaded it was faster but still sporadic, ranging from about 40 to just below line speed from test to test.  When I told them the download file links on the AAISP speedtest page were slow it wasnt really taken serious.  It was explained to me, my graph showed no signs of congestion and I was asked to do more tests and they arranged a DLM reset.

I remembered to do some more tests last night, and this time AAISP speedtester stayed fast as did TBB multi threaded, youtube also worked ok, but the test files and TBB single threaded were very slow under 3mbit/second, and fast.com just about got over 3mbit/sec.

I then analysed the AAISP speedtester traffic and noticed it actually is indeed multi threaded.

Rang them again today and had to be more pushy as I started been asked questions like "is normal stuff affected", now this is the thing, on talk talk business backhaul, as I never noticed problems, I didnt speedtest the line, I didnt check tbb single threaded, and I didnt download the aaisp test files.  I can not say with certainty it didnt have these issues.  But at the same time nothing went wrong to make me feel the need to check.

I have since done some more testing.

Downloading a file from one of my own web servers and also single threaded FTP seems fine, so not everything affected (if that was slow then I would have known something has changed), however only on my windows box, its slow to a linux shell wget/fetch.
Downloading a file from heztner's speedtest page is really extremely bad, anywhere from about 300kbit/second up to about 30mbit/second, odd as my server I tested from is hosted at hetzner.  Not rwin issues which would flatline at buffer limit but is very up and down like would get under congestion.
AAISP's own files remain all over the place never higher then about 30-40mbit can be as low as about 3. https://speedtest.aa.net.uk/100MB.zip

I at this point wanted to rule out my networking equipment, so I connected my mobile phone and enabled tethering to PFSENSE, so its basically going through all the same hardware same cables, same switches.  All I am merely doing is changing the gateway to use on the policy routing.

On the phone as gateway.

Hetzner files download at 120mbit/second single threaded.
AAISP files a little faster single threaded.
TBB 80mbit single threaded, 110 multi.

The only way I will know if TTw does fix it is if AAISP agree to switch me back, I hinted that I would accept it, but they didnt bite.  The DLM looks like its been done already as well modem is syncing at 80mbit with these issues.

I might run things for a while with my phone as gateway and starting to wonder if I can ditch the DSL until CF comes to town.

[underzone]

Very interesting, thanks. Good idea using cellular to test everything easily in one go.

[bogof]

For what it's worth, while I don't hit my Zen line rate downloading that AAISP file single threaded (which I'm sure will be a Zen issue), I do get well above what you manage.  I'll definitely give it a go once I migrate to AAISP.
I see there is also a 1GB.zip file available.

Code: [Select]

root@Home-Dream-Machine-SE:/ssd1/test# rm 100MB.zip ; time axel -U Chrome -n 1 -v -a -o 100MB.zip https://speedtest.aa.net.uk/100MB.zip
Initializing download: https://speedtest.aa.net.uk/100MB.zip
File size: 104857600 bytes
Opening output file 100MB.zip
Starting download

[100%] [..................................................] [  57.9MB/s] [00:00]

Downloaded 100.0 Megabyte in 1 second. (59332.46 KB/s)

real    0m1.772s
user    0m0.464s
sys     0m0.826s


[Chrysalis]

I think the single threaded issues are a hard sell, the current investigating is the cause of some day time packet loss that has started appearing.

This packet loss has a strange pattern it appears, has fairly high consistent loss for a while, then stops, then comes back stronger, then stops, repeat until it peaks, then repeat with it fading away, its happened the last two days in a row.  No SNR changes on the modem whilst its there or error spikes.

AAISP are sending me a router to do some monitoring, I dont know how it works but I will connect that when it arrives.

I have my own ideas I am going to test as well.

The very first day time period after the BTw migration did "not" have this packet loss, a DLM reset was done early morning the second day which bumped the sync to 80megs and lowered the SNRM from over 8db to about 6.8db.  That day and the day after (today) had the weird packet loss behaviour.

I did a little thinking about the heztner issue, their server been slow, mine been fast, both from same network, my server uses bbr congestion control which resists certain types of loss, youtube is also fine single threaded (google use bbr, they invented it I think).  Sure enough I have now confirmed bbr is masking the problem, as soon as I switched to cubic I get similar speeds to hetzner's test files.  bbr is amazing.

This would indicate there is perhaps some residual low level packet loss going on, meaning investigating the packet loss spikes in the day time might potentially fix this as well if they have the same cause.

Attaching my most recent test notes in the below box. Sorry they messy.

Code: [Select]

21 oct 8.37pm - all done on windows 10 high end gaming PC with intel gigabit ethernet lan, note results for some reason are even slower in linux but only slower on the dsl, mobile downloads in linux match windows speeds.
EE mobile is configured in pfsense using same hardware, and same firewall rules, same lan cables, traffic going through same switches.
run 1 aaisp speedtester (multi threaded) 56.4 down 19.8 up
run 2 aaisp speedtester 59.0 down 19.8 up
run 3 aaisp speedtester 61.7 down 19.9 up - 3 runs couldnt hit line rate which is 70+
aaisp speedtester EE mobile phone tethered 117 down 67 up
tbb speedtester 18.1 down single threaded, line rate multi threaded (75.6), only did one run as aaisp dont seem to accept this as evidence.
tbb speedtester EE mobile phone tethered, 94 down single, 119 down multi.
aaisp speedtest files (single threaded), jumping up and down between 5 and 10.4mbps
aaisp speedtest files EE mobile phone tethered around 110mbps
hetzner speedtest files single threaded 2.4 to 5.7mbps
hetzner speedtest file EE mobile phone tethered 96mbps
file download over ftp single threaded from one of my servers hosted at hetzner bbr congestion control, line rate 70+
file download over ftp single threaded from one of my servers hosted at hetzner cubic congrestion control, 2.3 to 5.7mbps, bingo bbr masks the problem, explains why youtube is now ok and fast.com not affected as much.

[Chrysalis]

For what it's worth, while I don't hit my Zen line rate downloading that AAISP file single threaded (which I'm sure will be a Zen issue), I do get well above what you manage.  I'll definitely give it a go once I migrate to AAISP.
I see there is also a 1GB.zip file available.

Code: [Select]

root@Home-Dream-Machine-SE:/ssd1/test# rm 100MB.zip ; time axel -U Chrome -n 1 -v -a -o 100MB.zip https://speedtest.aa.net.uk/100MB.zip
Initializing download: https://speedtest.aa.net.uk/100MB.zip
File size: 104857600 bytes
Opening output file 100MB.zip
Starting download

[100%] [..................................................] [  57.9MB/s] [00:00]

Downloaded 100.0 Megabyte in 1 second. (59332.46 KB/s)

real    0m1.772s
user    0m0.464s
sys     0m0.826s


If you curious I can give you a download link for a bbr hosted file, as I am curious if it masks Zen's problems, let me know.

[bogof]

i think that has a lot of legs.  One of the things that the Zen guys said to me previously was that on the GEA network the Samknows box they sent was showing up some small level of upstream packet loss.

[Chrysalis]

I dropped my sync speed to around the 74mbit mark to get the SNRM back, but no affect on the dropped TCP packets, AAISP did request me as part of using the test router to also isolate the power source, but I wont be able to test that until I get a new cable probably earliest delivery Sunday.  So there is nothing more I can locally test for now I think other than the modem swap which is what I will be doing probably when the test router arrives.

[Weaver]

I keep going on about my hatred of TCP-based speed testers and that means most speed testers in general. AA’s speed tester has sometimes come out with some literally impossible upstream figures for me, too high, at a level that is above the sync rate when allowing for DSL overheads. For some reasons it has varied in this behaviour. Currently fast.com reads 4.3 Mbps upstream, which is about 3 times,  dry roughly, more than reality.

Is there a possibility of doing an iperf test in udp mode ?

[Chrysalis]

I can do one, but most of my single threaded tests are not using speedtesters except fast.com at aaisp's request, aaisp are asking me to do speedtests, but I prefer to do manual downloads single threaded over http or ftp I also then watch the live traffic rate rather than what the client reports.  As you said speedtesters especially multi threaded ones dont necessarily show a good picture.

[Chrysalis]

Ok UDP test 1 parallel thread.

I waited until it was showing 0% loss for best possible FTTC result.

This was ran in cli from pfsense's own shell to a server I manage which was running iperf3 in server mode, I ran it in reverse so a download test.

Code: [Select]

EE

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.25 MBytes  27.2 Mbits/sec                 
[  5]   1.00-2.00   sec  6.07 MBytes  50.9 Mbits/sec                 
[  5]   2.00-3.00   sec  7.62 MBytes  63.9 Mbits/sec                 
[  5]   3.00-4.00   sec  8.39 MBytes  70.4 Mbits/sec                 
[  5]   4.00-5.00   sec  8.27 MBytes  69.4 Mbits/sec                 
[  5]   5.00-6.00   sec  8.07 MBytes  67.7 Mbits/sec                 
[  5]   6.00-7.00   sec  8.50 MBytes  71.3 Mbits/sec                 
[  5]   7.00-8.00   sec  8.60 MBytes  72.2 Mbits/sec                 
[  5]   8.00-9.00   sec  8.16 MBytes  68.4 Mbits/sec                 
[  5]   9.00-10.00  sec  8.14 MBytes  68.3 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.35  sec  79.6 MBytes  64.5 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  75.1 MBytes  63.0 Mbits/sec                  receiver

AAISP

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  66.0 KBytes   540 Kbits/sec                 
[  5]   1.00-2.00   sec  17.0 KBytes   139 Kbits/sec                 
[  5]   2.00-3.01   sec  18.3 KBytes   149 Kbits/sec                 
[  5]   3.01-4.00   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   4.00-5.00   sec  41.4 KBytes   339 Kbits/sec                 
[  5]   5.00-6.00   sec  42.8 KBytes   351 Kbits/sec                 
[  5]   6.00-7.00   sec  65.9 KBytes   540 Kbits/sec                 
[  5]   7.00-8.00   sec   152 KBytes  1.25 Mbits/sec                 
[  5]   8.00-9.00   sec   101 KBytes   831 Kbits/sec                 
[  5]   9.00-10.00  sec   153 KBytes  1.25 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.24  sec   856 KBytes   684 Kbits/sec  287             sender
[  5]   0.00-10.00  sec   658 KBytes   539 Kbits/sec                  receiver

[Chrysalis]

Sorry that was tcp, but I cannot enable the udp flag, if you know what causes this error, I will try it again, it shows up when adding the udp flag.

"iperf3: error - unable to start stream listener: Unknown error: 98"

[Weaver]

I would say that speed testers that load up the internet connection more fully by using for example six TCP connections will be quite good relatively, as they really do have a better chance of getting no bubbles in the pipe. In my case I want 4 * (n+1) (where n ≥ 1) TCP connections in use so that will fill my four pipes up well. The +1 is just there to ensure that I get more than one TCP connection per pipe otherwise there will often be a chance of bubbles, especially given my higher rtt than many in the UK. So for me, 8 TCP connections is much better than six and 12 or 16 TCP connections better still.

The designers of speedtesters still then need to make a sane choice about correcting their numbers so that they add in the overhead from TCP and IPv4/6 headers, as without doing that their figures will under-read. They also need to say whether it’s IPv4 or IPv6 in use and whether or not there are TCP timestamps in use in each direction. Timestamps are worth what 12/1500 so what’s that? - 0.8% error if they forget about this. Forgetting TCP+IPvx overhead is either 40/1500 = 2.66% [?] or 60/1500 = 4% overhead forgotten, so their numbers will be too low by these not entirely trivial amounts. The omission of the necessary declarations of what protocols are in use and how they’re choosing to quote the figures immediately tells us that the authors don’t know what they’re doing. Forgetting about 4.8%, in the worst case, isn’t great.

Remember that Andrew, one of the AA general managers is a member of the forum so you can PM him if you need any more help. I can dig out his email address if needed.

[Chrysalis]

The aim here isnt to get as fast as a result as possible I know I can max the line by adding threads.  The problem specifically been abnormal single threaded performance due to excessive dropped packets.

I suppose a way to put it is I dont care about small bubbles air pockets, but this is more like a big empty tub taking most of the room.

Your issues might be unique to bonding? its not normal to need to load a session with 4 threads to saturate a relatively low capacity line such as yours or even a medium capacity line such as mine.  So if you getting issues on single threaded testing thats highlighting a problem to you rather than testing with one thread been the problem.

Andrew didnt respond to the DM I sent a couple of weeks ago, and I already have a open support line with AAISP.

[Chrysalis]

Ok UDP is working now.

So from what I understand, iperf UDP is just forcing the packets without any type of flow management, and it reports the amount that get dropped/lost en route.

I set the target rate as 75000K, and it seemed fine.

The packet loss spikes also appeared to have stopped for now, but tcp single threaded remains slow with "something" preventing the congestion window growing on cubic and other loss based congestion providers.

I dont think there is much else to be done here until I get the test router, and I think the priority will be dealing with whatever is causing those packet loss spikes.

[bogof]

I keep going on about my hatred of TCP-based speed testers and that means most speed testers in general. AA’s speed tester has sometimes come out with some literally impossible upstream figures for me, too high, at a level that is above the sync rate when allowing for DSL overheads. For some reasons it has varied in this behaviour. Currently fast.com reads 4.3 Mbps upstream, which is about 3 times,  dry roughly, more than reality.

Is there a possibility of doing an iperf test in udp mode ?

Seems like an illogical hatred.  The speed test to hate is only the one that poorly represents the use-case you will use, or ones that are poorly implemented and give defective results.
If you want to download single files quickly, a UDP test is pointless. 

I still think one of the best tests you can do for most use cases is using a lightweight command line browser like axel and downloading either single or multiple threads depending on what you're trying to see.  Probably followed by using a TCP speedtester from the command line.  The heavyweight desktop environment adds lots of complexities and opacity.