Kitz Forum

Broadband Related => Broadband Technology => Topic started by: Weaver on June 29, 2020, 01:27:53 PM

Title: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 29, 2020, 01:27:53 PM
I ran the Andrews and Arnold LibreSpeed speed tester to look at ‘real’ upstream TCP payload (I think, just guessing, payload rather than IP PDUs, so the IP payload which is the meaningful value as regards assessing the link itself will be somewhat higher than this value.) An amazing value of 1.86 Mbps upstream was obtained, the all time record by a long way. The link speeds are a bit odd in that link #1 is about 100kbps faster than has ever been seen before. looking at the predicted totals, the speed obtained should be impossible as it is above the 96% MLF and even the 100% MLF predicted values (terms are defined below). The measured rate is 105.52% of the prediction at current MLF, and 101.30% of the rate at MLF of 100%.

How can an such a large improvement in efficiency be obtained? Seems like magic!

If I add a further amount still in order to account for TCP headers and either IPv4 or IPv6 headers and possibly TCP timestamps used in the AA speed tester, then that makes the mystery even harder to resolve. Mind you, the speed tester could already be tweaking the results by adding an extra amount to allow for the data sent in the headers on top of the payload values, but that would require introspection and a detailed knowledge of the actual L4/L3 protocols in use at the time of each test even including MTU values.


============ * Current upstream speed rates used by Firebrick * ============
The Firebrick's internet access links have the following upstream speeds (expressed as IP PDU rates) right now:
  #1: 546792 bps
  #2: 450849 bps
  #3: 321792 bps
  #4: 443207 bps
Total combined rate: 1.762640 Mbps

Fractional speed contributions:
  #1: 31.021%  [██████████████████████████████████████████████████]
  #2: 25.578%  [█████████████████████████████████████████ ‒‒‒‒‒‒‒‒]
  #3: 18.256%  [█████████████████████████████ ‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒]
  #4: 25.144%  [█████████████████████████████████████████ ‒‒‒‒‒‒‒‒]

- These are the upstream data rates on each link sent by the Firebrick given as IP PDU rates. These values are always well below the upstream sync rate of each link because of overheads due to protocol layers below layer 3. The reduced upstream rate given here is the theoretical maximum calculated for a link in the most optimistic possible scenario, one based on sending large packets of a certain chosen size. The quoted upstream rate is also further reduced by multiplication by a certain chosen 'modem loading factor', whose value is based on experience, tuned for best upstream performance.



Breakdown and detailed predictions :
LinkSync upstream   MLF        Protocol efficiency   Fudge factor      IP rate       Max rate (100%)    Fractional contribution
L1  64400096%0.8844339620.849056604   546792   56957531.0212%
L2  53100096%0.8844339620.849056604   450849   46963425.5781%
L3  37900096%0.8844339620.849056604   321792   33520018.2563%
L4  52200096%0.8844339620.849056604   443207   46167425.1445%
Totals:  207600017626401836083

Test date: 2020-06-27.

* Column headings explained:
* The Modem Loading Factor (MLF) is an upstream egress rate multiplier chosen to maximise throughout without overloading the link.
* Fudge Factor is defined as the MLF times the protocol efficiency; the overall multiplier to convert sync rate into IP PDU egress bit rate at the actual chosen MLF.
* The ‘Max Rate’ (100%) is the IP PDU egress bit rate that would be the case if the modem loading factor (MLF) were set to 100%.
* The ‘IP rate’ is the IP PDU egress bit rate at the actual chosen modem loading factor (96% in this case).
* The ‘fractional contribution’ is the contribution of each link rate as a fraction of the total.



Live sync rates of modems currently (obtained by querying modems directly):
  #1: down 2719 kbps, up 644 kbps
  #2: down 2716 kbps, up 531 kbps
  #3: down 2825 kbps, up 379 kbps
  #4: down 2828 kbps, up 522 kbps

Notice that line 1 sync rate upstream is freakishly high, no modem has hit such a high u/s value ever before. Line 3 is as usual really slow, a continuing problem. Lines 2, 4 are as normal, as expected.



There is a large variance in the upstream rates reported by this speed tester, as with other testers, nothing unusual about that. other upstream results recorded today were; 1.53 and 1.21 Mbps. I suppose these low numbers could be the result of traffic on my internet connection, or the AA tester server could be busy. My policy is always to disregard everything and take the maximum as the only value of interest, seeing as undesirable effects can reduce the rate but cannot increase it; the maximum is the true capability of the link, disregarding any other traffic in it, or protocol-related effects such as the performance of TCP - other transport layers might give spurious differing results for the supposed link capacity. I want a measurement of overall bonded link capacity not the performance of some particular arbitrary transport protocol. There’s also the efficiency of the Firebrick router’s load splitting capabilities using each of the links to the optimum according to what it can truly handle.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on June 29, 2020, 05:56:44 PM
Hmm . . . You know your own lines and circuits better than any of us.

I performed a search based on the string "Andrews and Arnold LibreSpeed speed tester" and this was result (http://speedtest2.aa.net.uk/). For fun, I decided to try it for myself . . .

It reports: "Ping 54.0 ms, Jitter 25.6 ms, Download 5.21 Mbps and Upload 1.28 Mbps".

My ZyXEL VMG1312-B10A reports: "5724 kbps and 1011 kbps", downstream & upstream, respectively. (Synchronisation speeds.)

I know which one I will trust.  ;)
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: j0hn on June 29, 2020, 06:30:03 PM
Ignore their test, it provides impossible results.

I ran the test 3 times and twice it gave an impossibly high upstream.

The last test gave a result above what both upstream and downstream is capable of.

I'm on M100 from Virgin (which gives 110/10).
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on June 29, 2020, 07:13:46 PM
Ignore their test, it provides impossible results.

We are in agreement. 
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: mofa2020 on June 29, 2020, 08:04:26 PM
Though I am from another country but when doing tests (SPEEDTESTdotNET) on my ISP servers I get about the same sync. speed "81mb download / 10mb upload" but my actual package is 70/7 mbps and the actual "usable" speed would be about 68.7/6.7 for all internet usages, for this part I consider testing on ISPs own servers may be like LAN file transfer even when my monthly quota is over and speed dropped more over speed test website takes more time to load "due to slower internet speed" yet when testing on my ISP server the speed is almost as my sync. speed as I said above.

This is my speed test on AA and the result is little bit more than my usable speed specially on the download side 5mbps more, guess testing on ISP's own server will give more speed in general "at least from my experience" apart from AA's test is not accurate enough when tested from other UK ISPs internet or outside UK.

Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 30, 2020, 01:42:42 AM
Thank you for letting me know folks. I have assumed that AA has changed the software this year (or last) as the UI is different and the brand name LibreSpeed wasn’t shown before, but it may be that that was just a new release of the same project as I seem to remember seeing a reference to a name change in github’s files and it may be that the name changed from "" (null string) to something.

The AA result is literally impossible by about 5% from calculations based on the sync rates plus assumptions of ideal, zero-loss bonding (unlikely!) and equally unlikely perfect TCP efficiency and based on the byte expansion bloat due to ATM with long max length IP PDUs of 1500 bytes total (ie IP packet including headers is an IP PDU; a PDU includes headers, an SDU is more like payload approximately - and an SDU is the packet or string of bytes given or passed down to the service in question or delivered/passed up by the service in question, sincere apologies if you know these terms already). If the AA test is showing the figure for TCP payload - the alternative of showing PDU bitrate - ie including headers - which is more accurate and meaningful and also higher still - then this means that the exaggeration in the test is even worse than 5%.

The following table shows my calculated factors for converting TCP payload rates into TCP PDU rates and vice versa and converting TCP payload rates into sync rates and vice versa, assuming 0.884434 protocol efficiency for ADSL2 with ATM which is the right protocol efficiency for my lines. If you are running VDSL2/FTTC that includes PTM not ATM is the sync rate conversion figures do not apply to you as the protocol efficiency value required will be a lot higher ie better, nearer to 100% - about 96.69% for most people is the approx number given by (iirc) Kitz but I forget - some FTTC users have a rather lower protocol efficiency according to Kitz, something around 91% which is due to the presence of G.INP set to both on and high. If you are using FTTC then you would have to use one of the latter numbers for protocol efficiency and that number would replace each use of 0.884434 for ADSL+ATM.

TCP efficiency
TCP TS+IPv6 efficiency   95.200000%of IP PDU (ie (TCP payload ÷ PDU) fraction)
TCP+IPv6 efficiency96.000000%of IP PDU
TCP TS+IPv4 efficiency  96.533333%of IP PDU
TCP+IPv4 efficiency97.333333%of IP PDU
TCP TS+IPv6 rate 84.198113%of sync rate (ie (TCP payload ÷ sync rate) fraction
TCP+IPv6 rate 84.905660%of sync rate
TCP TS+IPv4 rate85.377358%of sync rate
TCP+IPv4 rate86.084906%of sync rate

However since the numbers reported by the tester are usually far lower, disappointingly so, it could be that that was either some one-off bug or very weird one-off case multiplied by a permanent exaggeration error, although that seems unlikely given that the usual common numbers coming out of the test are so low.

I use the AA test given the fact that it’s topologically very close. I wasn’t aware that it is open to non-customers although I’m not surprised, but I would have thought that for non-customers it might under-read because of the longer route between ISPs.

Anyway, I should perhaps tell AA, because it is indeed literally impossible based on the sync rates and the known ATM byte bloat and TCP and IP worst and best case byte bloat figures above.



I have now sent AA an email referring them to this thread and telling them it’s just an FYI / bug report not an actual support request relating to a ‘problem’ in my case. Given the last four numbers in the above table - the numbers for TCP payload to sync rate conversion - the exaggeration is rather worse than 5%.

I wonder what happens in the case of protocol errors such as packet loss, retx, duplicating packets and out-of-sequence arrival ordering. You would think that all of these cases would make the numbers lower not higher, no? How do you make the numbers higher than normal ? (One obvious answer is that it is from a straightforward bug in the app or in the reports from the o/s.) I wonder if the numbers that the app gets for the amount of data transferred in the test are quoted in the application protocol before the test proper begins or whether the byte count in question is obtained by relying on the rx operating system. I don’t see how that can be in error, apart from a straight app bug. The time value could be very wrong though. Maybe the test run isn’t long enough. But wouldn’t a short test run because of impatient users produce low numbers because TCP slow start would distort the values greatly making the numbers too low?

If the app itself has straight magic tweak numbers to correct for measured wrongness in the app based on experience with true byte counts and true timings, then it could be that these tweaks don’t apply to me because they don’t apply to the four lines bonded case. If a magic number is used to compete date for slow start then maybe I beat their assumed slow start case occasionally somehow because I’m using multiple links bonded together. Doesn’t sound right?

All the code is available on github but I’m too tired to read all of it looking for answers.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 30, 2020, 10:56:25 AM
Btw, the 0.884434 protocol efficiency figure used above is defined as the bitrate increase multiplication factor that is due to byte stream bloat expansion only, and which is therefore ≤ 1.0 so in fact a decrease) caused by STM SSL5 and other protocols in eg the ADSL2 + ATM protocol stack. This number in this particular case comes from the 48 / (48+5) ATM expansion figure assuming a total of 32 bytes of DSL stack headers including the ATM AAL5 CPCS trailer and everything in protocol layers above, up to but not including IP headers. That number is the conversion factor for sync rates to IP PDU bitrates.

If you are using FTTC/VDSL2 then your protocol efficiency value will be different, higher ie better, as mentioned earlier.

In practice, it is found by experience that a more realistic lower tx rate is in fact often the maximum practical rate that can be used. Here I have defined the modem loading factor (MLF) as that multiplier that causes such a chosen further rate reduction below the reduction caused by the presence of byte bloat, but disregarding eg TCP and IP headers and eg TCP inefficiency.

I was using an MLF of 96% which is obtained from testing and a lower value is found to have a benefit in reducing latency presumably by reducing input queueing time into egress subsystems. Higher MLFs can mean that buffers become overfilled. According to simulation calculations, a lower MLF can mean that sometimes an egress input queue (tx queue) has zero bytes in it, so no queuing delay at all present in a certain fraction of the time. Higher MLFs can mean a significant probability that there will be some n bytes of other packets waiting in an egress queue ahead of the latest PDU being enqueued all of which represents additional delay / RTT / latency. That is part of the reason why the particular MLF value of 96% was in use here, it is a ‘to taste’ compromise in keeping latency reasonable whilst still having really good efficiency near to the maximum that can practically be achieved. Analysis shows that my 96% has a reasonably good chance of keeping the queues non-empty, so no ‘bubbles in the pipe’ with no wasted times without any data in flight in the link, yet the queue length while being usually non-zero is still small so that delay is minimal. This is the motivation for the 96% value of MLF; experiment shows that further increases do not improve effective tx rates. Indeed the reverse may be true.

Often however a conversion factor for sync rate to TCP payload bitrate is required, obtaining this however requires knowing the details of the TCP headers, whether it is IPv4 or IPv6 plus any IP extension headers and also the presumed length of packets being used, which are perhaps constrained by the MTU/MRU. The range of additional rate multiplication factors to allow for TCP with either IPv4 or IPv6 headers and either with or without TCP timestamps is approxinately IP PDU rate * 1.0277 to 1.05.

I didn’t note whether or not IPv6 or IPv4 was being used for the speed tester session. It varies according to the whims of the Safari iOS web browser I’m using with its presumed happy eyeballs race algorithm. I could have working it out by looking at current sessions tables shown in the router UI. Doesn’t tell me whether or not iOS and the other end are using TCP timestamps though.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 30, 2020, 12:07:16 PM
I’m wondering if this exceptionally high value, which is a one-off effectively, is caused by a one-off bug triggered by particular protocol traffic conditions? Doesn’t sound very likely to me. I ask myself whether the AA speed tester results are consistent with each other, that is, that say 200kbps higher reading represents an actual 200k difference in rate (apart from some possible exaggeration factor >1.0).

So at least I can compare AA results with one another. The other test I use is the Ookla speedtest app, which gives a lot lower numbers on upstream, seems to me to under-read on downstream tests and I’m assuming that it’s more about the performance of say TCP, if that’s what the test is using, than the performance of the link. I keep saying that TCP-based tests have their uses, because they give a worthwhile common situation rate that people can expect, but as they aren’t measuring the capacity of the link, they are inherently misleading and bogus. UDP with gently increasing spaced rates should be used imho, iirc someone in an earlier thread gave me a reference?



AA got back to me regarding my bug report; they are going to be doing a rebuild in the next few days. I don’t know whether or not this is to any recent releases contains relevant updates/bugfixes though.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 30, 2020, 01:27:27 PM
@burakkucat - assuming MTU of 1500 for yourself, all the other relevant numbers will be the same as mine - ADSL2 + ATM. So I make it 894 kbps upstream maximum at 100% MLF for 1011 kbps u/s sync rate; or 858 kbps if it were at 96% like me as a more realistic value than 96% MLF. So Burakkucat is seeing at least 43% u/s rate exaggeration! Much worse than me. Perhaps my TCP test sessions are slower because TCP doesn’t work so well with four bonded lines so reduces the inherent test exaggeration.



Idea (not great) : If the Ookla speed test code is any good then that might provide a worthwhile route to go down; the Ookla Speedtest has various organisations hosting servers, so if AA were to host one then that would give exceptionally relevant results for AA customers. It would be best if they had two instances of the server, one being limited to AA users only and one public, so that the AA users would not be seeing low results from an excessive number of simultaneous users including large numbers of non-AA visitors. I wonder if one can lock the Ookla server code down to default to one server instance, to make it default to the right value in the AA customer user case. I can see it getting messy like this though.



The Ookla test app using the server at Zzoomm in Henley on Thames gives 9.37 / 1.57 Mbps and at Mythic Beasts Ltd’s server 9.20 / 1.53. It would be much better to do multiple tests in the dead of night and take the max of the values.

Basically any number for upstream < 1762640 bps is plausible since that is the predicted max IP PDU rate down-calculated from the sum of the upstream sync rates, as given in the above table, corrected for bloat and reduced to the 96% MLF in use. That is still too high though as no account is taken of TCP and IP headers, if TCP is in use, nor possible TCP time inefficiency or out-of-order delivery related problems.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on June 30, 2020, 05:49:39 PM
@burakkucat - assuming MTU of 1500 for yourself, all the other relevant numbers will be the same as mine - ADSL2 + ATM. So I make it 894 kbps upstream maximum at 100% MLF for 1011 kbps u/s sync rate; or 858 kbps if it were at 96% like me as a more realistic value than 96% MLF. So Burakkucat is seeing at least 43% u/s rate exaggeration! Much worse than me. Perhaps my TCP test sessions are slower because TCP doesn’t work so well with four bonded lines so reduces the inherent test exaggeration.

Thank you for the analysis. And yes, I do have a MTU of 1500.

Somehow I cannot see A&A hosting A.N.Other's speed tester. They, A&A, have a reputation for doing their own thing.  ;)
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on June 30, 2020, 06:05:41 PM
The topic "Speedtest via CLI (https://forum.kitz.co.uk/index.php/topic,24901.0.html)" may be of vague interest . . .
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 30, 2020, 07:39:53 PM
Thank you very much indeed for that link.

The speedtest-cli speed tester gave a result of 9.17 / 1.81 - an almost equally unlikely upstream result; it detected AA as the ISP in use and auto-picked a server geographically close to AA’s head office and data centre.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on June 30, 2020, 10:57:02 PM
The speedtest-cli speed tester gave a result of 9.17 / 1.81 - an almost equally unlikely upstream result; . . .

I have noticed that speedtest-cli --simple will always give me a result that looks sensible when compared with the synchronisation speed reported by the modem but speedtest-cli seems to exaggerate the US speed.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on June 30, 2020, 11:26:56 PM
Aha, thanks, I tried that switch —simple. The upstream result was even more exaggerated for me at 1.92Mbps.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on July 02, 2020, 12:18:52 AM
Here is the spreadsheet used in the earlier calculations. Can be used for eg VDSL2/FTTC too as well as my ADSL2+ATM if you just select the right protocol stack state options in the current internet access type settings column.

I have attached the original Apple Numbers spreadsheet zipped up; then exported to XLSX format; the exported to straight CSV files zipped up



@Burakkucat - Isn’t your downstream sync rate really high just now ? Or am I just forgetting. Looks pretty good anyway.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on July 02, 2020, 12:31:38 AM
@Burakkucat - Isn’t your downstream sync rate really high just now ? Or am I just forgetting. Looks pretty good anyway.

I currently see a synchronisation speed of 5735 kbps DS & 1019 kbps US and a throughput speed of 4.56 Mbps DS & 0.98 Mbps US.
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on July 02, 2020, 01:01:12 AM
A long time question : since line #3 upstream is so low compared to the others, will TCP get confused by the varying delay times of packets depending on whether they went up through the fast line #1 or the slow line #3 ?
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on July 02, 2020, 01:23:25 AM
That is a query for someone who understands the mechanism of line bonding. It's something I can not answer and I would not like to guess.

Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on July 02, 2020, 03:26:30 AM
I see, a clever transmitter might tweak the times of transmission of the packets in order to be kind to TCP ? Or not. I’m assuming that in the latter case the values of the difference (tcp_tx time - arrival_time) might go up and down as the different lines are used and a TCP implementation observing transit durations in order to control TCP’s behaviour might get confused because one moment it’s seeing a 300kbps link and the next a 500kbps link. And that could muck up TCP’s delay/RTT-related algorithms, Does that seem a reasonable guess?
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: burakkucat on July 02, 2020, 04:21:09 PM
I'm struggling to formulate a picture (either murky or clear) . . .
Title: Re: All-time highest upstream measured speed of 1.86 Mbps TCP throughput: AA tester
Post by: Weaver on July 02, 2020, 04:35:17 PM
Me too. Perhaps someone who is good at deciphering detailed TCP behaviour from a pcap snapshot of traffic taken during an upload could comment? I’ve forgotten - I used to have a good technique for starting a long upload, tried this which got appallingly low results: https://testmy.net - Be aware, must select a UK server as it defaults to US-based ones, stupidly.