Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

When a flat-out download is in progress to my iPad, I can see, courtesy of Andrews and Arnold’s CQM graphs, that the link is being maxed-out. I’m wondering why the latency is so high during a download as measured by AA’s PPP LCP ‘pings’ (PPP LCP echo requests or whatever they’re called) is so high - eg 460 ms max even.

Is this due to bad queue management in my iPad (which is doing the downloading)? Does iPadOS need a better algorithm for managing this, or am I misunderstanding?

[burakkucat]

Is it a case that as the total bandwidth is being consumed the return ACKs are being delayed (or dropped)?

[Weaver]

I wouldn’t think so, as the graphs show that the uplink is not maxed out, so the TCP returned ACKs can get back up safely. I think the delay is all in the downstream with the PPP LCP pings being stuck behind a queue of stuff heading down, but I can’t be certain about this at all.

Actually, that would mean I was talking nonsense before and the queue management would need to be in the server’s OS, no?

Now I think about it again, I’m tentatively assuming that there’s no upstream ingress queue or a minimal length one to be managed by [what?] upstream-bound. Not thinking straight, tired and full of drugs. The upstream ingress queue is in the modem, yes? But I presume it could effectively be managed by an OS further back (ie nearer to the source)?

[Reformed]

Delay is downstream and due to buffer bloat somewhere on the downstream path. Pretty common.

Can't really manage upstream queues on an end device as they only have visibility of what they're sending. Needs to be where the bottleneck is as that device may see all network traffic before it hits the bottleneck and may queue.

[Chrysalis]

Modern internet is aggressive for high download speeds, to max out people's pipes who have multiple 100s of mbits available for downloading.

All of the following contribute to download buffer bloat.

Local CDN hosted content (low latency so easier to over saturate buffer).
Multi threading, very common on consoles, steam, uplay etc.
Modern congestion algorithm's, which again are optimised to fill fat pipes. (sender controlled so not a lot you can do other than to cap RWIN, or aggressively drop packets).

Google's BBR algorithm is new and actually better than most other's at maintaining performance whilst downloading, it's able to slow it self down without as much over buffer as other algorithm's, as always though it will likely take years for this to get adopted around the net.

Some info here regarding BBR and congestion algorithm's. https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/

[Weaver]

Thank you so much Chrys - that’s a superb article, something I hadn’t heard of. I’ve noted Microsoft’s compound TCP and CUBIC, New Reno and so forth, but BBR is new to me. It’s over to the o/s developers now, which means we/I will have to wait at their pleasure. Introduction into Linux distros would create a large implementation boost among servers and that would help us even if our own o/ses - Windows, OSX, iOS, Android etc - have not yet adopted BBR provided it’s effective when only the sending end implements it [?]

[burakkucat]

Some info here regarding BBR and congestion algorithm's. https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/

An excellent article for its its review of the current algorithms.   

[Weaver]

BBR looks really well thought-out; a massive improvement over its predecessors because the design goals are sane, not just all about maxed performance and congestion. Reno’s sawtooth looks like madness when viewed from today’s perspective now. I see all kinds of stupid behaviour like that in speed tests - which imho shouldn’t be using TCP anyway as then they’re testing the TCP implementation, not the link, although using TCP might be a useful secondary option to give additional info.

[Chrysalis]

Thank you so much Chrys - that’s a superb article, something I hadn’t heard of. I’ve noted Microsoft’s compound TCP and CUBIC, New Reno and so forth, but BBR is new to me. It’s over to the o/s developers now, which means we/I will have to wait at their pleasure. Introduction into Linux distros would create a large implementation boost among servers and that would help us even if our own o/ses - Windows, OSX, iOS, Android etc - have not yet adopted BBR provided it’s effective when only the sending end implements it [?]

Its already in new linux kernels and I use it on all my linux based servers.  I believe its in FreeBSD 13 or coming to 14.

Windows probably a long way off if ever.

However it isnt the default in linux, and probably wont be in FreeBSD either, so down to server admin's enabling it.

Even just downloading from an ftp, the difference is noticeable, on bbr, it doesnt flatline max my connection, but instead is just below it, its kind of like a built in QoS.

[Weaver]

I somehow think Microsoft will be compelled to adopt it even if only as an option. It needs to be the default in o/ses though.

[burakkucat]

Looking at the available configuration options, for TCP congestion control in the latest linux-5.15-rc6 sources, I have --

[config-5.15-rc6]$ grep 'CONFIG_TCP_CONG_' *
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_NV=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_TCP_CONG_DCTCP=m
CONFIG_TCP_CONG_CDG=m
CONFIG_TCP_CONG_BBR=m
[config-5.15-rc6]$

Right at the bottom we see the "newest" addition, BBR.

Every option is built as a module and its up to the userland process(es) to configure one of them, as appropriate.

[Edit to correct a nonsense in the last sentence.]

[Chrysalis]

The command to switch should be as simple as

'sysctl net.ipv4.tcp_congestion_control=bbr'
also recommended 'sysctl net.core.default_qdisc=fq' and 'net.ipv4.tcp_notsent_lowat=16384'

To keep them on reboot.

'echo >> net.ipv4.tcp_congestion_control=bbr /etc/sysctl.conf'
'echo >> net.core.default_qdisc=fq /etc/sysctl.conf'
'echo >> net.ipv4.tcp_notsent_lowat=16384 /etc/sysctl.conf'

[Alex Atkin UK]

I've not seen

Code: [Select]

net.ipv4.tcp_notsent_lowat=16384 mentioned before.  Its certainly a curious one given the default on Fedora is

Code: [Select]

net.ipv4.tcp_notsent_lowat = 4294967295 so just slightly different.

I see it seems to be related to making http/2 work best https://blog.cloudflare.com/http-2-prioritization-with-nginx/ and it specifically mentions servers rather than clients.  So not sure its necessary on the client side, though it does seem like it would reduce bufferbloat.

[Weaver]

@chrys and likewise ditto for IPv6 too in Linux ?

[Alex Atkin UK]

@chrys and likewise ditto for IPv6 too in Linux ?

Apparently net.ipv4.tcp_congestion_control=bbr applies to IPv6 too.

Probably worth reading https://www.bufferbloat.net/projects/codel/wiki/ specifically:
For servers with tcp-heavy workloads, particularly at 10GigE speeds, for queue management, we recomend sch_fq instead of fq_codel as of linux 3.12.
net.core.default_qdisc = fq_codel - best general purpose qdisc
net.core.default_qdisc = fq - for fat servers, fq_codel for routers.