Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

I'm thinking about the question - which is probably one for AA, my ISP - of how protocols such as TCP are likely to get on (depending on the sophistication, version and configuration of TCP too) in situations where four or more lines are IP-bonded together. Iirc AA said something about TCP performance with more than four lines. I'm reasonably certain some AA customers have more than three lines.

I'm assuming that modern well-featured and well-configured TCP implementations will fare rather better in a many-bonded-lines setup.

* As older operating systems get phased out, so nasty versions of TCP will get less frequent surely. Does anyone know anything about this?

* What's the likely distribution of TCPs out there with important modern features (SACK, window scaling, modern algorithms, ECN, timestamps)? And other good behaviour like adequate-sized (or right-sized?) buffers, window sizes? On typical servers for example?

* Is there any online tool available for testing servers’ TCP feature set ?

(Windows or Linux-based  tools are out at the moment, unless I manage to get my Raspberry Pi going - been too ill recently, and at the moment I'm confined to bed, iPad-only.)

* iOS: I use a lot of iPads, and I'm hoping these have a very modern intelligent implementation of IP and TCP?

Recap: By IP-bonded, I mean that even a single TCP connection (for example) will be load-split across all the lines, weighted in the right ratio according to each line’s particular speed. This applies in both directions: it is done by the ISP for downstream traffic, and by my router for upstream. So a single TCP connection truly goes at triple speed (measured). In the downstream direction, there seems to be no performance loss, comparing measurement with the expected total. In the upstream direction it’s less efficient I think, for some reason, but it's more than double speed upstream anyway, and everything is a bonus anyway.

(I know of a user who years ago had six (possibly bonded, but how) DSL lines with another ISP, but this could have been a setup where a TCP session was restricted to only use one line at a time. Don't know enough for that case to be helpful.)

[Chrysalis]

There is only one version of tcp.  But over time different developments have been made to enhance it, such as different congestion control providers and automatic buffer tuning.

I have never bonded a connection over multiple ip's (as some people do with xDSL) and I am surprised that even works as then a download session is split over two end points.  Does AAISP bonding do so all on one session with one ip address shared across all modems?

The one time I dabbled with multi homed internet (when I had both VM cable and ADSL connected to my PC), I didnt bond it, I just used it for redundancy and selective routing, so e.g. routing youtube over the ADSL and say bulk downloading over VM.

In terms of TCP implementation out there on the internet there is many things factoring what technologies get used.

e.g. a popular web server software in apache can prevent large TCP windows so windows capped at 65k which in turn limits download speeds per thread. 
In addition server administrators have to consider load on the server, resilience to DDOS and other things and this can lead to forced small TCP window sizes (which manages kernel resource usage), disabling of SACK greatly helps against certain types of DDOS attacks.
Timestamps are known to have small risks associated with them so some admin's will also disable timestamps.
Some isp's abuse ECN to throttle speed's and this has resulted in blog's advising people to always disable ECN. Although in my experience ECN seems to have no noticeable benefit anyway.

What is enabled/disable is down to if the system has been tuned and if it has, what it has been tuned for, e.g. if I was setting up a web server which had lots of hits, and most of data it serves is in small packets I would probably disable SACK, set small TCP buffer sizes and disable timestamps.

However if it was hosting large files for download and I wanted the end users to get fast as possible speeds then I would enable large TCP buffers, and enable SACK. 

The second situation however by the larger companies is now dealt with by using CDN's to have end users downloading from a location close to them meaning large TCP windows are not so important.

[ejs]

On slower connections, wouldn't TCP timestamps be not worth the extra per-packet overhead?

[Chrysalis]

Timestamps I feel are sort of like ECN, an idea in the real world that doesnt work.  But yes thats a second reason to disable them due to the extra overhead. On FreeBSD they cannot be disabled tho without disabling large windows, as one setting toggles both.

[Weaver]

AA's bonding is done transparently. A host has one public IPv4 and one public IPv6 address. Routers at AA load balance downstream traffic over the multiple pipes. My router has a public IPv4 for each WAN interface but these addresses are never seen by anyone unless (apart from in a trace route). And yes a single TCP session is shared across the three pipes so triple speed in both directions.

[Weaver]

If played around with timestamps in recent versions of Windows and found that things can go either way, they are extra overhead, of course, but sometimes they give an improvement depending on the type of network, on balance not such an important idea.

By versions, I meant e.g. Reno, New Reno, Tahoe and Microsoft’s optional ‘Compound TCP’ which I like a lot.

I have tested ECN in windows but not managed to make any conclusions I don't have enough experience with it.

[Weaver]

I've just been reading RFC 3481, which I though was very sensible and educational. That's what set off thinking about this.

[Weaver]

I need support for higher latency, as I have high-ish latency anyway due to too much buffering in the network plus generous quantities of interleave (maxed out by me for reliability) and the fact that possible packet recording in bonding situations effectively increases latency variation (like jitter) if my memory of reading of RFC3841 is right. So a decent window size, with window scaling if necessary is good. And SACK has to be a good thing.

I wish I had proper QoS and wish there was better queue management. Excessive queuing doesn't really help.

[ejs]

Last I saw Weaver, you had a fairly ordinary level of interleaving.

[Weaver]

@ejs - I have interleave set to "on" instead of "auto", and on the 21CN ADSL2+ profile higher setting (as opposed to "medium") at the ISP. It used to be set to DLM "auto". I don't know what the modems’ current actual interleave depth is unfortunately.

I see a minimum 40 ms latency reported on the AA CQM line monitoring graphs. (That's a round trip time, of 2 * 20 ms or one-way?) I can do a screenshot. Does anyone know what what the typical timings are vs interleave depth e.g. for depth=64 in ADSL2?

The latency peaks go up to the several 100 ms level when under load. This is all about queuing presumably?

[ejs]

Profile Info   WBC 160K - 24M Medium delay (INP 1) 3dB Downstream, UC Medium delay (INP 2) 6dB Upstream (ADSL2+)

That profile info looks like the usual values for interleaving on.

[Weaver]

Talking about TCP versions, I notice some relevant stuff in this article
    https://en.m.wikipedia.org/wiki/TCP_congestion_control
concerning packet reordering, which is relevant to me.

[Weaver]

I wonder if router load-balancing algorithms could be tuned to try and avoid packet reordering seen by a receiver within one TCP session (also SCTP etc and other protocols’ sessions, not just those obviously connection-oriented ones, at least) seen at the receiver without any risk of sacrificing throughput performance or fairness? AA say that (regardless of load-balancing over multiple lines) they send the smallest packet first, so presumably that's true in the general bonding case too AA. I'd need to think about what that does to packet recording in various scenarios. It accidentally prioritises VoIP, which is I think the reason for it, but TCP ACKs will get prioritised too which is a very good thing, surely. (Although they might or might not compete with VoIP over IPvX, don't know which wins out by being the shortest.)

I suppose a lot depends on how many packets in total a scheduler has to chose between in all of its ingress queues taken together. If you don't have enough stuff arriving soon enough and there's a bottleneck further downstream, then you've lost control. This is one reason why it's good for AA to run very slightly below the line rates reported by BTW.

I should probably do the same thing for upstream myself in my own router, yet that is hard because I can't get dynamic reporting of upstream line rates and there's no protocol for interrogating relevant nodes in the path to get info like this. We need protocols for routers and hosts to be able to talk to modems, to DSLAMs, BRAS servers and ISPs to query these nodes for all kinds of info. Some brave soul needs to write an RFC or two.