Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

Since I added a fourth modem to my Firebrick, I have found that the result given by speedtester s, various, seems very dependent on the egress rates set in the ppp elements for each link.

If I just reduce the rate by even as little as 0.5% then the measured combined upstream (TCP payload presumably) gets quite a bit better. This seems slightly odd.

Clearly pushing it too hard means packet loss due to buffer overflows in the input side of one or more of the modems. I am surprised at just how sensitive it is though.

Any comments, analysis, advice?

I wishe I understood exactly what the Firebrick's algorithm was for traffic splitting. It presumably allocates the correct fraction of the traffic to each link. I wish it scheduled each packet to go out when the modem is free, minus a little amount to make sure that the modem's input buffer does not run completely dry. The thing could also arrange send times to completely avoid packet reordering at the receiving end too. It could also assign flows to individual modems as long as maximum throughout is achieved, so that if there are multiple flows, then it could make the best choices whilst still keeping the link maxed out.

If there is just one flow then it is not so easy, obviously. If you have only one flow and you are supposed to send a long packet and then several short packets, then you could arrange it so that they all arrive at approximately the same time but with slight extra delays so the reception order has tiny gaps in it so the 'later' ones are indeed seen 'after' the supposedly earlier ones. That would not be optimal speed, as you could push two short packets down link 2 while the long packet is still going down link 1. But it would be pretty good, and might be a preferable option: 'no receiver out-of-sequence', but it would be receiver dependent ideally so an utter pain to specify.

[johnson]

I dont really have anything of use to add so please excuse my post, but have just spent a little while reading about channel bonding and found it interesting.

First page I ended up on was here, with a chap that has rolled his own bonding without help of an ISP by using VPN tunnels to a VM he rents in a datacenter, sounds complicated but pretty interesting, from examination of AAISPs wiki they seem to offer hosting to facilitate this type of link aggregation, but I guess the overheads would be far greater than your current solution and I have no idea if the upstream load balancing would be any better.

I then read a little about MLPPP (multi link ppp), the RFC is from the late 90s (not always a bad thing) but it seems its use didnt get much further than ISDN lines. Did find an interesting project here but it appears to have been abandoned some 8 years ago. Again, does it load balance any better?

So if I understand correctly AAISPs clever routers do some magic to determine which link to send you data based on constant monitoring of latency on each line, but egress distribution is down to the firebrick on your end. I have little doubt this close to state of the art, do AAISP themselves not perform the downstream task using firebrick type equipment? I would hope its more sensible than just round robin with proportions determined by stated link capacity. You have to tell the firebrick each lines current syncs though right?

It would be fun to try out a linux based router to see if some magic combination of pings, packet monitoring and telnet/http/etc interrogation of the zyxels could achieve a better distribution of egress packets, but probably requires a wizard.

One last thought about something you mentioned in your forth line thread, if automating the process of querying the modems and pushing a config to the firebrick remains difficult with ipad tools, a handy dandy raspberry pi could probably simplify the task. Could sit there on all the time consuming a few watts, polling each modem and updating the firebrick as needed.

Anyway, sorry for the ramble! 

[Weaver]

Indeed, a perfect task for a raspberry pi. I would have to work out how to interrogate the modems over telnet using C (or rather my new love affair, D ). The pain though would be accessing the Firebrick using http to upload config file changes. There is a convenient HTTP API for doing this, but I would have to work out how to speak HTTP from C. I have never done any such stuff, done far lower level work or stuff involving novel protocols.

To answer your question about AA, in the past they always used Firebricks, but the big expensive ones, 6000 series, to handle this kind of downstream traffic splitting. They get data rate info on a live dynamic feed from BT and TalkTalk, event driven I would expect. Their clueless server puts out an event and a record goes into the history log when the downstream rate changes. The upstream rate is also mentioned in the record that accompanies the rate change event, as it give IP rates for both up and downstream, and these are the real relevant numbers, having presumably subtracted the overhead from all the headers, trailer and ATM cells’ bloat so that no black magic guesswork about current protocols and parameter settings needs to be guessed at. AA’s what I call fudge factor - the IP effective data rate divided by the sync rate - is very close to what I get by careful calculation; theirs is very slightly more conservative, perhaps because they assume MTU 1492 whereas I use the real packet size in this case 1500 from an MTU/MRU of 1500.

Now many of their links are 10Gbps so it may be that they cannot use Firebricks any more and I know that they do use Cisco in some situations. Perhaps they are working on a new 10G Firebrick. The problem is I suspect that these devices will need to push more processing into hardware and that could make the whole design process a nightmare for smaller outfits unless the sub components become available as commodities. I saw that Facebook is giving away the design for a very fast switch - BackPack - so there is hope for small outfits. I don’t know anything about that device but AA need very very fancy routers or even firewall routers.

One thing I do wish that AA did: offer firewalling at the ISP end, remote controlled. I thought I saw that Plusnet offered this kind of service once. I think AA will do that kind of thing for you if you have a problem by setting up a Firebrick, and you can pay a fortune in colo charges plus data for a hosted Firebrick anyway and do it yourself.

[Weaver]

Aa told me that multilink PPP was no good because it assumed that the links were all of the same speed. I don’t know whether or not that is correct, as I have not looked into it. But in complete ignorance, doing it at a lower level seems a very good idea because all problems can be avoided. You could split L3 PDUs up as needed and then get the parts all to land there at the far end at the same time by apportioning the fractional lengths correctly. You could also apply L3 and even L4 header compression in PPP to make it more than 100% efficient and wipe out the slight overhead of the extra split fragment headers. if you are in control of both ends you can build up and send a dictionary so that flows can be identified just by an index and the two ends can keep all the per-flow info to expand that received flow index number into the full thing that it stands for.

[johnson]

Indeed, a perfect task for a raspberry pi. I would have to work out how to interrogate the modems over telnet using C (or rather my new love affair, D ). The pain though would be accessing the Firebrick using http to upload config file changes. There is a convenient HTTP API for doing this, but I would have to work out how to speak HTTP from C. I have never done any such stuff, done far lower level work or stuff involving novel protocols.

The beauty of a linux box is there are any number of ways of doing that. Telnet from C is pretty straightforward, uploading to a http api you could probably just use curl, no idea about dlang but I'm sure there are libraries that are just an apt-get/dnf/pacman away, endless possibilities. Also the first time you write a little python glue to get something done I guarantee you'l be hooked.

Did you say you had a pi sitting around unloved? If you need an sdcard with latest raspbian and sensible defaults I'd be happy to oblige.

Ok so AAISP use monster routers with some black box packet distribution algorithm, do we have any idea how the firebrick chooses paths? Sensitivity to set throughput speed feels like its probably a good thing as tuned enough for that to matter, does it test/account for latency at all? Your lines have some interleaving on correct? Are they equal in delay?

Edit: You are going WAY over my head with the MLPPP talk, but the reason I started looking at it was it seemed like the logical place to start with for aggregating these type of links... but I guess you need a supporting ISP. I should go read some more.

[Weaver]

The lines all have max interleave on, all about 35-58ms to first hop, I assume that is round trip time not one way? They are not the same speed upstream at all, line 1 is 515k, line 3 which is having problems is only 318k. The others are around 440k, so a substantial difference with line 1 way outshining the rest, by about 17% or 62% in the case of the sorrowful line 3.

[Ixel]

I've found that it's important to have an egress rate set a bit below the upstream sync rate, it can really change the upstream performance. When I was recently interleaved on one of my lines on the upstream by DLM I had an incorrect egress rate on one line, and it caused a fairly reduced upstream speed until I changed the egress rate. I wish there was a way of adding a script to the FireBrick to automatically fetch the upstream rate from AAISP.

[Weaver]

I calculated the IP PDU rate as accurately as I could giving 88.4433962264151% with total headers including CPCS trailer at 33 bytes and packets assumed to be a full 1500 bytes long because that is my MTU. And then found that 97.5% or 97% of that is the best value; it really needs to be reduced a bit, just as Ixel says