So, had some interesting results in.
@Chrysalis 's results against my servers were not overly disimilar to mine, though in their case the BBR didn't seem to improve matters.
AAISP got a staffer to run tests on their line, and one on FTTP BTW did manage on the two london AWS servers to hit single thread line rates, and the Frankfurt ones were not far off.
So this gave me a thread to pull on as that is a similar connection but better performance.
The testing I had been doing previous was predominantly on the router, logged in via SSH. (Unifi Dream Machine SE), where AAISP's server would be line rate, but reduced to the AWS servers, improved with BBR.
I installed Ubuntu on the gigabit laptop I have here so I could use that as a PPPoE client.
From the Ubuntu laptop, routing via the Dream Machine as a DHCP client, even the single thread to the AAISP speedtest server was down to about 600Mbps (though oddly, the single thread test to AWS london non-BBR was notably faster than this). Wired clients however can achieve line rate though in multithreaded tests.
So at this point it looks like the setup is on the edge once the overhead of routing through the Dream machine happens.
Running PPPoE on the laptop, my results are much more like from their staffer, the two London tests and even the Frankfurt BBR being very close to line rate. The only odd anomaly is that it takes a while for the AA speedtest to "ramp up" to line rate often (maybe 10-15s)
I also used the same laptop and routed via the AAISP Technicolor router and it's similar to PPPoE on the laptop, though seems more prone to having the connection startup ramp effect.
I think what this says to me at the moment is that for the gigabit single threads to make it all the way through from client to server everything has to be just right, and the Dream Machine doesn't really seem to be there, particularly when shoveling data between interfaces. It can do the gigabit speed across multiple connections, but a single connection seems to have some kind of variability preventing achieving full rates. I'm not sure what that ramping behaviour is, but as I was just running desktop Ubuntu, with GUI etc, I wouldn't really like to bet what is going on there. It was seen on both the PPPoE on the laptop and the Technicolor, so perhaps it is a function of the laptop / ethernet card, but not when running the iperf3 tests on the Dream Machine router itself (though I did see it with a Hetzner file download). And I have seen downloading big files from Hetzner to an AWS ramp like that over a large number of seconds, so it's not unique to this network or equipment. It was interesting that testing from my laptop via the Dream Machine router was a bit faster to my AWS London cubic iperf3 server than AAISPs own server, I don't know if that points to any possible tweaks to that server.
I think what I'm going to do is try and get another router setup and retest with that, it will be a while before that happens.
Been very impressed with AAISPs response to this, they've been very helpful.
