Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

Value your criticism.

[Alex Atkin UK]

You said ISPs didn't do this a few days ago on the Zen GEA thread Alex

You misunderstood what I said:

They already do this but without any tags to let them know some are more important than others, so they just drop when the buffer is full.

They throttle the connection, they do not perform QoS, all traffic is treated equally.  They HAVE to throttle the connection, as it would be insane to send more traffic down the backhaul than can ever be delivered due to the link speed at the other end.  You drop the excess traffic as soon as you can.

If you mean the bit before then yes, this topic does have crossover with why we can't prioritise time sensitive traffic without going against net neutrality.

If we followed QoS based on server tagging the problem could be even worse, as a dishonest service could tag all traffic as high priority whereas an honest one not, giving the dishonest provider an unfair priority.  So then it comes back to deep packet inspection by the ISP, with can be abused by the ISP.

Net neutrality is the only thing we can really do to try and avoid this abuse.

@Weaver Its worth noting that plenty of software for more advanced users DOES have speed throttling, things like Torrent and Usenet clients.  Its smartphones and tablets where users are assumed to be not very technical where they do not bother.

For really technical users, you use deep packet inspection in the router, but that sure comes with a financial cost as the CPU overhead for that is huge.

Unfortunately throttling download at the router has limited utility, as you can't control how fast the data is coming into the ISP where it will already be dropped.  Thus why I haven't even bothered to enabled download throttling at all, its upstream throttling that does most of the work as you CAN prioritise what order local packets are sent and drop less important traffic, before it hits the WAN link.

I'm using pfSense with QoS and no buffer bloat at all.  Get an A and A rating on Thinkbroadband for buffer bloat, not so when QoS is turned off.

It should be possible to use QoS on the router to remove buffer bloat, just depends on the routher.

Same here, but I'm only throttling upstream, as downstream only really does anything if you throttle it BEFORE the smaller link, at the ISP.

Faster than 99.7% of tests in Yorkshire and Humber
In top 1% on your ISP using FTTH/FTTP
Quality 0.10 (A) is better than average 0.46 (A) for FTTH/FTTP

[Weaver]

@Alex - XGS_Is_On misunderstood some of my earlier points - bad writing on my part. I meant that speed control should be implemented in the o/s, in higher layers, ie by mucking about with delivery from the app then inside the o/s then at the point where it goes into to eg TCP, QUIC and UDP. I imagine throttling can also be done by tweaking transports’ window sizes dynamically, and delaying outgoing data-bearing PDUs and ACKs very carefully. I wasn’t thinking about a non-protocol-aware single packet throttler using one of the standard algorithms that is employed for all packets, although that might be employed at the top of a USB service.

I was also giving some thought to potential for misuse, as we also have to now that flower-power is 50 years in the past.

I was aware that AA is typical in its throttling; it was just an example, chosen because I know the exact details and can even control the downstream throttling myself. (I can also use the upstream throttling feature in my Firebrick router.)

I would also like to get a remote o/s to report back state information, and get routers at bottlenecks to be able to report bottleneck sizes, both max and instantaneous, by having some new protocol. Hooray, an RFC. Probably an ICMP new PDU type, plus something in PPP. Also really need something in L2 perhaps so that one can talk to devices that only speak ethernet, eg switches / bridges, and where one can’t guarantee that eg IPvx is present. A way of using it would be to talk to the current default gateway and to the L4-path-end remote o/s. The default gateway router can then talk to the router one hop upstream, and that hop is assumed to be the first mile and also the bottleneck. However, one could proceed n hops upstream up the chain, like traceroute, querying each box in turn, searching for the most critical, narrowest bottleneck report. I would love standardisation of a "query-to-ISP" protocol, so one could ask about max, best-case bottleneck widths up+downstream and ask about current bottleneck with current traffic level. AA has something like this, a query-to-ISP API. Can send commands to the ISP too, iirc. It’s used for all sorts of things. There’s also a ‘what’s my current remaining traffic usage left’ thing, concerned with usage quotas. That’s implemented with https and ASCII. Not very machine-friendly as it’s too heavyweight; rather it’s intended for eyeballs. I’d prefer an alternative that doesn't require you to have TCP and http and TLS protocols implemented. Perhaps just UDP, or maybe a bare-TCP alternative. I’m aware of protocols such as LLDP that can query and discover kit on the LAN, and these - I forget - use only L2 iirc and can be used to query switches for example. Someone remind me.

I realise that this is all wildly off topic, and apologise to the OP. This digression should be filed under ‘fantasy o/s and standards design’.

[Chrysalis]

I forgot to add my new shaping for downloads is on outbound pipes to shape the ack's I found it way more responsive and effective than trying to use downstream shapers.  Adding artificial latency directly manipulates the throughput negotiation.  Restricting the size of the ack pipe can be more forceful way of doing it as well.  But once on FTTP all this stuff I will unconfigure.

[Weaver]

Good point, just as I was thinking of doing it.

[Alex Atkin UK]

I forgot to add my new shaping for downloads is on outbound pipes to shape the ack's I found it way more responsive and effective than trying to use downstream shapers.  Adding artificial latency directly manipulates the throughput negotiation.  Restricting the size of the ack pipe can be more forceful way of doing it as well.  But once on FTTP all this stuff I will unconfigure.

Upstream is still useful, it halved my latency under load when uploading.

Like you said, shaping ACKs is really the only control you have over downstream but rather less likely to be necessary on FTTP.

I do remember years back though when downstream definitely worked, when Xbox first streamed their E3 conference over Xbox Live I watched it while running a download and the stream never dropped quality, the download was throttled back.   Its still all rather smoke and mirrors to me, I don't understand HOW it was doing it, just that it worked.   But that was on OpenWRT.

[XGS_Is_On]

@Alex - XGS_Is_On misunderstood some of my earlier points - bad writing on my part. I meant that speed control should be implemented in the o/s, in higher layers, ie by mucking about with delivery from the app then inside the o/s then at the point where it goes into to eg TCP, QUIC and UDP. I imagine throttling can also be done by tweaking transports’ window sizes dynamically, and delaying outgoing data-bearing PDUs and ACKs very carefully. I wasn’t thinking about a non-protocol-aware single packet throttler using one of the standard algorithms that is employed for all packets, although that might be employed at the top of a USB service.

The OS has no way of knowing what else is using the link. A host level shaper has to run insanely conservatively or can't be run at all.

I was aware that AA is typical in its throttling; it was just an example, chosen because I know the exact details and can even control the downstream throttling myself. (I can also use the upstream throttling feature in my Firebrick router.)

A&A are actually a good case study on how making equipment 'smart' also makes it expensive and reduces its throughput dramatically. For the price of an FB2900, no options, it's possible to get a BRAS with as much throughput as an FB9000.

I would also like to get a remote o/s to report back state information, and get routers at bottlenecks to be able to report bottleneck sizes, both max and instantaneous, by having some new protocol. Hooray, an RFC. Probably an ICMP new PDU type, plus something in PPP. Also really need something in L2 perhaps so that one can talk to devices that only speak ethernet, eg switches / bridges, and where one can’t guarantee that eg IPvx is present. A way of using it would be to talk to the current default gateway and to the L4-path-end remote o/s. The default gateway router can then talk to the router one hop upstream, and that hop is assumed to be the first mile and also the bottleneck. However, one could proceed n hops upstream up the chain, like traceroute, querying each box in turn, searching for the most critical, narrowest bottleneck report. I would love standardisation of a "query-to-ISP" protocol, so one could ask about max, best-case bottleneck widths up+downstream and ask about current bottleneck with current traffic level. AA has something like this, a query-to-ISP API. Can send commands to the ISP too, iirc. It’s used for all sorts of things. There’s also a ‘what’s my current remaining traffic usage left’ thing, concerned with usage quotas. That’s implemented with https and ASCII. Not very machine-friendly as it’s too heavyweight; rather it’s intended for eyeballs. I’d prefer an alternative that doesn't require you to have TCP and http and TLS protocols implemented. Perhaps just UDP, or maybe a bare-TCP alternative. I’m aware of protocols such as LLDP that can query and discover kit on the LAN, and these - I forget - use only L2 iirc and can be used to query switches for example. Someone remind me.

I realise that this is all wildly off topic, and apologise to the OP. This digression should be filed under ‘fantasy o/s and standards design’.

This would be an incredibly bad idea. It'll expose a ton of extremely commercially sensitive and security sensitive information that'll give exactly where to attack a network in order to effectively DDoS it alongside allowing the mapping out of ISPs' networks. It'd break in the presence of tunneling as it'd have to look up the relevant tunnel and give that number rather than the link in question, where capacity is being shuffled between tunnels the max and current will change constantly, and it'd be no good in the context of links where bandwidth is reserved, and there will be plenty where commercial grade traffic with a CIR is running over links with residential traffic. Behaviour in the face of ECMP and asymmetry could be interesting, too.

It'd also need to run insanely frequently to be anything like real-time - the shapers on the kit I use have granularity in the single-digits microseconds range. Moving ~15 gigabits a second requires upwards of 1.3 million packets a second.

However it could be done every few seconds right now through the operator exposing SNMP and allowing everyone to query it.

On downstream shaping the best way to do it is not to do it - tunnel everything to a host on the Internet with a much bigger connection than yours and have it shape upstream: its upstream is your downstream.

[Weaver]

Understood completely about the security implications. This is the start of a musing, not the end of one. Your security points have to be taken seriously and worked through properly. I’m assuming that one would only be allowed to find out information that pertains to one’s own current link to the ISP. Non-customers and the wider Internet wouldn’t have access to these services, and one user wouldn’t be able to query info relating to another session.

I understand your point about Firebrick economics; Burakkucat has made vaguely similar observations. Firebrick Ltd also doesn’t have economy of scale from huge volume. I spent many years writing low-level fast code, so I don’t know how much, if at all, the code in the various time-critical hotspots and paths in the Firebricks could be improved by a specialist or even just a better, more aggressive compiler.

For clarification: Firebricks have zero QoS - that’s one criticism I have of the Firebrick. They do have a heuristic though, that small packets queue-jump; they’re treated as high-priority. So that caters for VoIP, and hopefully some ACKs too.

[Chrysalis]

Upstream is still useful, it halved my latency under load when uploading.

Like you said, shaping ACKs is really the only control you have over downstream but rather less likely to be necessary on FTTP.

I do remember years back though when downstream definitely worked, when Xbox first streamed their E3 conference over Xbox Live I watched it while running a download and the stream never dropped quality, the download was throttled back.   Its still all rather smoke and mirrors to me, I don't understand HOW it was doing it, just that it worked.   But that was on OpenWRT.

Yep I will keep upstream shaping in place.

[XGS_Is_On]

They do have a heuristic though, that small packets queue-jump; they’re treated as high-priority. So that caters for VoIP, and hopefully some ACKs too.

Weighted Fair Queuing. 

[Chrysalis]

Sadly WFQ can kernel panic pfsense and opnsense, there is an old bug posted on FreeBSD that never had any work done on it.

XGS I did try using a private VPN I maintain and shaping the outbound traffic as you said once, and it did work really well, so I can confirm it does work wonders for my use case.

[Weaver]

XGS, from what I’ve read, AA’s routers’ code’s algorithm isn’t anywhere near as sophisticated as WFQ. I believe there’s just the one length test and that’s it. (If I were writing such a very simple thing, I would just maintain two queues and always pull packets from the high priority queue first if there are any packets in it.) I may be insulting Firebrick, but that’s the impression I have repeatedly got from the AA website and RevK’s various posts.

[XGS_Is_On]

I understand your point about Firebrick economics; Burakkucat has made vaguely similar observations. Firebrick Ltd also doesn’t have economy of scale from huge volume. I spent many years writing low-level fast code, so I don’t know how much, if at all, the code in the various time-critical hotspots and paths in the Firebricks could be improved by a specialist or even just a better, more aggressive compiler.

It was more that when you add a ton of functionality to equipment it gets progressive more expensive per Mbps of throughput. I'm sure the Firebrick code is very good, AK wouldn't allow it to go any other way, but it does make for a very expensive product if not using enough of the functionality.

Same story with something adding the level of complexity you discuss. This level of intelligence belongs on the edge where the bandwidths are much lower, core and backbone kit should be pretty simple and have a minimal feature set to preserve raw forwarding capacity. Make the core too clever you add bottlenecks and, per A&A, need racks of kit to handle the same capacity as a single chassis.

Congestion avoidance is actually already a thing on the edge though - have a review of software defined networking. Clever stuff but runs on top of simple, extremely fat networks as a premium service. Sites can infer effective bandwidth between them based on link performance and dynamically adjust shaping.

Does also mean it can only be done between devices you control, and accepts that Internet access outside of your network remains an unknown. The telemetry overhead is tens of kilobits per second per station as it cares about performance per overlay tunnel not per flow, however the processing requirements mean the performance of routing and switching ASICs is out of reach. Per-flow is left to individual stations to manage their queues, the capacity of each queue using the telemetry for link bandwidth.

Even software routers performance drops like a brick as you increase CPU load - see Mikrotik numbers with and without rules.

A feature for the classes not the masses 

[dee.jay]

Side question, does QoS actually work on a router going out to the public internet?

I was always under the impression that QoS is only really useful in situations where you have complete end-to-end control of every single device in a traffic flow, as that implies you are able to set the requisite marking of the packets, and thus get all devices to respect that marking.

As soon the public internet is involved, there's no chance that'll happen.

[Alex Atkin UK]

Even software routers performance drops like a brick as you increase CPU load - see Mikrotik numbers with and without rules.

A feature for the classes not the masses 

Indeed.  Its telling that high-end consumer routers use hacks to offload even basic NAT functionality, even adding a port forward on those often disabled this and turns a Gigabit capable router into something like a 400Mbit capable router.  Add QoS and you're having a real bad day.