Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Author Topic: Rate limiting - back-propagation  (Read 234 times)

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Rate limiting - back-propagation
« on: April 07, 2017, 11:31:11 PM »

The DSLAM presumably tells the BRAS what real data rate my line can accept. Is this accurate? The BT servers that accept data from my ISP rate-limit downstream data. But BT also in turn tells my ISP what the rate is, and my ISP happens to rate-limit downstream traffic heading for me too. I presume that the ISP does its rate-limiting at the last possible point before the data is handed over to BT. It would be better for the ISP if the rate-limiting were done at the ingress to their network as well, except that there would be a big cost in RAM for all that state information for the policer instances (and a cost in CPU cycles too) and a corresponding failure of scalability. (It presumably would give limited cost savings because senders’ TCP instances, if say TCP is being used, would reduce the data rates naturally anyway but the ISP is not getting charged for its own internal traffic, presumably.)

Anyway. There is a pattern here. If rate-limiter info could be successfully propagated all the way back in a chain to the source, then there would be no packet loss. TCP for example would work extremely efficiently and there would be savings in the internet where servers that send data to clients connected by low speed internet access pipes don't have to send some of the data across the internet for it only to get thrown away at the recipient's ISP (or later). ECN has a certain amount of overlap with this. Is there an opportunity for improving the internet here? With a router-to-router protocol, but presumably perhaps sending hosts should be included as well. Just a new type of ICMP?

There is a security challenge here because abuse allows denial-of-service. This could be fixed by added complications with varying levels of cost.
Logged

Chrysalis

  • Content Team
  • Kitizen
  • *
  • Posts: 4363
Re: Rate limiting - back-propagation
« Reply #1 on: April 07, 2017, 11:39:53 PM »

What you say already happens.

TCP already self regulates, people like myself have questioned why BT are doing what they do as the only real benefit seems to be for situations where data been pushed down a line might be more than its requesting, and the only scenario I can think of for that is a DDOS attack.

TCP actually starts of at a very slow rate and just sends a couple of packets, which e.g. would be mss x two.  So someone on a 1500 mtu might only initially receive data at just under 3kB/sec.  It then grows and keeps growing until signs of congestion appear at which point it will stop growing or slow down.  There has been work done over the years to actually speed the process up as it was a system designed when dialup was the norm, hence the very slow rates it starts at.  e.g. we have google's proposal which starts with 10 packets instead of 2, and we also have various different congestion providers which behave differently in terms of how quick the window grows and how it reacts if congestion is detected.

The ramp up process is usually fast enough to not notice it if your max speed isnt very high, although I have occasionally come across servers which have a really slow ramp up and tend to drop speeds like a stone when it hits a certian speed, I think those are deliberate throttles tho.
Logged
Sky Fiber Pro - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab

WWWombat

  • Kitizen
  • ****
  • Posts: 1375
Re: Rate limiting - back-propagation
« Reply #2 on: April 08, 2017, 01:06:37 AM »

On top of @Chrys' reply...

If rate-limiter info could be successfully propagated all the way back in a chain to the source, then there would be no packet loss.

Packet loss isn't bad!

The problem on the internet is that the thing that is rate-limited isn't always your ADSL connection. It could happen at any point in the path from A-B, any individual sub-link. If rate-limits get propagated back to the source, then it really needs information about every individual sub-link en-route from A to B.

Of course, connections A-C and A-D will exist in parallel, so those would all need tracking in parallel.

Looking at it in reverse, then, "the source" will be sending to hundreds or thousands of connections, and would have to be tracking limits for dozens of sub-links on each.

All of these intermediate links are running dynamically, so congestion could come and go. As could the supply and demand of any one active service (5Mbps for 5 minutes, then dropping to 2Mbps for 20 minutes).

In the end, there's just too much to keep track of to make a co-ordinated solution work.

The best solution is, once near congestion on an intermediate link, to just throw things away. And hope that the TCP implementations will back off a little. Things work somewhat better if the process of dropping packets is done in a relatively fair manner.

The older solution was to queue packets. Then to add bigger buffers to support bigger queues. Until they discovered this was bad for latency...
Internet engineers had created the phenomenon known as bufferbloat.
Bufferbloat introduction

From that page, there is a quote that is well worth remembering:
Quote
We can start by understanding how we got into this mess; mainly, by equating “The data must get through!” with zero packet loss.

Hating packet loss enough to want to stamp it out completely is actually a bad mental habit. Unlike real cars on real highways, the Internet’s foundational TCP/IP protocol is designed to respond to crashes by resending an identical copy when a packet send is not acknowledged. In fact, the Internet’s normal mechanisms for avoiding congestion rely on the occasional packet loss to trigger them. Thus, the perfect is the enemy of the good; some packet loss is essential.

The solution to bufferbloat is to stop using the buffers, and to start using a method that employs packet loss in a controlled manner. And one of the most important features is that this "packet loss in a controlled manner" can happen at any link in the network without complicated tuning to cope with other parts of the network. It doesn't attempt to coordinate or propagate information around ... the lost packet is, in fact, the "thing" that is propagated.

That page links to a set of videos that help explain bufferbloat too, and an IETF demo of the different algorithm (Codel or FQ_Codel) in operation.

I note that it includes videos on a history of network queueing theory.
Bufferbloat videos
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: Rate limiting - back-propagation
« Reply #3 on: April 08, 2017, 08:08:34 AM »

@wwwombat - I think I mentioned the scalability problem.

Presumably the reason that the likes of BT (and my own ISP as well, in turn) do this kind of rate-limiting is partly concerned with costs and perhaps partly concerned with implementing QoS. If you take it on yourself to drop packets earlier in the path then you can make the drop decisions, perhaps in a QoS friendly way.

They don't seem to want to just let TCP regulate the flows, that is perhaps understandable given the fact that not everyone is using TCP.
Logged

Chrysalis

  • Content Team
  • Kitizen
  • *
  • Posts: 4363
Re: Rate limiting - back-propagation
« Reply #4 on: April 08, 2017, 09:44:02 AM »

aaisp I think only do it because BTw require them to do it, all BTw CPs are required to implement the rate limit for their users.

aaisp implement some simple QoS where if they see lines pushing throughput around line rate I believe they prioritise smaller packets so they never dropped.  Plusnet run a QoS and actually set the profile their side lower than the BTw line rate profile to allow that QoS to work better.

The TCP self regulation works based on number of metrics.

On the sender side you have the send window and congestion window.  The congestion window controls how much data can be sent between each acknowledgement.  The larger it is, the faster data can potentially flow.  The send window is just a cap for the congestion window, so the congestion window cannot grow larger than the sender window.

On the receiver side you have the receive window (RWIN), which is also a cap for the congestion window, so ultimately the congestion window controls the flow of data, however indirectly both the send window and receive window can also limit the flow of data as they act as caps for the congestion window.  If an acknowledgement comes later or doesnt come at all this is typically considered as congestion and the congestion window will shrink to slow down the flow of data, in bad enough cases it can actually revert back to slow start.

In normal cases it should only be congestion on either end point, all intermediate routers shouldnt be bottlenecking the throughput but it does happen sometimes sadly.  If it does happen tho it causes the same affect, it will either delay or prevent the ack and as such slow down the rate of data, so in that respect it doesnt matter where the congestion is and there is no need for the sender to be aware of the specs of intermediate routers, congestion is congestion no matter where it occurs.

TCP isnt perfect, it is prone to bursting too high when rtt is very low, if there is many parallel TCP streams then that can also cause an overflow of data, this perhaps gives what BTw are doing some merit, however the overflow wont be significant, its not broken to the point where the sender will e.g. try to send a gigabit/sec of data down a 80mbit pipe.

Generally delayed packets will have a noticeably less impact on retransmits than packet drops, hence the motivation of certian isps like VM to apply buffer bloat systems.  However the side affect is as wombat said increased latency.

I concentrated my first reply around what BTw are doing with ip profiles tho as I assumed that is what was the point of your question.
Logged
Sky Fiber Pro - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: Rate limiting - back-propagation
« Reply #5 on: April 08, 2017, 10:42:31 AM »

Presumably in the case of AA they get charged by BTW so any excess traffic that they can drop will save them a bit of money. The default is that pipes are artificially limited to run at 95% of the reported maximum capacity. This gives VoIP more of a chance of working. You can change this rate yourself though, and I put mine up to 100%. In fact you can try it at more than 100%, I once did a very brief test but unfortunately I would have had to do a lot more measurements than I did at the time as there was too much noise in the figures I was getting from the particular speed tester I used.
Logged

Chrysalis

  • Content Team
  • Kitizen
  • *
  • Posts: 4363
Re: Rate limiting - back-propagation
« Reply #6 on: April 08, 2017, 11:03:30 AM »

It might be that yeah, I be surprised for single TCP streams if its anything significant however perhaps in situations e.g. steam downloads where is 30+ TCP streams the excess may add up to be enough to affect the balance sheet.

I think increasing your aaisp limit to over 100% probably wont have any affect as you will still have a rate limit applied on BTw's network which is independent of what the isp does.
Logged
Sky Fiber Pro - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: Rate limiting - back-propagation
« Reply #7 on: April 08, 2017, 03:09:23 PM »

I would agree about increasing the limit to be above 100%. It was just an experiment to find out what happens. The very dubious preliminary results suggested that performance might actually be slightly worse if anything
Logged

Weaver

  • Kitizen
  • ****
  • Posts: 4004
  • Retd sw dev; A&A; 3 × 7km ADSL2; IPv6; Firebrick
Re: Rate limiting - back-propagation
« Reply #8 on: April 11, 2017, 01:54:31 AM »

I did some more extensive tests. I tried setting the AA downstream rate limiter low - 99% then 95% - and then high - 110%. Nothing made any difference at all as measured by three different speed testers on multiple runs. Ookla is the most accurate and that did not even show any reduction at 95%, which is a bit odd. Maybe the speed tester is just too inaccurate and it is a small sample size. Other possibility that comes to mind is that maybe I have to do something magical to get the parameter changes to take effect.

I discovered that there is some documentation about the rate limiter settings, which shows pictures of the effects shown on the CQM graphs and says that the latency can be reduced by going  down from 100% to 95%, visible in the graph example shown.
Logged