Kitz Forum
Computers & Hardware => Networking => Topic started by: Weaver on January 02, 2020, 03:47:24 AM
-
If you do a flat-out upload, and a flat-out download is also in progress, do you find that the download rate drops?
This happened recently and the download rate dropped a lot, also the latency / RTT went very high for a long time (~400ms).
-
Hi
This is quite normal and speedtesters like the one at ThinkBroadband will record this by pinging the connection during a download/upload then giving a rating, with A being the best. When you download, you also upload acknowledgements, if those acknowledgements become delayed or dropped because the upload is saturated, the other end thinks you are receiving the data too fast and so slows down.
People often call this buffer bloat.
The only way to resolve this is to set up a QoS on your router and prioritise acknowledgements. I've done this on pfSense, when enabled get an A rating at Thinkbroadband, when disabled it's D to F.
Regards
Phil
-
Hi Philip, I’m familiar with the technique of prioritising ACKs to maximise performance. The Firebrick is not sophisticated enough to have this feature or indeed any proper QoS, a limitation that is a shame and I would happily pay for a software upgrade. It does however have a very simple strategy of prioritising all small packets, aimed at VoIP, so ACKs will presumably be assisted by this? I don’t know what to do about piggybacked ACKs though.
This is how I get poor latency figures, bufferbloat of 120 ms typically, but up to 400ms in unusual situations, and max rtt off the scale occasionally (over 500ms I think) in odd cases. I can run the usual-case bufferbloat by reducing the egress rate in the tx rate limiters by 1 or 2% below the rate for the usual best tx-only case max throughput figure. An upstream rate reduction of 1-2% or so is literally unnoticeable, doesn’t seem to actually worsen things at all, just reduces the RTT right down towards the ultimate minimum, which is 50ms my spreadsheet tells me that an RTT if around 150ms -nwhich is higher than necessary - should keep upstream tx queue bubbles to effectively zero and ensure that upstream tx is truly flat out when it’s supposed to be.
-
Saturating upload is playing hell with my IoT devices. I picked a bad time to start transition and, temporarily, simplify my home network at the expense of functionality.
-
It does however have a very simple strategy of prioritising all small packets, aimed at VoIP, so ACKs will presumably be assisted by this?
A simple absolute prioritisation of small packets worked very well when I tried it. I was using a Cisco router that makes that sort of thing really easy, and prioritising packets under 200 bytes meant I could max out the upload without any apparent affect on download speed.
Are you able to configure queue depth on the Firebrick? To me this whole "bufferbloat" idea is a symptom of poor design. Networks do drop packets and protocols adapt, so it's silly to queue packets for hundreds of milliseconds. If you ran a shorter queue then packets would be dropped rather than delayed. TCP doesn't mind a few ACKs being lost as long as there's always one reaching the sender before the window empties. With a long queue all ACKs are late.
-
You’re right, it’s silly to queue multiple ACKs within a single flow. But if there are lots of flows then the many alien competing packets might be delaying your small packet. There could be many flows and then it’s difficult to know what to do. Need to dramatically slow down the flows, apply flow-control / back pressure
<rant>Routers ought to be flow-aware, measuring the second-address/dest-adds/src-port/dest-port (if the protocol has ports)/protocol/flow-id (if IPv6) - or, if it’s IPv6 and you have a 20 bit flow-id that is non-zero then you just look at the flow-id alone, and nothing else, so that tells you whether or not there are ‘ports’ in the header so you don’t have to understand every random type of L4 protocol header format there is.</rant>
Re Firebrick - I haven’t seen anything about limiting the queue depth. Shame. It doesn’t seem to have any anti-bufferbloat.
-
<rant>Routers ought to be flow-aware, measuring the second-address/dest-adds/src-port/dest-port (if the protocol has ports)/protocol/flow-id (if IPv6) - or, if it’s IPv6 and you have a 20 bit flow-id that is non-zero then you just look at the flow-id alone, and nothing else, so that tells you whether or not there are ‘ports’ in the header so you don’t have to understand every random type of L4 protocol header format there is.</rant>
Many do, probably most business class gear. See https://www.cisco.com/en/US/docs/ios/12_0t/12_0t5/feature/guide/cbwfq.html or just the following snippet ..
"Flow classification is standard WFQ treatment. That is, packets with the same source IP address, destination IP address, source Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) port, or destination TCP or UDP port are classified as belonging to the same flow. WFQ allocates an equal share of bandwidth to each flow. Flow-based WFQ is also called fair queueing because all flows are equally weighted."
Or https://wiki.mikrotik.com/wiki/Manual:Queues_-_PCQ
-
I have quite a wish list for the Firebrick in terms of software upgrades / features, and active intelligent queue management and priority queue assignments according to L2, L3 and also L4-awareness (eg ACKs) for TCP and SCTP and arbitrary match rules are in that list. Ordinary very good home routers sometimes have these features. The MikroTik spec looks perfect. I would add header remarking at L2 and L3 as well, if they don’t already have such, as well as classifier-matching against priority markers in L2 and L3 headers.
-
I would add header remarking at L2 and L3 as well, if they don’t already have such, as well as classifier-matching against priority markers in L2 and L3 headers.
I must admit I haven't checked what L2 QoS is supported by Mikrotik. However nowadays we virtually never worry about L2 QoS for a couple of reasons. Firstly the CoS field only appears in the 1Q header so it's only applicable to trunks and not to access ports. Secondly all the kit that we deploy nowadays can do QoS on L3 parameters, even if the switch is only forwarding at L2. I suppose that's down to more processing power and ASICs. Originally we would configure policies to read L3 properties and set L2 CoS for the transmitted frame.
-
Think a pretty valid assumption is made that the layer 2 network shouldn't be congested and, if an uplink port is saturated, it's really easy to add another and LACP them.
As mentioned 802.1p requires 802.1q though the use case is pretty limited. Mostly used for voice VLANs to ensure large data transfers don't interfere with calls.
I go out of my way to try and avoid having switches do anything other than 802.1q. Ensures they can forward at line rate and keeps things simple.
Marking packets is only really of value if the WAN network has QoS on it that honours marking. MPLS networks may have some CoS on them so setting DSCP may be useful but for basic Internet usage DSCP marking has no real value.
Routers matching based on 802.1p is not really necessary. Segment the traffic into a VLAN with its own layer 3 SVI on the router.
Regarding your smaller packets anything that does WFQ should be fine. I am surprised the Firebrick can't do WFQ. However, I am also a little surprised that the Firebrick, regardless of model, has no 10 Gb ports or SFP+ cages. A&A would do well to either get shot of them in their network or get SFP+ cage models that can handle 10 Gb. Our kit does far more than Firebricks do and can handle 25 Gb+ in a single rack unit.
aesmith: you're a superstar. Spot on.
-
Although I have studied QoS, the only real application in my opinion is for VoIP within the enterprise. Outside of that, nobody should be using QoS. If you don't have enough bandwidth and are having to control traffic then you need more bandwidth. It's merely a sticking plaster for anything else.
Obviously, where you physically CANNOT increase bandwidth, then obviously some prioritisation within the LAN is worthwhile - i.e. torrent traffic low priority (example)
-
> we would configure policies to read L3 properties and set L2 CoS for the transmitted frame.
Which makes a lot of sense.
I would pay more for a further software enhancement ticket, like the full or basic (whatever it‘s called) option, for the Firebrick.
-
> we would configure policies to read L3 properties and set L2 CoS for the transmitted frame.
Which makes a lot of sense.
It made sense back in the day when L2 switches could only make queuing decisions based on L2 parameters. Nowadays any half decent switch can look directly at DSCP or other L3 properties to make these decisions, with the additional benefit of being able to prioritise on access ports as well as trunks.
-
Agreed, however KISS - no point in activating QoS / rate limiting / whatever anywhere besides bottlenecks.
The fewer buttons you turn on the less there is to go wrong.
Nothing wrong with playing with it all for a home lab though.
You've just made me realise something: I've been working in this field for 20 years and have not once used any kind of QoS / rate limiting / whatever on layer 2 switches.
-
You've just made me realise something: I've been working in this field for 20 years and have not once used any kind of QoS / rate limiting / whatever on layer 2 switches.
I think the main reason was almost political. When IP telephony first came out people were worried about putting voice on the LAN, and to some extent suppliers were worried that their flagship product might be shown up, what with all the PBX manufacturers ready to jump on any suggestion of problems. So in early deployments we always had end to end QoS.
-
Well for me, with my pipe that is both slow and weird (because it’s 4 ways IP-bonded), VoIP never really worked properly - Mine wasn’t good enough to ship as a professional release grade system - an early iffy beta stage. Audio quality problems and unreliability.
I can’t imagine how it could work for me without serious QoS what with the slowness, bloat and full loading and latency of 40 ms minimum because of ADSL with interleave. But maybe that’s just my ignorance and paranoia.
-
Could you send your voice just on one line? Your talking around 85K for one RTP stream so plenty of room even on an ADSL1 uplink, assuming properly prioritised.