Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Jon21]

Yeah, I had, I guess an informal chat on their IRC channel, and line training was mentioned as a possible cause. Interleaving has been removed now though and still seeing the same spikes. Maybe it was being mistaken for minimum latency as I showed them the BQM. I wouldn’t of thought so but I certainly wasn’t convinced by the answer.

[Alex Atkin UK]

Especially as even if we consider temporary interleaving as line-training, that shouldn't cause spikes and the fact their own monitor doesn't show it would completely invalidate that excuse, it would impact all traffic going over your line equally.

The fact its happening on TBB but not their own places the blame entirely between TBB and AAISP.  Though I'd still argue its not an indication there is actually a problem.

[j0hn]

AAISP's CQM uses PPP LCP echo and the ThinkBroadband BQM uses ICMP Ping.

ICMP Ping can have have a much lower priority than other traffic which could explain why 1 shows latency spikes and the other doesn't.

I think someone already suggested it but it could help to setup another BQM (they allow 3 I think) and have it ping the AA gateway.

Also have you tried another router? Sky don't use PPP while AA do which might not play so well with your openwrt x86 router.

[Alex Atkin UK]

AAISP's CQM uses PPP LCP echo and the ThinkBroadband BQM users ICMP Ping.

ICMP Ping can have have a much lower priority than other traffic which could explain why 1 shows latency spikes and the other doesn't.

Thanks for confirming this, it conclusively proves IMO why CQM is the more the accurate result.

[Jon21]

AAISP's CQM uses PPP LCP echo and the ThinkBroadband BQM uses ICMP Ping.

ICMP Ping can have have a much lower priority than other traffic which could explain why 1 shows latency spikes and the other doesn't.

I think someone already suggested it but it could help to setup another BQM (they allow 3 I think) and have it ping the AA gateway.

Also have you tried another router? Sky don't use PPP while AA do which might not play so well with your openwrt x86 router.

BQM for the gateway that is reported in Openwrt:


I haven't tried another router as yet. I do have an Asus router that I could swap in for a short while, so will try that at some point over the weekend.

[Reformed]

Thanks for confirming this, it conclusively proves IMO why CQM is the more the accurate result.

Unless your entire Internet usage is PPP LCP confined to A&A's network it's not more accurate. It's also not as simple as 'ICMP is lower priority'. On carrier grade routers with distributed packet processing and ASICs, so not Firebricks, ICMP to the router may be handled differently, ICMP through it is not.

CQM is not representative of user experience end to end it's a tool to diagnose potential issues across access network and backhaul between customer and LNS.

I wonder how games usually do it.  If they were sensible, they would use their own method by sending UDP packets over the actual port and protocol the game uses, the only way to get the true latency to the game server.

Timestamps, sequence numbers and fixed tick count between client and server. Any or all of those depending on the game's protocol and, yes, they usually use the game's own port and protocol in-game, lobby varies.

[Alex Atkin UK]

Unless your entire Internet usage is PPP LCP confined to A&A's network it's not more accurate. It's also not as simple as 'ICMP is lower priority'. On carrier grade routers with distributed packet processing and ASICs, so not Firebricks, ICMP to the router may be handled differently, ICMP through it is not.

CQM is not representative of user experience end to end it's a tool to diagnose potential issues across access network and backhaul between customer and LNS.

Surely CQM proves there is no issue between the router and the ISP, which is the point?

If latency is elsewhere on the network its almost certainly not within AAISP so nothing they can do about it.

The problem is like you said, if your own router is slow to respond to ICMP then ping packets timing out would indicate latency that would not exist for normal traffic.

Another pitfall can be if you setup a DMZ, ICMP will get translated over NAT to the target machine, rather than being replied to via the router.  I learned that the hard way.

Quote from MrSaffron themselves:

Each pixel is 100 seconds, so yellow may be just 1 or 2 samples (1/second) having higher latency, i.e. if the thin blue line does not move then what you have is jitter and this can be caused by simply loading a web page.

I've not had a jitter-free connection in years yet its clearly not my ISP as there are Zen users on various technologies in the BQM forum post with very different graphs.

My connection is never idle though, I have Wireguard, OpenVPN, pfSense pinging various servers for its gateway monitoring and plenty of port scans, web server and SSH attack attempts being blocked at the router.  Also with multi-wan the firewall gets restarted whenever one of the mobile connections latency gets too high, more often than I'd like, which will temporarily kill response on all WANs.  None of this is down to ISP issues.

[Reformed]

https://www.ispreview.co.uk/index.php/2022/04/peak-time-latency-strikes-part-of-virgin-medias-uk-network.html

CQM would've been useless, all would have looked fine, meanwhile service heavily degraded, issue resolved by VM/Liberty.

Incorrect that A&A can't do anything about issues off their network or, in this case directly connected to it. Depending on where the issue is they certainly can.

Are you blaming the same LINX and NCUK routers that connect to most of the ISP customers in the UK for the latency spikes? It's either them or A&A.

You choose to have all that stuff running at home. You could have a single ping for each WAN to something running on a CDN, maybe a second just in case but 'various' helping prevent an issue is vanishingly unlikely.

Having multiple VPN services or P2P is your call to have running. Port scans shouldn't impact anything. If dropping a few packets a second is an issue for an x86 router it's not going to have a good time with gigabit.

The router response I referred to ASICs and distributed processing in carrier grade routers. Won't impact you as you're using x86, doesn't impact me as I don't have multiple VPNs running on my ARM-based routers and no P2P to induce buffer bloat due to microbursts.

At least I think it doesn't. I'm that guy that doesn't even run BQM as I don't see the point. Wonder if I'm the only anything like regular user that doesn't have CQM or BQM?

This brings up another thought actually. At some point PPP has to go. The technology is there now for MEF VLANs to replace it and is used on CityFibre. No idea if their national product allows PPP but it'll go eventually.

What are A&A going to use once there's no LCP echo to time?

[Alex Atkin UK]

I never said CQM was good for measuring all problems, I was specifically pointing out that its useful to eliminate the connection from you to the ISP as the problem, narrowing down where you need to look (though with a consideration that there might not actually BE a problem).  Without PPP, I guess they will have to fall back to ping, though PPP is used as it means the same kit can be used for terminating all connections including L2TP and legacy connections.

I agree VLANs are better, removing overhead is always better (not least the CPU overhead on routers), but PPP remains for legacy reasons as its not appealing for an ISP to run two different methods for terminating/accounting the network.  I even posted an article about this a few months back about why PPP needs to go.

I think my BQM has been spiky ever since I switched from OpenWRT to pfSense, before I added all this extra cruft.  Will be interesting to see if anything changes once I'm on FTTP.

Interesting mentioning bufferbloat, as while my Zen line isn't always busy, the routers LAN ports are.  Though given its a dual-port LAG I kinda doubt that's causing any sort of bottleneck, plus as I said above, I don't think this spiky behaviour is new.

[Weaver]

> What are A&A going to use when there’s no LCP echo to time?

Well as of course you know all Firebricks can do ICMP ping-based timing, as ThinkBroadBand does now. I do ICMP-based pings in my FB2900 to check that a certain link is really up and working, not just looking like it’s up. I ping the nearest node to me in AA-land to make sure it’s still talking, not to get timings.

> At some point PPP has to go.

I hear you, and you’re in good company as I’ve heard  people within BT say just so, people who really know what they’re talking about. I can understand some of the arguments but I still don’t get it. If ASICs can parse other headers such as IPv4, IPv6, TCP and so on, those challenges are awkward enough, so just getting hardware to also parse PPP doesn’t sound like an impossible task. Let hardware speak PPP too. It’s only two bytes. More if you also throw in PPPoE or L2TP in addition.

[Chrysalis]

https://www.ispreview.co.uk/index.php/2022/04/peak-time-latency-strikes-part-of-virgin-medias-uk-network.html
<snip>

interesting that a small press website has enough influence to cause VM to check their network. O_o

Also  I hope you are right on PPP, it would be good if it was retired.  But this will surely be down to whether BTw and its customers want to bother changing what already works for them?

[Reformed]

interesting that a small press website has enough influence to cause VM to check their network. O_o

Also  I hope you are right on PPP, it would be good if it was retired.  But this will surely be down to whether BTw and its customers want to bother changing what already works for them?

It didn't. The issue was raised with the relevant people the day before and chased up that day.

BTW wanted rid of PPP for GEA but couldn't get it done for operational reasons.

[Jon21]

Line has gone back to interleaved. There must be something that DLM doesn’t like when it goes on Fastpath. No idea what as everything appeared to be ok with ES, CRC etc. Guess that could be a separate issue that I can ask A&A about.

[Reformed]

Well as of course you know all Firebricks can do ICMP ping-based timing, as ThinkBroadBand does now. I do ICMP-based pings in my FB2900 to check that a certain link is really up and working, not just looking like it’s up. I ping the nearest node to me in AA-land to make sure it’s still talking, not to get timings.

Indeed. I was referring to the claim that ICMP is not reliable. I would hope the FireBrick can manage a simple reachability check!

I hear you, and you’re in good company as I’ve heard  people within BT say just so, people who really know what they’re talking about. I can understand some of the arguments but I still don’t get it. If ASICs can parse other headers such as IPv4, IPv6, TCP and so on, those challenges are awkward enough, so just getting hardware to also parse PPP doesn’t sound like an impossible task. Let hardware speak PPP too. It’s only two bytes. More if you also throw in PPPoE or L2TP in addition.

So you can't have PPP without it being inside something to actually transport it which has to be Ethernet, and L2TP is how those VPNs are aggregated. Those L2TP tunnels are then stuck inside UDP and IP which in turn rides over MPLS/VPLS within the core and at some point is going to be within VLAN tags for segmentation on trunks. I'm ignoring Ethernet besides the PPPoE part That's a lot of encapsulation just for broadband.

Or you can place customers' IP into stacked VLANs and transport those over the network. Openreach and CityFibre deliver customers as stacked VLANs for instance.

I'm not an expert but would assume there are substantial gains to be made from this else TalkTalk and Sky wouldn't be doing it. PPP is a dialup technology, broadband is an always on connection. Time to treat customers in that manner rather than having them use what's basically unlimited dialup over Ethernet. Use the Ethernet header and you're good.

[j0hn]

Line has gone back to interleaved. There must be something that DLM doesn’t like when it goes on Fastpath. No idea what as everything appeared to be ok with ES, CRC etc. Guess that could be a separate issue that I can ask A&A about.

You can ask them to make sure you are on the "Speed" DLM policy.
The Standard policy only allows half the number of ES before applying interleaving.