Kitz Forum
Broadband Related => FTTC and FTTP Issues => Topic started by: Jon21 on March 28, 2022, 11:36:05 PM
-
[Moderator note: This post and all that follow have been split off from the "Are Zen still recommended? (https://forum.kitz.co.uk/index.php/topic,26860.0.html)" topic.]
[In the end] I’ve gone with AAISP again. Ironically, there one of the cheapest to actually get the line migrated away from Sky (the actual taking over the line cost, not the ongoing monthly cost). On 6 month initial term, so will re-evaluate then.
-
Migrated over to AAISP just after midnight and tbh, the BQM is a jittery mess. No idea why there’s so many latency spikes, surely not normal?
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/0106228d673c73357377277ff2b8fd6297c835f6-12-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/0106228d673c73357377277ff2b8fd6297c835f6-12-04-2022)
-
Have you looked at the latency in clueless.aa.net.uk ?
-
Have you looked at the latency in clueless.aa.net.uk ?
That appears to show a flat line for the same time period. Why would the BQM be spikey then? Nothing has changed with regards to equipment on my end. Only thing that was changed was the login details.
-
On FTTP does it make any difference, really? Same service, same fibre, all you're paying for is the same engineer to come round and fix it which it will rarely need.
-
On FTTP does it make any difference, really? Same service, same fibre, all you're paying for is the same engineer to come round and fix it which it will rarely need.
Whilst I agree if your on FTTP, AAISP probably wouldn’t be first choice. Is there really any need to come in to every thread that AAISP are mentioned and trash it? You don’t like them, that’s fine, it’s your money, you can choose whoever.
-
What a strange response!
The spikes on the graph I can’t believe would affect usage at all really and it will always depend on routing.
-
Whilst I agree if your on FTTP, AAISP probably wouldn’t be first choice. Is there really any need to come in to every thread that AAISP are mentioned and trash it? You don’t like them, that’s fine, it’s your money, you can choose whoever.
I was talking about Zen.
-
I was talking about Zen.
My apologies. I guess it depends if you need a static IP or some other features as to whether it would be worth it over another provider (for both FTTC/P).
-
My apologies. I guess it depends if you need a static IP or some other features as to whether it would be worth it over another provider (for both FTTC/P).
Static IP is certainly a big reason my options are limited, though I have no complaints about Zen, especially compared to the unreliability of Lebara and Three. I don't know how anyone copes with only mobile Internet as their broadband connection.
-
That appears to show a flat line for the same time period. Why would the BQM be spikey then? Nothing has changed with regards to equipment on my end. Only thing that was changed was the login details.
Tracert and a ping -t don't show the spikes. Had a quick chat on IRC, as it's not showing on the CQM and the line is in "training", I haven't really got any further. Bit disappointed to be honest. Clearly something different between how pings are responded to on the BQM between Sky and AAISP.
Might be worth the thread being split?
-
The problem with TBB BQM is there could be a problem specifically between your ISP and TBB which would show spikes where 99% of the time you would be taking a different path to other sites so not have the problem.
-
The problem with TBB BQM is there could be a problem specifically between your ISP and TBB which would show spikes where 99% of the time you would be taking a different path to other sites so not have the problem.
True. Perhaps it’s a problem that A&A just aren’t aware of. Just seemed to get stonewalled with doesn’t show on the CQM and your line is in training, which seems a bit more of a first line response to me. I see Chrysalis BQM also shows similar.
-
Migrated over to AAISP just after midnight and tbh, the BQM is a jittery mess. No idea why there’s so many latency spikes, surely not normal?
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/0106228d673c73357377277ff2b8fd6297c835f6-12-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/0106228d673c73357377277ff2b8fd6297c835f6-12-04-2022)
Mine is similar. 10+ years ago I would care more, but ultimately my ssh is still smooth, and I dont have any throughput or streaming issues so I accept it. Graphs in my sig.
-
Mine was similar on FTTC on both lines. That was with Zen previously with Plusnet it was slightly worse. I never noticed anything from the spike’s while using the connection
-
Well it only takes a single poll with high ping to create a yellow spike, a better indicator is the blue which represents the average, if thats quite thick compared to the green, then that will likely indicate consistent jitter.
-
I guess my main gripe, is why is an ISP that seems to pride itself on its network, not that bothered about it? If they were a bargain basement ISP, I’d just accept it. Sky didn’t seem to present that issue. That’s why I don’t buy the idea that because the line is in training, that could be what is causing the spikes. I had interleaving on with Sky, no issues there.
-
I guess because it’s probably not a fault, more what @chrys said. It’s very sensitive to network performance, not really an indicator of a fault.
-
I guess my main gripe, is why is an ISP that seems to pride itself on its network, not that bothered about it? If they were a bargain basement ISP, I’d just accept it. Sky didn’t seem to present that issue. That’s why I don’t buy the idea that because the line is in training, that could be what is causing the spikes. I had interleaving on with Sky, no issues there.
The problem with judging your line based on ping is that Sky could simply prioritise pings different on their core network to give the illusion of lower latency, especially if you're using their router where they could enable QoS to allow ping priority over other traffic. They could be completely congested and have a flat-line TBB graph if they configured the core routers to do so, thus why its only a rough guide.
As mentioned, the blue is the key as its the average ping time response. On a network obeying net neutrality, blue should be a good indication of jitter, if its barely visible it means pings are consistently around the same and the yellow as mentioned it just the odd response or two getting delayed for any number of reasons. Same with red, it just means some responses didn't return in the timeout window which I'd imagine is quite short considering how often you need to ping to generate the graphs, and the graphs scale would be useless if they expanded to show the odd 2-3 second delayed responses.
Notice even my VPS in a data centre is not free from yellow spikes:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/d8130f7f98dade9f0ec94cdd293a4a73fa5b5523-13-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/d8130f7f98dade9f0ec94cdd293a4a73fa5b5523-13-04-2022)
My Zen line (remember my router is grossly overpowered so is more likely to respond quicker):
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/43d191b07d1e27a7b1ec7bdbc10c03c7ff1b0387-13-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/43d191b07d1e27a7b1ec7bdbc10c03c7ff1b0387-13-04-2022)
Now compare that to a mobile Internet connection:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/b8cb5da599a37404eae35194a8827468608ae407-13-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/b8cb5da599a37404eae35194a8827468608ae407-13-04-2022)
-
A test would be to get a TBB pingbox to monitor a server at AA located where there’s very high bandwidth from there to the rest of the internet. You could compare the results of the TBB monitoring with ‘manual’ ping timings from elsewhere into that AA central server. With this I’m trying to establish whether or not there are queuing delays at the TBB pingbox or inside TBB’s network.
-
Might also be worth looking at https://control.aa.net.uk/.
My connection on there latency isnt spiky.
On my main smokeping which checks pings to my VDSL from germany, there is some spikes but they nothing like whats on TBB.
So it is important to not just rely on one thing as a performance metric, for me as my connection works well day to day for what I use it for its fine, and SSH command line is really sensitive to both jitter and packet loss, so I would notice on that if things were bad.
-
This is worth a read to help clarify why you are seeing more 'yellow' than you would like:
https://www.iwl.com/idocs/limitations-of-icmp-echo-for-network-measurement
Many people assume that ICMP Echo, more familiar as "ping", is a valid tool for measuring the performance and behavior of the internet.
That assumption is incorrect. Ping is a tool of value mainly for determining whether connectivity exists or not. Ping is a weak tool for measuring delay, variation in delay (jitter), and throughput.
-
I wonder how games usually do it. If they were sensible, they would use their own method by sending UDP packets over the actual port and protocol the game uses, the only way to get the true latency to the game server.
This seems useful too: https://kb.vander.host/operating-systems/how-to-do-udp-ping-with-linux/
-
Might also be worth looking at https://control.aa.net.uk/.
My connection on there latency isnt spiky.
On my main smokeping which checks pings to my VDSL from germany, there is some spikes but they nothing like whats on TBB.
So it is important to not just rely on one thing as a performance metric, for me as my connection works well day to day for what I use it for its fine, and SSH command line is really sensitive to both jitter and packet loss, so I would notice on that if things were bad.
This is the CQM for today so far. Even that is spiky at times, around 10am to 12pm ish. Yet the connection was mostly idle. Those spikes for max latency are showing as over 400ms. Admittedly, I haven't come across any noticable issues.
-
Have you tried putting a different modem on the line?
The intermittent yellow spiking on TBB graphs is a known 'feature' for several BCM based modems.
-
Have you tried putting a different modem on the line?
The intermittent yellow spiking on TBB graphs is a known 'feature' for several BCM based modems.
I’ve gone back to using a Draytek Vigor 130, seems to be more stable on my line.
-
Here are the AA CQM graphs for the most recent 24 hours for two of my lines (using PPP LCP ‘echo requests’):
(https://i.postimg.cc/bwtrVVLc/6-A2-ABCC5-94-A0-41-A7-8-D1-B-7-BD3-AB27-E170.png)
(https://i.postimg.cc/GtM2sZQV/8781-B360-9388-4-CA2-B2-B2-CEFE822-CDAF9.png)
-
Have you tried putting a different modem on the line?
The intermittent yellow spiking on TBB graphs is a known 'feature' for several BCM based modems.
Wasnt aware of that, interesting Kitz.
-
Jon21 wrote:
> That’s why I don’t buy the idea that because the line is in training, that could be what is causing the spikes.
I agree. You were fobbed off there, and I think you should consider complaining about a violation of AA’s stated ‘no bulls***’ policy. That response was not worthy of AA, and maybe the person you spoke to believed what they said in, which case the right answer should have been ‘I don’t know, I’ll ask Stuart’, not making up a load of random nonsense. They are allowed to not know some things. They don’t always get things right and they’ve really screwed up for me in the past at times but have always put things right and had a great attitude. (For example there was a huge amount of trouble once with a switch they sold me because of config problems, and it was out of action for a week, but I appreciated the huge difficulty of the diagnosis task, and was myself no hero in that I couldn’t solve it either, so I was very supportive of their problems.)
As for the problems with the network, I don’t agree. I’m not seeing them. But my network is not the same as yours.
-
Did AAISP really say it was down to line training? Missed that bit sorry, yeah as Weaver said bounce that back to them.
-
Yeah, I had, I guess an informal chat on their IRC channel, and line training was mentioned as a possible cause. Interleaving has been removed now though and still seeing the same spikes. Maybe it was being mistaken for minimum latency as I showed them the BQM. I wouldn’t of thought so but I certainly wasn’t convinced by the answer.
-
Especially as even if we consider temporary interleaving as line-training, that shouldn't cause spikes and the fact their own monitor doesn't show it would completely invalidate that excuse, it would impact all traffic going over your line equally.
The fact its happening on TBB but not their own places the blame entirely between TBB and AAISP. Though I'd still argue its not an indication there is actually a problem.
-
AAISP's CQM uses PPP LCP echo and the ThinkBroadband BQM uses ICMP Ping.
ICMP Ping can have have a much lower priority than other traffic which could explain why 1 shows latency spikes and the other doesn't.
I think someone already suggested it but it could help to setup another BQM (they allow 3 I think) and have it ping the AA gateway.
Also have you tried another router? Sky don't use PPP while AA do which might not play so well with your openwrt x86 router.
-
AAISP's CQM uses PPP LCP echo and the ThinkBroadband BQM users ICMP Ping.
ICMP Ping can have have a much lower priority than other traffic which could explain why 1 shows latency spikes and the other doesn't.
Thanks for confirming this, it conclusively proves IMO why CQM is the more the accurate result.
-
AAISP's CQM uses PPP LCP echo and the ThinkBroadband BQM uses ICMP Ping.
ICMP Ping can have have a much lower priority than other traffic which could explain why 1 shows latency spikes and the other doesn't.
I think someone already suggested it but it could help to setup another BQM (they allow 3 I think) and have it ping the AA gateway.
Also have you tried another router? Sky don't use PPP while AA do which might not play so well with your openwrt x86 router.
BQM for the gateway that is reported in Openwrt:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/9271a8bababbb6bd3f5cefa3d04e9d09761dfe88-16-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/9271a8bababbb6bd3f5cefa3d04e9d09761dfe88-16-04-2022)
I haven't tried another router as yet. I do have an Asus router that I could swap in for a short while, so will try that at some point over the weekend.
-
Thanks for confirming this, it conclusively proves IMO why CQM is the more the accurate result.
Unless your entire Internet usage is PPP LCP confined to A&A's network it's not more accurate. It's also not as simple as 'ICMP is lower priority'. On carrier grade routers with distributed packet processing and ASICs, so not Firebricks, ICMP to the router may be handled differently, ICMP through it is not.
CQM is not representative of user experience end to end it's a tool to diagnose potential issues across access network and backhaul between customer and LNS.
I wonder how games usually do it. If they were sensible, they would use their own method by sending UDP packets over the actual port and protocol the game uses, the only way to get the true latency to the game server.
Timestamps, sequence numbers and fixed tick count between client and server. Any or all of those depending on the game's protocol and, yes, they usually use the game's own port and protocol in-game, lobby varies.
-
Unless your entire Internet usage is PPP LCP confined to A&A's network it's not more accurate. It's also not as simple as 'ICMP is lower priority'. On carrier grade routers with distributed packet processing and ASICs, so not Firebricks, ICMP to the router may be handled differently, ICMP through it is not.
CQM is not representative of user experience end to end it's a tool to diagnose potential issues across access network and backhaul between customer and LNS.
Surely CQM proves there is no issue between the router and the ISP, which is the point?
If latency is elsewhere on the network its almost certainly not within AAISP so nothing they can do about it.
The problem is like you said, if your own router is slow to respond to ICMP then ping packets timing out would indicate latency that would not exist for normal traffic.
Another pitfall can be if you setup a DMZ, ICMP will get translated over NAT to the target machine, rather than being replied to via the router. I learned that the hard way.
Quote from MrSaffron themselves:
Each pixel is 100 seconds, so yellow may be just 1 or 2 samples (1/second) having higher latency, i.e. if the thin blue line does not move then what you have is jitter and this can be caused by simply loading a web page.
I've not had a jitter-free connection in years yet its clearly not my ISP as there are Zen users on various technologies in the BQM forum post (https://forum.kitz.co.uk/index.php/topic,21177.msg450176.html) with very different graphs.
My connection is never idle though, I have Wireguard, OpenVPN, pfSense pinging various servers for its gateway monitoring and plenty of port scans, web server and SSH attack attempts being blocked at the router. Also with multi-wan the firewall gets restarted whenever one of the mobile connections latency gets too high, more often than I'd like, which will temporarily kill response on all WANs. None of this is down to ISP issues.
-
https://www.ispreview.co.uk/index.php/2022/04/peak-time-latency-strikes-part-of-virgin-medias-uk-network.html
CQM would've been useless, all would have looked fine, meanwhile service heavily degraded, issue resolved by VM/Liberty.
Incorrect that A&A can't do anything about issues off their network or, in this case directly connected to it. Depending on where the issue is they certainly can.
Are you blaming the same LINX and NCUK routers that connect to most of the ISP customers in the UK for the latency spikes? It's either them or A&A.
You choose to have all that stuff running at home. You could have a single ping for each WAN to something running on a CDN, maybe a second just in case but 'various' helping prevent an issue is vanishingly unlikely.
Having multiple VPN services or P2P is your call to have running. Port scans shouldn't impact anything. If dropping a few packets a second is an issue for an x86 router it's not going to have a good time with gigabit.
The router response I referred to ASICs and distributed processing in carrier grade routers. Won't impact you as you're using x86, doesn't impact me as I don't have multiple VPNs running on my ARM-based routers and no P2P to induce buffer bloat due to microbursts.
At least I think it doesn't. I'm that guy that doesn't even run BQM as I don't see the point. Wonder if I'm the only anything like regular user that doesn't have CQM or BQM?
This brings up another thought actually. At some point PPP has to go. The technology is there now for MEF VLANs to replace it and is used on CityFibre. No idea if their national product allows PPP but it'll go eventually.
What are A&A going to use once there's no LCP echo to time?
-
I never said CQM was good for measuring all problems, I was specifically pointing out that its useful to eliminate the connection from you to the ISP as the problem, narrowing down where you need to look (though with a consideration that there might not actually BE a problem). Without PPP, I guess they will have to fall back to ping, though PPP is used as it means the same kit can be used for terminating all connections including L2TP and legacy connections.
I agree VLANs are better, removing overhead is always better (not least the CPU overhead on routers), but PPP remains for legacy reasons as its not appealing for an ISP to run two different methods for terminating/accounting the network. I even posted an article about this a few months back about why PPP needs to go. ;)
I think my BQM has been spiky ever since I switched from OpenWRT to pfSense, before I added all this extra cruft. Will be interesting to see if anything changes once I'm on FTTP.
Interesting mentioning bufferbloat, as while my Zen line isn't always busy, the routers LAN ports are. Though given its a dual-port LAG I kinda doubt that's causing any sort of bottleneck, plus as I said above, I don't think this spiky behaviour is new.
-
> What are A&A going to use when there’s no LCP echo to time?
Well as of course you know all Firebricks can do ICMP ping-based timing, as ThinkBroadBand does now. I do ICMP-based pings in my FB2900 to check that a certain link is really up and working, not just looking like it’s up. I ping the nearest node to me in AA-land to make sure it’s still talking, not to get timings.
> At some point PPP has to go.
I hear you, and you’re in good company as I’ve heard people within BT say just so, people who really know what they’re talking about. I can understand some of the arguments but I still don’t get it. If ASICs can parse other headers such as IPv4, IPv6, TCP and so on, those challenges are awkward enough, so just getting hardware to also parse PPP doesn’t sound like an impossible task. Let hardware speak PPP too. It’s only two bytes. More if you also throw in PPPoE or L2TP in addition.
-
https://www.ispreview.co.uk/index.php/2022/04/peak-time-latency-strikes-part-of-virgin-medias-uk-network.html
<snip>
interesting that a small press website has enough influence to cause VM to check their network. O_o
Also I hope you are right on PPP, it would be good if it was retired. But this will surely be down to whether BTw and its customers want to bother changing what already works for them?
-
interesting that a small press website has enough influence to cause VM to check their network. O_o
Also I hope you are right on PPP, it would be good if it was retired. But this will surely be down to whether BTw and its customers want to bother changing what already works for them?
It didn't. The issue was raised with the relevant people the day before and chased up that day.
BTW wanted rid of PPP for GEA but couldn't get it done for operational reasons.
-
Line has gone back to interleaved. There must be something that DLM doesn’t like when it goes on Fastpath. No idea what as everything appeared to be ok with ES, CRC etc. Guess that could be a separate issue that I can ask A&A about.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/4a347944d9987a82f47819a269ab853b58e40a9b-17-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/4a347944d9987a82f47819a269ab853b58e40a9b-17-04-2022)
-
Well as of course you know all Firebricks can do ICMP ping-based timing, as ThinkBroadBand does now. I do ICMP-based pings in my FB2900 to check that a certain link is really up and working, not just looking like it’s up. I ping the nearest node to me in AA-land to make sure it’s still talking, not to get timings.
Indeed. I was referring to the claim that ICMP is not reliable. I would hope the FireBrick can manage a simple reachability check!
I hear you, and you’re in good company as I’ve heard people within BT say just so, people who really know what they’re talking about. I can understand some of the arguments but I still don’t get it. If ASICs can parse other headers such as IPv4, IPv6, TCP and so on, those challenges are awkward enough, so just getting hardware to also parse PPP doesn’t sound like an impossible task. Let hardware speak PPP too. It’s only two bytes. More if you also throw in PPPoE or L2TP in addition.
So you can't have PPP without it being inside something to actually transport it which has to be Ethernet, and L2TP is how those VPNs are aggregated. Those L2TP tunnels are then stuck inside UDP and IP which in turn rides over MPLS/VPLS within the core and at some point is going to be within VLAN tags for segmentation on trunks. I'm ignoring Ethernet besides the PPPoE part That's a lot of encapsulation just for broadband.
Or you can place customers' IP into stacked VLANs and transport those over the network. Openreach and CityFibre deliver customers as stacked VLANs for instance.
I'm not an expert but would assume there are substantial gains to be made from this else TalkTalk and Sky wouldn't be doing it. PPP is a dialup technology, broadband is an always on connection. Time to treat customers in that manner rather than having them use what's basically unlimited dialup over Ethernet. Use the Ethernet header and you're good.
-
Line has gone back to interleaved. There must be something that DLM doesn’t like when it goes on Fastpath. No idea what as everything appeared to be ok with ES, CRC etc. Guess that could be a separate issue that I can ask A&A about.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/4a347944d9987a82f47819a269ab853b58e40a9b-17-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/4a347944d9987a82f47819a269ab853b58e40a9b-17-04-2022)
You can ask them to make sure you are on the "Speed" DLM policy.
The Standard policy only allows half the number of ES before applying interleaving.
-
Well PPP has caused some complaints in the opensource community, usually the pattern is someone complains of slow speed for the hardware, once its revealed they on PPP that gets blamed, its considered to be an inefficient, heavy on CPU, unable to be multithreaded and such. If its heavy on consumer devices it may also be on the ISP side, hence the gains for cityfibre?
-
I'm that guy that doesn't even run BQM as I don't see the point. Wonder if I'm the only anything like regular user that doesn't have CQM or BQM?
You are not alone. A certain grumpy old black cat shares the same view and has never used BQM or CQM.
As for the excentric who started that fad . . . :-X
-
Its useful to be able to look at if you are having problems, or to confirm QoS is working.
Though I just discovered pfSense does keep records of its gateway monitoring which is interesting.
Though I wish pfSense was more clever about its monitoring as it relies on the gateway responding to ping, not all ISPs do that. Would make more sense if it did PPP echos for PPP WANs, or monitored the quality of actual packet loss on the links themselves rather than relying on lost ping packets to detect a problem on the WAN.
I suspect its this relying on ping that is largely my problem with instability as it makes the mobile networks look less stable than perhaps they are, due to ICMP probably being low priority.
-
I agree with the earlier posters, I never ever look at CQM unless there’s a problem. I don’t obsess over it. I do look at it to see what’s been happening with traffic into/out of my LAN though, when I’m unsure about something. But I disagree with my dear friend Burakkucat, in that I do find it very useful indeed at times, I just don’t obsess about it.
-
In the I run it all the time camp but dont obsess.
If I had a problem with my broadband and then had nothing monitoring it to help diagnose it would be annoying.
-
I'm sure I have no idea what you mean about obsessing over stats. ::)
-
Nice, what UI is that from?
-
Based on my interpretation of the Star Trek LCARS interface.
Lots of fudgery in the background with PHP scripts polling SNMP, directly from the server itself, plus pulling stats from more PHP scripts running on pfSense.
I have my server boot straight into a full-screen window in Firefox to render it and I have 10" touch-screens in two different rooms hooked up to it. So I get a summary of the network and the server itself at a glance.
To keep things on topic, I might see if I can put WAN latency charts in that space underneath once I figure out how to extract the data from pfSense, as I do tend to login to pfSense a lot to look at that data when the mobile WANs are being glitchy.
-
Had a response from A&A:
Sorry for the delay. I suspect location here is the key. Our CQM will be more accurate because the Firebricks that monitor your connection are on the same network. If you're still seeing the latency spikes on Think BB graphs then maybe try pinging 81.187.81.187 and see if you see the same increase.
I've sent them a link to the BQM that is monitoring that IP address.
-
Will be interesting to hear what they have to say about it.
-
Will be interesting to hear what they have to say about it.
Still don’t think they are that bothered by it, despite giving them the BQM for the gateway. Do have a review date of Friday but I’m not sure if that is for the latency or whether it’s to do with the interleaving.
-
But if its not an issue, they might at least be able to explain why. They are the only ones who can really look into the routing and see if its a real-world problem or not.
Considering BQM is also using their hardware, they may also have an idea if there is a glitch at TTB.
-
You can ask them to make sure you are on the "Speed" DLM policy.
The Standard policy only allows half the number of ES before applying interleaving.
This was spot on. On the standard policy, looking to get it changed to speed, sometime next week.
-
Well something has changed just after 9pm.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/8b845406088bc0c0bf93de45ac69d63182809e0a-24-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/8b845406088bc0c0bf93de45ac69d63182809e0a-24-04-2022)
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/27c48849877bc5ab1dfd72d8dd74573e1eab65e0-24-04-2022.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/27c48849877bc5ab1dfd72d8dd74573e1eab65e0-24-04-2022)
-
Thats interesting, I had the same on my graphs ;), but then after midnight it went back to normal.
-
Well something has changed just after 9pm.
I also see some packet loss on BQM just after 21:00 also on my Zen FTTP monitor.
-
I also see some packet loss on BQM just after 21:00 also on my Zen FTTP monitor.
Same on my Zen FTTC and my AAISP L2TP connection over Vodafone 4G.
In fact looking at my list http://csdprojects.co.uk/ping/ I see it happened on Virgin too with Plusnet somehow unaffected.
-
First post here, just about to join AAISP tomorrow so this post caught my eye.
Leaving Aquiss/CityFibre for AAISP after issues with congestion on working days that starts a few minutes before 9am. CityFibre didn't really take much interest and closed the ticket on Aquiss as no fault found. The TTB BQM was really helpful in seeing the issue graphically.
What I noticed was that using f8lure.mouselike.org as a second opinion, which also sets up a chart from a Firebrick at AAISP (so you get 2 for 1), didn't always mirror the chart of TTB. Usually the AAISP chart showed no congestion issues, even though TTB and f8lure did. Just recently the AAISP chart is showing the same pattern of congestion, when sometimes TTB or f8lure are not showing, but whichever chart is showing congestion it always kicks in from a bit before 9am getting worse over the course of 10 or 15 minutes and continues, can be most of the working day or just really bad for an hour before recovering mid-morning, so seems to me several routings that CityFibre have me taking are all running close to capacity and easily get very congested with business traffic.
Whilst it might sound like I'm obsessing over the charts, I'm not, the issue is seen as generally slower speeds (at worse from 900Meg down to around 10Meg) and less than great video calls, just backed up graphically by the monitoring.
So a bit disappointing to read that AAISP with TTB monitoring there seems to be some issues with some packets taking longer as reported here. I'd recommend the OP sets up a second opinion via f8lure to see if it is a wider problem or just a routing issue to the Thinkbroadband network. When I switch over I will report back with my BQMs for comparison.
This is a normal day on Aquiss/CityFibre, being today, although it gets a bit rougher on the chart from 9am nothing that causes a problem:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/89c57ea7a1acc442fc1014e13cb5928a9f5df937-28-04-2022.png)
What happens on a bad day, CityFibre business traffic presumably. I've never known congestion at 9am in the morning with any ISP ever until now, 9pm maybe :-\:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/baff14ab58505410df8cd0bf0f10d4856659956f-19-04-2022.png)
-
If its showing on one chart and not another I'd say its almost certainly down to congestion on a route out of Aquiss and nothing to do with CityFibre. If it was a CityFibre problem it should occur equally across every path as it would be BEFORE reaching the ISP rather than after.
-
Well I'm on A&A. Will need to wait to see what the BQMs look like as I've been on and off setting things up so it's more red then anything else currently. ::)
pfSense was doing funny things. It seemed to get hold of an IPv4 gateway, then after a few seconds got a IPv6 gateway then the IPv4 gateway went off line in pfSense. I think I have fixed this now by deleting the IPv4 gateway and letting it dynamically add a new one a fresh. Then the IPv6 gateway was always showing offline in the gateway monitor but had already read about this on their wiki pages as their IPv6 gateway didn't respond to pings, so added bottomless to ping for monitoring, and that sorted that.
As you do on joining a new ISP I hit some speed tests, but couldn't get anything over about 650Meg from anywhere, including AAISP libra speed test. I had read about their LNS upgrades and faster connections to support FTTP higher speeds moving to new 'witless' LNSs, so I did a tracert and first hit was aimless, no witless. So following the instructions to get on to a witless LNS by prefixing beta- to the PPP login name, I was now showing first hop to a witless box, and speeds are now able to get up to around 920Meg. Odd I'm not getting a witless LNS by default being on 1000/100 FTTP so will raise it with them.
I'll drop back later with some BQMs.
If its showing on one chart and not another I'd say its almost certainly down to congestion on a route out of Aquiss and nothing to do with CityFibre. If it was a CityFibre problem it should occur equally across every path as it would be BEFORE reaching the ISP rather than after.
Aquiss don't have any kit at all, they just resell CityFibre wholesale products, and CityFibre do everything connectivity wise, even the IP addresses they give out and DNS servers all belong to CityFibre/Entanet. Whilst I'm sure many people have great service with Aquiss via CityFibre, when you get a networking issue then you are back to dealing with a large ISP so the odd person reporting an issue stands little chance of being taken seriously.
-
Welcome to the forum, EC300!
-
Okay I can confirm I'm getting the same looking BQM in between me dropping PPP whilst I've been configuring things. This appears to be an issue somewhere on the route to thinkbroadband, it isn't happening on f8lures chart, or on other locations I've tested using a pingplotter.
Still not connecting via witless LNS so speeds max out at around 600-650, but prefixing beta- and all is great, so I've sent support an email about it.
-
I have used f8lure.mouselike.org on occasion. Very useful.
-
Okay I can confirm I'm getting the same looking BQM in between me dropping PPP whilst I've been configuring things. This appears to be an issue somewhere on the route to thinkbroadband, it isn't happening on f8lures chart, or on other locations I've tested using a pingplotter.
Still not connecting via witless LNS so speeds max out at around 600-650, but prefixing beta- and all is great, so I've sent support an email about it.
Thanks. Would you be able to confirm where Pingplotter is showing the issue? I've attached a screenshot of a quick run of it it, but not entirely sure what it's showing.
-
What I do is click on each hop and see which chart tends to match the final destination as that normally indicates where the variability is being added. I couldn't see any pattern on any of those hops so I think that tends to point to it happening between hop 6 and 7.
It definitely is route related as f8lure doesn't show that pattern.
What are your speed tests like to Thinkbroadband? Mine are not as good as they were via CityFibre (outside of working hours) or IDNET prior. Hard to get anywhere near 800Meg at the moment on any speed test with AAISP, the worst ISP for top speeds I've had. Considering the premium and they promise to not slow you down it's a bit disappointing.
Speed test via AAISP
(https://www.thinkbroadband.com/_assets/speedtest/button/1651328148191624355.png)
Aquiss CityFibre on a Saturday
(https://www.thinkbroadband.com/_assets/speedtest/button/1650700813654272455.png)
I also posted this in the Thinkbroadband forum. The routing for a speed test server in London goes via Africa!
AAISP is coming in at 227ms for a round trip, and seems to go via Johannesburg. What are others seeing doing a tracert or ping to 197.227.5.218?
Via AAISP
Tracing route to 197.227.5.218 over a maximum of 30 hops
1 6 ms <1 ms <1 ms pfSense.localdomain [192.168.1.1]
2 6 ms 7 ms 7 ms y.witless.thn.aa.net.uk [90.155.53.134]
3 7 ms 7 ms 9 ms g.aimless.thn.aa.net.uk [90.155.53.47]
4 7 ms 7 ms 7 ms 8-1-1.ear1.London2.Level3.net [217.163.102.225]
5 * * * Request timed out.
6 * * * Request timed out.
7 200 ms 206 ms 200 ms be2385.ccr51.jnb01.atlas.cogentco.com [154.54.40.94]
8 227 ms 227 ms 227 ms 206.249.0.154
9 227 ms 227 ms 227 ms 197.226.230.42
10 227 ms 227 ms 238 ms 197.226.230.8
11 227 ms 227 ms 227 ms 197.227.5.218
Via a work connection
Tracing route to 197.227.5.218 over a maximum of 30 hops
1 1 ms <1 ms <1 ms *
2 3 ms 3 ms 3 ms *
3 3 ms 3 ms 3 ms *
4 3 ms 3 ms 3 ms *
5 3 ms 3 ms 3 ms *
6 3 ms 3 ms 3 ms be42973-10.man1.pag.gb.m247.com [164.39.60.82]
7 3 ms 3 ms 3 ms hu0-0-0-3.man1.agg1.gb.m247.com [217.138.224.20]
8 3 ms 3 ms 4 ms hu0-0-1-0.man1.edg.gb.m247.com [217.138.224.17]
9 3 ms 4 ms 3 ms xe-0-4-0-7.r02.londen03.uk.bb.gin.ntt.net [83.231.235.229]
10 8 ms 8 ms 9 ms ae-12.r20.londen12.uk.bb.gin.ntt.net [129.250.4.191]
11 24 ms 13 ms 9 ms ae-13.a03.londen12.uk.bb.gin.ntt.net [129.250.3.249]
12 * 9 ms * 195.219.23.72
13 8 ms 10 ms 8 ms if-ae-66-2.tcore1.ldn-london.as6453.net [80.231.60.144]
14 9 ms 8 ms 8 ms 195.219.83.158
15 8 ms 8 ms 8 ms 197.227.5.218
This IP is a speed test server which can be picked from the list at nperf.com (select Mauritius Telecom, in London with a 10Gb/s link). Previous ISPs the speed test came in at 900+, but with AAISP and that routing, lucky to get over a hundred!
(https://pic.nperf.com/r/3381921621277277-88i1rUgT.png)
-
Definitely a routing problem well outside AAISPs control:
1 l2tp.thn.aa.net.uk (90.155.53.19) 11.726 ms 11.623 ms 11.776 ms
2 g.aimless.thn.aa.net.uk (90.155.53.47) 11.804 ms 11.964 ms 12.133 ms
3 * 8-1-1.ear1.London2.Level3.net (217.163.102.225) 12.810 ms 12.515 ms
4 * * *
5 * * *
6 be2385.ccr51.jnb01.atlas.cogentco.com (154.54.40.94) 187.923 ms
be2489.ccr51.jnb01.atlas.cogentco.com (154.54.88.222) 188.749 ms
be2385.ccr51.jnb01.atlas.cogentco.com (154.54.40.94) 188.330 ms
7 206.249.0.154 (206.249.0.154) 232.307 ms 232.309 ms 233.144 ms
8 197.226.230.42 (197.226.230.42) 232.840 ms 215.135 ms
197.226.230.52 (197.226.230.52) 232.358 ms
9 197.227.5.218 (197.227.5.218) 231.891 ms
197.226.230.4 (197.226.230.4) 232.801 ms
197.227.5.218 (197.227.5.218) 231.997 ms
1 vt1.cor2.lond1.ptn.zen.net.uk (51.148.72.22) 12.433 ms 12.675 ms 12.781 ms
2 lag-9.p1.thn-lon.zen.net.uk (51.148.73.160) 12.534 ms lag-9.p2.thn-lon.zen.net.uk (51.148.73.174) 12.645 ms 13.310 ms
3 lag-1.br1.thn-lon.zen.net.uk (51.148.73.153) 13.183 ms 12.616 ms lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 13.307 ms
4 ldn-b3-link.ip.twelve99.net (213.248.84.100) 13.545 ms 13.407 ms 13.648 ms
5 * * *
6 be2385.ccr51.jnb01.atlas.cogentco.com (154.54.40.94) 189.989 ms 188.179 ms 188.092 ms
7 206.249.0.154 (206.249.0.154) 233.790 ms 253.497 ms 253.338 ms
8 197.226.230.42 (197.226.230.42) 446.773 ms 197.226.230.62 (197.226.230.62) 458.513 ms 196.20.225.96 (196.20.225.96) 466.971 ms
9 197.226.230.4 (197.226.230.4) 461.670 ms 442.344 ms 197.227.5.218 (197.227.5.218) 439.881 ms
1 vodafone (192.168.7.1) 0.321 ms 0.311 ms 0.244 ms
2 * * *
3 192.168.213.21 (192.168.213.21) 35.133 ms 42.625 ms 38.985 ms
4 192.168.213.22 (192.168.213.22) 41.990 ms 44.695 ms 45.016 ms
5 * * *
6 * * *
7 63.130.104.194 (63.130.104.194) 37.544 ms 39.148 ms 38.021 ms
8 ae13-100-xcr1.ltw.cw.net (195.2.23.77) 35.964 ms 41.788 ms 42.967 ms
9 * * *
10 be2348.ccr41.lon13.atlas.cogentco.com (130.117.51.73) 41.407 ms 27.691 ms 42.978 ms
11 be2870.ccr22.lon01.atlas.cogentco.com (154.54.58.174) 42.981 ms
be2868.ccr21.lon01.atlas.cogentco.com (154.54.57.154) 27.100 ms
be2572.ccr21.lon02.atlas.cogentco.com (154.54.61.254) 43.837 ms
12 be2485.ccr51.jnb01.atlas.cogentco.com (154.54.88.206) 215.353 ms
be2436.ccr51.jnb01.atlas.cogentco.com (130.117.0.90) 186.087 ms
be2389.ccr51.jnb01.atlas.cogentco.com (154.54.80.202) 221.990 ms
13 206.249.0.154 (206.249.0.154) 271.720 ms 257.713 ms 256.929 ms
14 197.226.230.62 (197.226.230.62) 485.985 ms
197.226.230.42 (197.226.230.42) 461.610 ms
196.20.225.96 (196.20.225.96) 469.629 ms
15 197.226.230.8 (197.226.230.8) 502.669 ms
197.227.5.218 (197.227.5.218) 467.627 ms
197.226.230.4 (197.226.230.4) 471.723 ms
1 three (192.168.8.1) 0.591 ms 0.338 ms 0.385 ms
2 * * *
3 172.25.211.65 (172.25.211.65) 34.176 ms 41.451 ms 32.893 ms
4 * * *
5 * * *
6 172.25.217.81 (172.25.217.81) 36.631 ms 29.921 ms 24.184 ms
7 * * *
8 172.25.195.150 (172.25.195.150) 32.632 ms 34.719 ms 35.486 ms
9 * * *
10 * hu0-1-0-0.rcr21.b015534-1.lon01.atlas.cogentco.com (149.11.142.121) 36.114 ms
hu0-4-0-3.agr21.lhr01.atlas.cogentco.com (149.14.198.73) 31.304 ms
11 be2185.ccr21.lon01.atlas.cogentco.com (154.54.61.62) 35.613 ms 29.447 ms 41.657 ms
12 206.249.0.154 (206.249.0.154) 253.934 ms
be3487.ccr41.lon13.atlas.cogentco.com (154.54.60.5) 64.009 ms
be2870.ccr22.lon01.atlas.cogentco.com (154.54.58.174) 23.901 ms
13 be2572.ccr21.lon02.atlas.cogentco.com (154.54.61.254) 33.278 ms *
196.20.225.50 (196.20.225.50) 245.575 ms
14 197.226.230.4 (197.226.230.4) 256.548 ms 267.329 ms
206.249.0.154 (206.249.0.154) 254.184 ms
15 196.20.225.96 (196.20.225.96) 253.007 ms
197.226.230.62 (197.226.230.62) 257.820 ms
197.226.230.40 (197.226.230.40) 248.042 ms
16 197.227.5.218 (197.227.5.218) 237.107 ms 262.340 ms 256.356 ms
-
It's a strange one, just for fun you can run a speed test against Mauritius Telecom on Speedtest.net as well, how slow can you go :-)
What are their aimless boxes, Firebricks? I know they've got faster Firebricks (witless) for faster FTTP connections, but I'm struggling to get anything near top whack for FTTP 1000/100, so just wondering if the aimless routers are a bit of a bottleneck at 2 x 1Gbps, which might explain the struggle maxing out my connection.
(https://www.speedtest.net/result/13098075986.png)
-
Looks like level3 routing weird to cogent network.
Does that speed test ISP have any other transit other than cogent or LINX peering?
-
Looks like level3 routing weird to cogent network.
Does that speed test ISP have any other transit other than cogent or LINX peering?
Looks more like an issue within Cogent itself to me.
10 be2348.ccr41.lon13.atlas.cogentco.com (130.117.51.73) 41.407 ms 27.691 ms 42.978 ms
11 be2870.ccr22.lon01.atlas.cogentco.com (154.54.58.174) 42.981 ms be2868.ccr21.lon01.atlas.cogentco.com (154.54.57.154) 27.100 ms be2572.ccr21.lon02.atlas.cogentco.com (154.54.61.254) 43.837 ms
12 be2485.ccr51.jnb01.atlas.cogentco.com (154.54.88.206) 215.353 ms be2436.ccr51.jnb01.atlas.cogentco.com (130.117.0.90) 186.087 ms be2389.ccr51.jnb01.atlas.cogentco.com (154.54.80.202) 221.990 ms
The only good traceroute we have is one that doesn't use Cogent to reach that IP.
-
> just wondering if the aimless routers are a bit of a bottleneck at 2 x 1Gbps, which might explain the struggle maxing out my connection.
That might well be right. I’m a very long time AA user. They are serious about the ‘not being the bottleneck’ thing, so talk to them if you’re getting any performance problems and chastise them suitably. :)
-
That might well be right. I’m a very long time AA user. They are serious about the ‘not being the bottleneck’ thing, so talk to them if you’re getting any performance problems and chastise them suitably. :)
Managing to maxout the speedtests this morning so I will see how it goes, if things start dropping off again I will raise it with them. As long as it doesn't drop down to < 15Meg with packet loss Tuesday morning at 9am (when businesses return after the bank holiday) as it did with Aquiss/CityFibre I'm doing better ::)
Looks like level3 routing weird to cogent network.
Does that speed test ISP have any other transit other than cogent or LINX peering?
Not sure about transit arrangements, maybe it will sort itself out.
-
AAISP changed their routing to lower the latency, and also confirmed the other ISP doesnt peer at LINX.
Also this is affecting other broadband ISP's as well.
https://forums.thinkbroadband.com/aaisp/t/4711473-routing-differences-aaisp-taking-the-scenic-route.html
My current traceroute (interleaving seems quite heavy about 13ms hit :( )
Tracing route to 197.227.5.218 over a maximum of 30 hops
1 22 ms 22 ms 22 ms y.witless.thn.aa.net.uk [90.155.53.134]
2 22 ms 22 ms 22 ms e.aimless.tch.aa.net.uk [90.155.53.45]
3 23 ms 22 ms 23 ms xe-0-1-0-3-1.r04.londen05.uk.bb.gin.ntt.net [192.80.17.249]
4 22 ms 23 ms 23 ms ae-7.r20.londen12.uk.bb.gin.ntt.net [129.250.4.140]
5 22 ms 22 ms 22 ms ae-13.a03.londen12.uk.bb.gin.ntt.net [129.250.3.249]
6 * * * Request timed out.
7 22 ms 22 ms 22 ms if-ae-66-2.tcore1.ldn-london.as6453.net [80.231.60.144]
8 22 ms 22 ms 22 ms 195.219.83.158
9 22 ms 22 ms 22 ms 197.227.5.218
-
Yes can confirm I'm seeing the same so all fixed now.
Tracing route to 197.227.5.218 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms pfSense.localdomain [192.168.1.1]
2 7 ms 7 ms 7 ms y.witless.thn.aa.net.uk [90.155.53.134]
3 7 ms 7 ms 7 ms e.aimless.tch.aa.net.uk [90.155.53.45]
4 7 ms 7 ms 7 ms xe-0-1-0-3-1.r04.londen05.uk.bb.gin.ntt.net [192.80.17.249]
5 7 ms 8 ms 8 ms ae-7.r20.londen12.uk.bb.gin.ntt.net [129.250.4.140]
6 7 ms 7 ms 7 ms ae-13.a03.londen12.uk.bb.gin.ntt.net [129.250.3.249]
7 * 7 ms 7 ms 195.219.23.72
8 7 ms 7 ms 7 ms if-ae-66-2.tcore1.ldn-london.as6453.net [80.231.60.144]
9 7 ms 7 ms 8 ms 195.219.83.158
10 7 ms 7 ms 7 ms 197.227.5.218
Speed tests are back up there as well.
(https://pic.nperf.com/r/3382265865030995-iQv1phbC.png)
-
Also this is affecting other broadband ISP's as well.
Blinks slowly, wondering if my post above that already showed this has become invisible.
Problem is its not limited to level3, even when routed to Cogent in London their own internal routing is broken.
Its great Andrew was able to manually route around it, but surely the bigger problem is its announcing a bad route to begin with?
-
Yeah pretty much, AAISP worked around it luckily for AAISP customers, but cogent have a problem to fix, and that UK ISP probably should arrange some LINX peering as well.
-
Just an update, Andrew from AAISP has picked up this over at Thinkbroadband. A routing change has been made to IPv4 from 9pm yesterday, which has, at least for me, given me a very good looking BQM. IPv6 has not been changed as of writing this. Route is now via lonap, however this has it seems reduced the speed test throughput on the TBB tester on IPv4 tests (now also via lonap on IPv4) in that I only see a flat line of around 350Mbps on single thread tests, typically this was always > 600, IPv6 tests good as normal on TBB going via the original routing. Just goes to show how things can flip around just by taking a different route.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/1d4652373d5844504cc187900d61223306b2bf0c.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/1d4652373d5844504cc187900d61223306b2bf0c)
-
I found about this today, and IPv6 I see also has the new routing now as well. :)