All my DSL links and my fixed location 3G link went ‘up’ and ‘down’ like a yo-yo yesterday (Sunday 2020-09-06 16:00) although not both at the same time. This would seem to be due to PPP LCP packet loss, not DSL links themselves really dropping. That is, if my router detects packet loss as defined by absence of quality monitoring PPP LCP echo request responses then it would drop the link. Also if AA sees the same thing from their end then they would drop the link and send me a report. Also the clueless.aa.net.uk control and monitoring server lost the vital constant quality monitoring data that forms graphs of line activity and reliability for a critical part of the badness period.
Even the 3G link into my router went apparently bad from 16:00 to 17:00, although the time period is not the same as the badness of the other links that are via DSL.
AA issued this report:
AAISP Incidents: [MSO] Broadband: Some BT/TT lines dropping and packetloss since 4PM
[MSO] Broadband: Some BT/TT lines dropping and packetloss since 4PM
Posted: 06 Sep 2020 12:19 PM PDT
Started: 2020-09-06 16:00:00
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Update 6 Sep 2020 18:31:40 We're seeing BT lines on three out of nineteen LNSs having high packetloss and PPP drops, we're still investigating.
Update 6 Sep 2020 18:55:52 We've just disabled one of our hostlinks to BT, this would have caused a PPP drop, and we're waiting to see if this improves service.
Update 6 Sep 2020 19:00:14 We're still seeing loss and are still investigating.
Update 6 Sep 2020 19:09:58 A large number of lines (Both BT and TalkTalk) would have just lost connection, and should reconnect shortly. Please to bear with us as we continue to investigate this fault.
Update 6 Sep 2020 19:38:06 We currently have some BT lines with packetloss and some TalkTalk lines which have PPP failures when trying to reconnect, we're still working on this.
Update 6 Sep 2020 19:52:30 Quick summary of where we are at the moment: We're having problems with our networking in our Equninx datacentre which has interconnects to BT and TalkTalk. We also BT and TalkTalk have interconnects in our Telehouse datacentre too - which are ok. The problem is, is that the Equinix side is half not working, which is causing these problems.
Update 6 Sep 2020 20:00:59 Having disabled our TalkTalk interconnect in Equinix, the affected TalkTalk lines are reconnecting now. We still have some BT lines with packet loss, which we are still investigating.
Update 6 Sep 2020 20:07:03 BT lines which were affected by packet loss are now looking better (ie normal!)
Update 6 Sep 2020 20:13:50 Service should be restored for everyone. We have moved all our BT and TalkTalk traffic away from our Equinix datacentre and service is all being routed via Telehouse (Usually traffic would be split roughly 50/50). This is a temporary fix whilst we investigate the actual cause of this evenings problems. The problem is to do with our network and not related to to datacentre or BT/TalkTalk problems specially, we will continue with our investigations. Update expected: 2020-09-07 13:00:00