Hello,
We've been made aware of this thread by one of our customers so though it would be right to reply and add some more context.
I don't think it's fair to say that there was any failure by us because as orange4 says, a lot happened behind the scenes.
What did help overall is that there were a lot of AA users that were adding dialog here to compare their issue. There was the odd customer with us too. Sadly, the one or two customers with us that did see the issue didn't report it to us based on the posts here.
We had one solid report of the issue, which we didn't back down on. Another report but with throughput that varied wildly and one that was right most of the time. With this, TTB were working with very little information which looked like a single line fault. Had we had more lines, we'd have had more data.
We've spent several weeks arguing, fighting and pushing TTB. We've had their networks team on this, their head of networks; all checking configurations, showing us the output from switches and all of which seemed correct - they were being as transparent as they could be. They simply couldn't find the issue. If indeed the configuration change required is as now believed, it is as simple as few lines of config but could have been overlooked and may have just been due to a new pair of eyes checking it over and identifying it, or the configuration was as Juniper recommends which wasn't best for the use case.
Added: One of the first things we suggested to them was an incorrect CoS configuration on 14th May a day after the first report. They had concluded this was set correctly and the configuration of that and their NGE were all correct. They allegedly checked this again twice after that date around 2 weeks apart. The need to review and explain this lapse will be included in a report we've demanded they compile on the incident.
When we had the first report, we posted in a private group of which some ISPs are present who also use TTB to see if anyone else was impacted. We had no responses. This didn't help prove the issue to TTB, because as far as they were concerned, they had no other reports - which may very well be the case.
Only after a post by Andrew in that group a few days ago could we both approach TTB and show evidence the issue was bigger than we initially thought.
We offer a 40/10 service, and according to a post with Andrew today on that group, AA don't. We did not have any 40/10 user report this and from the understanding of the issue, those with lower-rate syncs would not have noticed. Even those customers who would have been impacted, might not be too worried about the reduction or as is evident, even report.
TTB are extremely hard to deal with when you believe there is an issue but only have the odd line affected (from those reported), so yes, whilst they did have more to work with because they then had another partner report the problem, I don't feel it's fair to say we deserve no credit at all.
For PhilipD, banger or anyone else with us, if you ever see a problem please do contact us. It would have helped us massively in getting a quicker resolution for all as has happened here. Additionally, we're automatically crediting those we can identify as potentially impacted for the full duration that we can correlate the issue to so please get in touch for that if you've not already had an email.
Matt