Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Jasonkruys]

Hi All,

Hi all, I'm hoping someone with more intimate knowledge of the OpenReach setup might be able to give me some insight. Zen have been working with Openreach for 2 weeks now to try and resolve, and in that time I've wasted two days waiting in for OR that didn't turn up because they tried 'remote' fixes without telling me, and a 2 hour engineer visit that hasn't fixed it.

Service: FTTP 1000/110

Issue: Frequent (sometimes for hours at a time) crashing of download speed to <500kb/s to the point where it appears I have no service. Upload seems largely unaffected. Prior to that I have had nearly a year of perfect service.

The issue has been seen on:
My Own router,
Zen's FritzBox router,
A Laptop directly connected to the ONT initiating the PPPoE connection.
The original ONT
A new ONT (changed at the 3rd, actually attended, OR visit)[/li][/list]

Over the last 2 weeks the following has been done:

Zen have re-routed me on their network to rule out a known compatibility issue
Open Reach have applied various 'undisclosed' network fixes/adjustments - hence the two wasted days in waiting for them

An engineer attended, saw the issue, and through initially 1st line, then 2nd line:

Performed Multiple tests from the Operations Team that showed OK
Applied Multiple 'undisclosed' network adjustments
Light levels OK at ONT (17.7)
Light levels reported ok by remote ops team across the whole circuit and at my ONT (17.7 and 20)
SSFP test run - showing OK at 940Mb/s at the port - where does this test terminate? Is it the exchange?
Circuit delete and rebuild
ONT Swap in the house
Re-spliced fibre at CSP (just in case - it looked OK)
CBT and Splitter checked for 'disturbance' (they are 10m from my CSP in an under-pavement capsule next to each other) - no issues seen.

Following continued issues, OR have again checked light levels across the network and "performed a series of resets across each part to clear off any congestion or unseen issues" - still no change.

Using the TBB BQM, it shows Packet-loss of up to 10% (according to the scale) that correlate directly with the periods of extremely low speeds - suggesting (in my mind) a data/electronic problem, rather than a fibre/light problem. The intermittent packet-loss started at exactly the point where I started seeing issues, with a completely clean graph before then. So far, OR seem unwilling to consider there is defective/failing hardware anywhere, and repeatedly apply 'soft' fixes to no effect. As far as I know, no-one has even been to the exchange yet..

Anyone with any insight? @Black Sheep?

The SSFP test that was run reported 940Mb/s at a time where the OR engineer and myself were seeing <20Mb/s at the ONT in the house - where does the OR SSFP test terminate, and what is between that and my ONT? Presumably the test does not terminate at the ONT? Given upload seems to be largely unaffected, is there a component of the network that comes AFTER the SSFP test termination point, AND has a different circuit (even on the same card?) to the Upload path?

I've provided all data I have on speeds (testing by CLI and plotting with Grafana every 15 mins to see if there was a pattern), and the packet-loss indications from the TBB BQM to Zen, and to be fair they have been excellent in checking in with me, keeping me informed and pursuing this - but so far 2 weeks and no progress! There appears to be no pattern to the drops - does not correlate with Peak times, Heat/weather - it has been awful since 0415 this morning so far peaking at 20Mb/s, but as low at 349kb/s.

My background is data-comms, mostly wireless, but I do have experience fault finding 'mobile' trunk communications that also have fibre links. Not that OR or anyone puts any stock in my thoughts, my nature is to use testing and data points to narrow down - hence my interest in the exact set-up at the exchange.

Apart from some of the things that happened whilst the engineer was here, feedback from OR on what they have actually done/tried is very hard to come by. The last email I got suggested they may have run out of ideas, but I do believe that Zen will keep pushing it - they have been great.

Any insight, even for my own curiosity, would be great. Thanks. 


 

[meritez]

Hi Jason,

Bogof's sixteen page thread may be a good place to start:
https://forum.kitz.co.uk/index.php/topic,27091.0.html

I will confirm I'm currently using a Gamma BT Wholesale 1000/115 FTTP connection, and do not see the same problems you are facing.

[Jasonkruys]

Hi Jason,

Bogof's sixteen page thread may be a good place to start:
https://forum.kitz.co.uk/index.php/topic,27091.0.html

I will confirm I'm currently using a Gamma BT Wholesale 1000/115 FTTP connection, and do not see the same problems you are facing.

Good grief. Just looked and Zen performed a GEA migration on 24th May, can't be a co-incidence. I've just pointed that out to Zen with a quick email.

[craigski]

If you check your stats/graphs/BQM etc, does it correlate with 24/5?

I would expect your would see a change in latency on BQM at the point it was migrated.

[Jasonkruys]

Yes, all issues started at precisely 0400 on the 24th.

From that point the BQM graphs are showing packetloss that matches exactly with periods of slow or No download on my 15 minute testing - most obviously and notably during the 2 hour periods of <500kb/s. Before the initial spike in packetloss at 0400 on the 24th all was clean. Ping, jitter also all over the place both on the BQM plot and my own internal 'outogoing' tests since that point, and as a I've, Openreach have tried literally everything else (as above), it is difficult to ignore the one thing that has changed. I only remembered I had BQM set up yesterday, so hadn't investigated that until now, but every single data point I have across BQM, Other tests, and service seen in real life points back to 0400 on the 24th, and when I looked today, that is when a GEA migration had been fulfilled - it wasn't even on my radar.

Heard back from Zen, the fault manager I have been dealing with is going to try and get a migration back put in, despite the fact they think the new link is bigger, better, faster, more capacity, less congestion and testing OK.

I should add, Zen have been great so far in trying to resolve, so I've got my fingers crossed.

[craigski]

it wasn't even on my radar.

A good job the kitz forum was on your radar to help find that missing piece of the puzzle 

I should add, Zen have been great so far in trying to resolve, so I've got my fingers crossed.

From what I have read above, and it does turn out to be a Zen backhaul issue, its seems Zen have not been great, on the basis of how much time and effort yourself and OR have spent on finding a potential known issue with Zen network?

[Jasonkruys]

A good job the kitz forum was on your radar to help find that missing piece of the puzzle 
From what I have read above, and it does turn out to be a Zen backhaul issue, its seems Zen have not been great, on the basis of how much time and effort yourself and OR have spent on finding a potential known issue with Zen network?

Fair point on the Zen comment. My 'praise' was largely for the contact and keeping me updated. It is disappointing that this is clearly an issue seen before, but not once has been considered. When 'investigating', they must have noted the migration date co-incided with the issue starting.

Latest contact is they want to run more tests over the weekend - so after two weeks of their network team on the case, and Openreach apparently doing everything they possible can to resolve and basically saying 'not us', they want to leave me with a broken connection for at least another weekend. I have to say, my hope of resolving this is dwindling 

I'm going to give it the weekend, but I've expressed my disappointment and asked them to advise how I can get authorisation to transfer out early.

So, next question is - who should I move to? Either if they don't resolve, or possibly when my contract is up? I really hope Zen resolve it, but my confidence has been severely knocked today - especially as this clearly an issue that they know about.

[Alex Atkin UK]

Its especially weird that upload is unaffected given it would need faster downstream than that to achieve the upstream.  Very unsettling to see Zen still having these major GEA issues.

[Jasonkruys]

Its especially weird that upload is unaffected given it would need faster downstream than that to achieve the upstream.  Very unsettling to see Zen still having these major GEA issues.

Indeed on both counts.

Are you on a Zen GEA?

It's a shame, because if they actually acknowledged the issue and put some effort into looking at the data - those on Zen GEA not affected, versus those that are, they might actually spot a trend that sets them on the path to solving it. Equally, customers with a bit of knowledge and the ability to provide objective measured data rather than 'it doesn't work' could help, but if they are guarded and make out it's not an issue and deny it, 1. They annoy good customers like myself (well I think I'm good anyway) and.2. they don't ever solve the problem.

As said, I'll see what they come back with. But my long and happy Zen custom is quickly turning sour. They're not even cheap - I picked Zen due to the lauded customer service, and ability to speak to a support team that weren't reading scripts from a different time zone.

I am still, perhaps naively, hoping they come through.

[Alex Atkin UK]

No GEA here as Zen are not at my exchange, but that obviously could change at any time.

[bogof]

My considered opinion was that Zen are a complete bunch of clowns or crooks with these FTTP GEA issues from my interactions with them.  They had a great opportunity to try and get to the bottom of the issue with myself and other capable individuals at the end of the phone, yet they squander it and just annoy us to the point of leaving.  My saga went on over the course of 4 months or so.   You'd imagine this is only going to get worse with more takeup of FTTP, and yet no-one it seemed cared one jot really. 

You could request they migrate you back to BT backhaul, which they don't like but can do.  But note, even that doesn't seem guaranteed, as less than a month after migrating me back AND FIXING MY ISSUES they re-migrated me back to Zen backhaul with the issues instantly returning, apparently forgetting to set the flag to prevent migration.  To add insult to injury, each time they migrated me in one direction or the other my authentication would break, too, and I'd have to call them to get them to put some more hay out for the hamsters...

I normally like to put such things down to incompetence rather than malice; but I'm not sure that they don't actually know what's going on, and instead they're just stringing folk along, particularly as I had one day on their network where I connected to some different hiterto unseen gateway which wasn't fully set up, and had no reverse DNS, that never appeared again and worked perfectly (perhaps better than Openreach).  They denied all knowledge of it.  Very smelly.

Good luck.  My advice, run, don't walk, if you have the chance, unless you like having your time wasted.  They're great at that.

[bogof]

Its especially weird that upload is unaffected given it would need faster downstream than that to achieve the upstream.  Very unsettling to see Zen still having these major GEA issues.

My upstream never seemed at all affected, but my downstream wasn't impacted anything like to the extreme the poor OP is facing, really tragic.

[Jasonkruys]

My considered opinion was that Zen are a complete bunch of clowns or crooks with these FTTP GEA issues from my interactions with them.  They had a great opportunity to try and get to the bottom of the issue with myself and other capable individuals at the end of the phone, yet they squander it and just annoy us to the point of leaving.  My saga went on over the course of 4 months or so.   You'd imagine this is only going to get worse with more takeup of FTTP, and yet no-one it seemed cared one jot really. 

You could request they migrate you back to BT backhaul, which they don't like but can do.  But note, even that doesn't seem guaranteed, as less than a month after migrating me back AND FIXING MY ISSUES they re-migrated me back to Zen backhaul with the issues instantly returning, apparently forgetting to set the flag to prevent migration.  To add insult to injury, each time they migrated me in one direction or the other my authentication would break, too, and I'd have to call them to get them to put some more hay out for the hamsters...

I normally like to put such things down to incompetence rather than malice; but I'm not sure that they don't actually know what's going on, and instead they're just stringing folk along, particularly as I had one day on their network where I connected to some different hiterto unseen gateway which wasn't fully set up, and had no reverse DNS, that never appeared again and worked perfectly (perhaps better than Openreach).  They denied all knowledge of it.  Very smelly.

Good luck.  My advice, run, don't walk, if you have the chance, unless you like having your time wasted.  They're great at that.

Thanks, I am quickly coming to the same opinion, but lets see.

Who did you move to? My requirements are a static ipv4 address, ipv6 enabled, and a call centre/technical team/helpdesk I can have a reasonable conversation with that's not in a different time zone.

I was with Aquiss years ago on ADSL, looks to be an option? I assume they use BT/OR Wholesale links and don't piggy back off Zen...that would be just my luck.

I was tempted to look for a provider that doesn't use PPPoE (not Sky - their help desk is horrendous and who I moved from), but that just leaves talk talk? What is their customer service like?

[bogof]

I moved to AAISP.  Service is great; reassuringly expensive tho, and the quota setup is a bit annoying for me; I only really need 2TB / month so I go on 10 TB for a month and then onto 1TB for 2 months to save some pennies.  Would be nice if you could just pay pro-rata between the 1 and 10TB option (or similar). 

I also considered Cuckoo on Talk Talk Business network.  1 month exit clause.  Still PPPoE though I understand (only retail TT is DHCP I think).  In a way I wish I'd given them a go. 

Best of luck.  It would be nice if they just got this fixed once and for all...

[Jasonkruys]

Thanks, I keep looking at AAISP but I worry about the cost/quota. Need to crunch some numbers. Cuckoo looks interesting - maybe I'll give that a go if I can get.out of contract if Zen won't resolve the issue.

I'm just so angry now that they've had me, Openreach and even members of their own team running about for two weeks for a known issue that 1. They didn't even consider in the first place, and 2. Seem to be refusing to accept even now every single data point proves it.