Kitz Forum

Broadband Related => ADSL Issues => Topic started by: siofjofj on September 15, 2020, 04:09:40 PM

Title: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on September 15, 2020, 04:09:40 PM
Hi all,

This topic is related to the same fault that we were discussing here https://forum.kitz.co.uk/index.php/topic,21455.0.html (https://forum.kitz.co.uk/index.php/topic,21455.0.html), however a significant amount of time has elapsed and I now believe that the initial assumption I made in that thread (MSAN ports going bad) is incorrect, so I thought a new thread would be worthwhile for clarity. Sorry for the very long post, but this story is somewhat complicated.

In a nutshell, the problem I am having is repeated loss of ADSL sync, occurring on an intermittent basis. We can go for a few days with an absolutely solid connection and not a single CRC error, however this can then change to a completely unusable connection for several hours with the modem staying in sync for only seconds before re-syncing over and over again. On average, I'd say the connection is unusable for about 10% of each day, and extremely frustrating outside of that with random disconnections occurring.

When the connection is playing up, the symptoms are as follows, as tested with a known good modem and corded telephone connected to the test socket using a known good microfilter:


This problem has now been going on for roughly 2.5 years and, with me now having to work from home which involves numerous video calls, is becoming unbearable. In this time we have had visits from roughly 10 Openreach engineers, none of whom have managed to fix it. The results of the tests performed by the Openreach engineers on every visit are as follows:

As a result of this, a large number of Openreach visits end up being 'Right When Tested' so nothing gets done. Sometimes, the engineer performs a 'DSL Close-Out Test', using their tester as a modem and monitoring for CRCs and loss of sync. If the line is having a good period at that point, they see nothing, so again we have 'Right When Tested'. If the line is acting up at that point, they see the errors, particularly when the telephone is picked up or if I sneakily ring the phone from my mobile during the test.

If the errors were seen during the test, on every case they have gone away and done something, usually swapping between pairs at various different junction boxes until no more errors are seen during the DSL close out test. On a few occasions, the port on TalkTalk's MSAN was changed (hence the other thread). When they come back to the house, the DSL Close-Out Test passes with no errors and they leave. Without fail the fault then reoccurs some time (even minutes later in same cases) after they have left.

Lately, we have changed ISP to NowTV. Partially this was because of the suspicion of a bad line card on TalkTalk's MSAN (see other thread) so, since Sky have their own LLU equipment, we thought this would fix the problem. It didn't, the problem is just as bad as before.

Two weeks ago, we had an Openreach engineer come out and he saw the huge number of CRCs on his tester. He then went to the joint box in the pavement on the opposite side of the road to the house and did the same test there, finding no errors. He then swapped us to the green/black pair in the cable from the joint box to the house and tested again at the house. This time, he got even more CRCs than when testing at the house the first time. He therefore put us back on the orange/white pair and put in a job for the replacement of the 'lead-in' (the cable between the joint box and our house). The house is 1970s build with an armoured telephone cable buried directly in the ground from the joint box on the other side of the road, up the street a bit.This means this job is very difficult and expensive as it requires digging up our front garden, the pavement outside the house down the street and then the road to get across to the joint box on the other side.

Last week, a civils team together with a PSTN engineer came along. Apparently the job had been changed by management from 'change the lead-in' to 'find the fault on the cable run, then dig there and repair it'. Of course, no-one knows where on the lead-in the fault was, so they guessed at under the pavement outside my neighbour's house as they thought it might be the buried joint there. They dug a hole in the pavement, but found no telephone cables in the hole so filled it in again and left.

A few days later, we had another broadband engineer visit who put in for replacement of the lead-in again (though he was unable to reproduce the fault as the line was fine at that time, he did believe the previous engineer who said that there is one). Yesterday, another civils team turned up. They dug up the front lawn and put in ducting for a new lead-in between the house and pavement. So far so good. However, they ran out of time so had to come back today to dig up the pavement and road to get to the joint box. This morning they arrived with a broadband engineer who tested the line and couldn't find the fault (line having one of its good periods), so he told them to cancel the lead-in change, leaving the newly installed duct in the front lawn empty. Instead, they dug in front of next door's house again and the engineer spliced my lead-in onto the spare pair on next door's lead in (this effectively replaces only 5m of the 30m lead in). As soon as the engineer left, but while the civils team were still there, the disconnections restarted. I showed one member of the civils team the disconnections and the noise on the phone, and he agreed that it is still broken, so he tried to call the engineer that just left. We were told to simply report the fault again via our CP.

This is a complete nightmare, since going back to reporting the fault via NowTV leaves us with a 50/50 chance they will find nothing again, so we will have to keep trying appointment after appointment until an engineer happens to come when the problem is apparent. Then the issue is that if the engineer can't fix the problem that day, the next engineer does the tests again and cancels the job if the fault can't be found the next day.

My questions to the good people at kitz are as follows:
1) Does anyone have any ideas about a better way to approach Openreach or NowTV about this, because right now I am getting nowhere.
2) Does anyone have any other ideas about what the actual problem with the line might be? It may be that the fault simply disappeared (in the way it does) when the engineer tested from the joint box, so perhaps the lead-in is not the problem after all.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: burakkucat on September 15, 2020, 05:05:37 PM
All evidence that I have examined points to one (or more) joint(s) showing HR or semi-conductive tendencies. As for the location of the defect, I cannot say . . .  :no:

Attaching a TDR to the end of the lead-in at your home and making a series of measurements should give a good preliminary location. Going to the PCP and making a second series of TDR measurements, then cross-referencing both sets of results should narrow down the target area.

Next perform a sequence of classic resistive fault location (RFL) tests. Firstly from your end of the lead-in and then at the PCP.

Cross-referencing both sets TDR and RFL data will pin-point the faulty plant.

What can you do? My suggestion is that you compose a concise e-mail message, enclosing all details of the relevant facts, dates & times, and send it to clive.selley@openreach.co.uk, requesting a higher level intervention.

[Edited to correct a typo.]
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on September 15, 2020, 05:28:34 PM
Thanks burakkucat. I saw the suggestion of emailing Clive Selley in another thread, so was indeed wondering if any kitizens thought my fault was 'worthy' of such an intervention.

Do you think it would be worth including line stats with such an email? I have attempted to show some of the visiting engineers the stats in case the information would be useful to them, but none ever seemed interested.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: burakkucat on September 15, 2020, 06:58:23 PM
I saw the suggestion of emailing Clive Selley in another thread, so was indeed wondering if any kitizens thought my fault was 'worthy' of such an intervention.

It certainly is a worthy case.

Quote
Do you think it would be worth including line stats with such an email?

Just mention that the information is available and can be supplied, if necessary.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: Black Sheep on September 15, 2020, 07:24:16 PM
I totally concur with B*Cat .... this is the 'High Level Complaints' team, bread & butter type jobs.

An absolutely certain high resistance fault (going from your description) ... and shouldn't be 'that' hard to find, TBH ?. Luckily, the HLC team will generally ensure the right man for the task attends site.

Go for it.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on September 15, 2020, 08:32:49 PM
Thanks guys. I'll get on it.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: burakkucat on September 15, 2020, 10:14:35 PM
. . . the right man for the task attends site.

Preferably with a Tester 301C and an Ohmmeter 18C.  :D  ;)
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: Black Sheep on September 16, 2020, 10:58:39 AM
Preferably with a Tester 301C and an Ohmmeter 18C.  :D  ;)

Ha ha ..... proper meters !!  ;) ;D
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on September 20, 2020, 05:23:40 PM
Well, that looks to have been very successful!

On Friday, a few days after sending the email, the Openreach 'patch lead' (i.e. the leader for the area) turned up at the house. She had read all the notes from the previous engineer visits and asked us for our description of the problem. She was very apologetic for how difficult it had been to get this fixed, and said that one of their best engineers would be sent to the job tomorrow (Saturday!) and would be able to stay on it until we were happy it was fixed (or alternatively until a plan for fixing it could be established if it was not possible to fix it that day).

On Saturday, sure enough the engineer arrives. We discuss the fault and what has already been done by other engineers, and off he goes to investigate. He ultimately concluded, after spending all morning looking, that the fault was not on our lead-in but somewhere else in the network. He found a spare stopped line in the joint box opposite my house, so swapped us onto that. The connection has been solid for 36 hours now and the phone is completely silent. I'm currently using NowTV's supplied router so can't see if we are getting CRCs when the phone rings (hardly any stats visible on this), but I will swap back to my own when NowTV are happy it is stable so I'll be able to check this later. The engineer even said he would return on Sunday to check I was happy, but I texted him this morning to say he didn't need to come since all was well. He said I can contact him directly if the issue reoccurs.

So, thank you for the advice burakkucat and Black Sheep, it worked out really well!

I feel very sorry for the engineer that diagnosed the lead-in and caused (apparently) £3000 worth of digging to be done. It looks like he got caught out by the fault disappearing (as it does) when he got to the joint box and then reappearing when he got back to the house.

I also feel sorry for the civils team who, while digging up the front garden made a shocking (thankfully not literally) discovery. During the dig, they found a duct with 'G.P.O.' written on it. This was a surprise because, as seen at the joint box and at the house, the lead-in appeared to be directly buried in the ground. However it was (seemingly) good news since this would mean the entire route would not need digging up. They therefore broke into the duct to allow pulling in a new cable, only to find a thick cable marked 'POWER' inside it. The house builders seem to have used G.P.O. branded ducting for the electricity supply which, in my opinion and the opinion of the civils guys, is utterly ridiculous! This duct was then swiftly repaired.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: burakkucat on September 20, 2020, 05:48:17 PM
A good result. Thank you for sharing the news.  :)

Just in case you are not aware of its existence, there is a "Thank an Engineer (https://www.formwize.openreach.co.uk/run/survey3.cfm?idx=505d040e0a0e0c)" page that you now might like to use?
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on September 20, 2020, 05:56:36 PM
Indeed I would. Thanks for the reminder!
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: Black Sheep on September 21, 2020, 09:06:05 AM
To reiterate what B*Cat has said, thank you for the feedback.  :)
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on October 04, 2020, 08:51:41 PM
I thought I'd just finalize this thread by posting my line stats now that the fault has been fixed and the connection has been rock solid for two weeks. The DLM has now relented, and I'd say I have a very good ADSL2+ connection. Not a single CRC in the two days since I swapped back to my own router!

Stats summary
Code: [Select]
Stats recorded 04 Oct 2020 20:46:34

DSLAM type / SW version: BDCM:0xa474 (164.116) / v0xa474
Modem/router firmware:  AnnexA version - A2pv6F039v.d26h
DSL mode:                ADSL2+
Status:                  Showtime
Uptime:                  2 days 0 hour 30 min 3 sec
Resyncs:                0 (since 02 Oct 2020 20:14:32)

Downstream Upstream
Line attenuation (dB):  20.0 12.3
Signal attenuation (dB): 22.4 11.8
Connection speed (kbps): 21495 1117
SNR margin (dB):        3.1 9.1
Power (dBm):            18.6 12.1
Interleave depth:        1 16
INP:                    29.00 2.00
G.INP:                  Enabled Not enabled
Vectoring status:        Unknown

RSCorr/RS (%):          0.0001 0.0331
RSUnCorr/RS (%):        0.0000 0.0000
ES/hour:                0 0

Connection stats
Code: [Select]
xdslctl info --stats
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    8000
Last initialization procedure status:   0
Max:    Upstream rate = 1200 Kbps, Downstream rate = 22169 Kbps
Bearer: 0, Upstream rate = 1117 Kbps, Downstream rate = 21495 Kbps
Bearer: 1, Upstream rate = 0 Kbps, Downstream rate = 0 Kbps
Link Power State:       L0
Mode:                   ADSL2+ Annex A
TPS-TC:                 ATM Mode(0x0)
Trellis:                U:ON /D:ON
Line Status:            No Defect
Training Status:        Showtime
                Down            Up
SNR (dB):        3.1             9.1
Attn(dB):        20.0            12.3
Pwr(dBm):        18.6            12.1

                        ADSL2 framing
                        Bearer 0
MSGc:           -6              28
B:              245             62
M:              1               1
T:              0               1
R:              6               10
S:              0.0000          1.9211
L:              5110            304
D:              1               16
Q:              2               0
V:              0               0
RxQueue:                38              0
TxQueue:                19              0
G.INP Framing:          18              0
G.INP lookback:         19              0
RRC bits:               0               0
                        Bearer 1
MSGc:           28              -6
B:              0               0
M:              16              0
T:              1               0
R:              16              0
S:              32.0000         0.0000
L:              8               0
D:              1               0
Q:              0               0
V:              0               0
RxQueue:                0               0
TxQueue:                0               0
G.INP Framing:          0               0
G.INP lookback:         0               0
RRC bits:               0               0

                        Counters
                        Bearer 0
SF:             0               421900
SFErr:          0               0
RS:             1770896654              2866969
RSCorr:         1000            909
RSUnCorr:       0               0
                        Bearer 1
SF:             10274339                0
SFErr:          1               0
RS:             21832968                0
RSCorr:         70              0
RSUnCorr:       1               0

                        Retransmit Counters
rtx_tx:         143656797               0
rtx_c:          1471            0
rtx_uc:         0               0

                        G.INP Counters
LEFTRS:         0               0
minEFTR:        21485           0
errFreeBits:    57111488                0

                        Bearer 0
HEC:            0               0
OCD:            0               0
LCD:            0               0
Total Cells:    264358181               460165178
Data Cells:     193286003               11292004
Drop Cells:     0
Bit Errors:     0               0

                        Bearer 1
HEC:            0               0
OCD:            0               0
LCD:            0               0
Total Cells:    0               0
Data Cells:     0               0
Drop Cells:     0
Bit Errors:     0               0

ES:             0               0
SES:            0               0
UAS:            379             379
AS:             174663

                        Bearer 0
INP:            29.00           2.00
INPRein:        0.00            0.00
delay:          0               8
PER:            0.00            16.32
OR:             0.01            16.65
AgR:            19875.69        1045.34

                        Bearer 1
INP:            8.00            0.00
INPRein:        8.00            0.00
delay:          8               0
PER:            17.00           0.01
OR:             16.00           0.01
AgR:            15.93   0.01

Bitswap:        41838/41838             1/1

Total time = 2 days 37 min 22 sec
FEC:            1000            909
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            379             379
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           0
Latest 15 minutes time = 7 min 22 sec
FEC:            5               0
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           0
Previous 15 minutes time = 15 min 0 sec
FEC:            2               0
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           N/A
Latest 1 day time = 37 min 22 sec
FEC:            7               3
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           0
Previous 1 day time = 24 hours 0 sec
FEC:            302             612
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           0
Since Link time = 2 days 31 min 1 sec
FEC:            1000            909
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Retr:           0
NTR: mipsCntAtNtr=0 ncoCntAtNtr=0

Thanks once again for the help.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: burakkucat on October 04, 2020, 08:58:48 PM
I thought I'd just finalize this thread by posting my line stats now that the fault has been fixed and the connection has been rock solid for two weeks. The DLM has now relented, and I'd say I have a very good ADSL2+ connection. Not a single CRC in the two days since I swapped back to my own router!

. . .

Thanks once again for the help.

You're welcome.  :)
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on November 20, 2020, 07:55:32 AM
Arrgh! After just under 2 months of having a nice and reliable internet connection, following the line being swapped to a completely different pair all the way back to the exchange, since 14:00 yesterday it has become almost unusable again.

Same story as last time, frequent resyncs occurring randomly and when the phone is picked up, high pitched noise and some crackling audible on the phone and the same characteristic SNRM and CRC traces.

Any ideas about what might be going on here? The line has been entirely changed to a different pair (with the exception of the DIG lead-in to the house), and this change definitely helped as it was absolutely solid for 2 months (just checked my recorded stats and the SNRM was arrow straight for all that time with only 1 or 2 CRCs per day), yet now it has gone wrong again in exactly the same way. I'd suspect intermittent REIN were it not for the fact that picking up the phone kills the broadband, which presumably cannot be explained by REIN?
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: burakkucat on November 20, 2020, 04:57:31 PM
. . . since 14:00 yesterday it has become almost unusable again.

Same story as last time, frequent resyncs occurring randomly and when the phone is picked up, high pitched noise and some crackling audible on the phone and the same characteristic SNRM and CRC traces.

<snip>

I'd suspect intermittent REIN were it not for the fact that picking up the phone kills the broadband, which presumably cannot be explained by REIN?

Unfortunately you are experiencing a similar problem as before . . . There is (at least) one joint showing HR or semi-conductive tendencies.  :(
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on November 20, 2020, 05:12:40 PM
Unfortunately you are experiencing a similar problem as before . . . There is (at least) one joint showing HR or semi-conductive tendencies.  :(
Yes, I thought as much. Seems rather unlucky that it's happened again a relatively short amount of time after the pair was swapped. Interestingly, several of the neighbours have commented about various internet and noisy phone issues that have been very difficult to get resolved. I wonder if a section of the big multicore cable (or a joint in it) feeding the estate is bad (full of water maybe?), as a result a huge number of the pairs within it have (or liable to get) issues.

I'd kept the mobile number of the Openreach patch lead who managed the issue last time, so texted her this morning explaining the situation. Very kindly, she has arranged for an engineer to come out on Monday and have a look again without me having to go through my ISP again. Hopefully it will get sorted then  :fingers:
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: Alex Atkin UK on November 20, 2020, 07:46:12 PM
Certainly possible, I know many years ago they completely re-crimped the multicore to my local cabinet (although it was the exchange side in that case) because of water damage.  To make matter worse it was aluminium cable and my line snapped while they were doing it due to corrosion, which fortunately I noticed immediately and informed them before they left.  I watched them re-crimp it again with no slack at that point, that e side if it snaps again will basically become unusable.

But this is just as likely to happen from the cabinet to the house if its an old joint that has taken on water, especially underground.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: Weaver on November 21, 2020, 01:04:52 PM
the picking up the phone thing suggests a microfilter or SSFP gone bad, or a bad design.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on November 21, 2020, 02:03:56 PM
the picking up the phone thing suggests a microfilter or SSFP gone bad, or a bad design.
Yes, that was one of the things I suspected the first time this issue occurred. However, since then, I've tried the Mk2, Mk3 and (after Openreach changed the NTE5A to an NTE5C) Mk4 SSFPs as well as the Excelsus Z-350UK plug-in microfilter and whatever the plug-in filter Sky and TalkTalk supply with their routers is. I also had a go with double filtering (specifically the Mk3 SSFP with the Z-350UK plugged into its filtered output and the phone plugged into that), none of which made any difference. I also get the crackling with no microfilter at all and just the phone connected to the test socket, and the random disconnections and SNRM oscillations with no microfilter and just the modem connected to the test socket.

That said, you've just reminded me that I should probably check on the current situation with a different modem, phone and microfilter before Openreach come, just in case it is one of those this time round, so thanks :)

Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: Weaver on November 21, 2020, 09:27:59 PM
Double micro filtering (with an ADSL Nation superb one) was necessary for me with one evil DECT phone that I had. You were right to try that. Things are definitely wrong.
Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: neil on November 28, 2020, 07:59:36 PM
I also had same issue couple of times 2-3 times in last two months
SNR dropping for upload only and syncing at 600kbps at night and internet working fine during daytime
internet is working fine now
but phone is noisy 

lineman suggested to change my pairs to newly installed cable on the poles instead of using underground cable.
But so far internet is working fine so, didn't bother

Title: Re: Help With Hard to Find Fault - Possible High Resistance
Post by: siofjofj on November 28, 2020, 08:16:47 PM
Similar story for me this time round. Everything mysteriously cleared up on Sunday morning and was absolutely fine for 24 hours so I messaged my contact at Openreach to say that it's now behaving itself again and that there probably won't be anything to find, however they were welcome to still attend if they thought it would be worthwhile. They decided not to, and there have been no problems for the rest of the week. Hopefully it won't come back again.