Kitz Forum
Broadband Related => ADSL Issues => Topic started by: Weaver on March 10, 2022, 02:10:32 AM
-
During Tuesday night my line 2 was very unstable, with several loss-of-sync events, loss of downstream sync speed and hollow curve disease phenomenon seen in modem #2. This modem was swapped out on Tuesday evening for the modem that had been in use on line 3. This however did not help with the hollow curve disease or loss of sync rate. One theory is that potential four modems were in some way damaged in January/February 2020 in a lightning surge event and that this brought about the long-term period of HCD (hollow curve disease) during the whole of the two years to date. Recently modem cold restarts after prolonged off time have been a medium term cure for HCD, with a return to full performance for an unknown length of time.
Since a modem swapout hasn’t helped, the current thinking is that I used a second bad modem, the line 3 modem, one of the four damaged two years ago. So the next thing to do is to pull a completely by-me unused ‘brand new’ modem (but maybe not literally, since very many in stock are second-hand) from stock.
I informed the ISP Andrews and Arnold (AA) and support’s David is looking into it, with repeated tests to be run overnight.
This is the latest SNR-vs-tones graph from Thursday early morning :
(https://i.postimg.cc/jd2M3Jzj/205-B665-D-C0-E6-4531-AEA2-0-E009857-C707.png)
Here is the graph of SNRM-vs-time over night last night:
(https://i.postimg.cc/gjMbFXcK/194-DF817-0-E11-46-A4-A9-F0-94578-A60-F4-D0.png)
-
Yuck! :yuck: I think this is the very first time you have shown the SNRM v time plot corresponding to an HCD event.
In all cases where I have witnessed a bad SNRM v time plot, similar to that you have shown, the fault has been a bad joint. HR or semi-conductive. The cure has always been to identify the joint and re-make it.
I really wish that there was some way to apply 95V AC ringing current on your three lines. In their normal state they just have the constant exchange battery DC voltage present. I wonder if that constant DC whetting voltage without the occasional overlay of the AC ringing current actually accelerates joint faults?
I remember reading about how, in the early years of the 20th century, before there was the National Grid, some local electricity supply companies actually provided a DC mains supply. One artefact of a DC mains supply was that light bulbs would fail far sooner than the equivalent bulbs used with an AC mains supply. One trick to extend the working life of the bulbs with a DC supply was to turn the bulbs around in their sockets once a week (or was is once a month). Hence, by analogy, my suspicion is that the occasional application of ringing current to each of your lines may minimise the bad-joint effects. Something for A&A to consider, perhaps? :-\
-
I sent the SNRM-vs-time plot to AA.
-
I have now swapped out two modems for brand new ones. Were boxed, never used and the hollow curve disease is still around in line 4; can’t tell in line 2 as it won’t even sync, it’s so ill.
Here’s the first stage of line #4’s notch development:
(https://i.postimg.cc/mkLvTHXM/44-A92-EA1-E448-4817-A8-B0-044-C12-A889-D3.png)
And here’s the second stage:
(https://i.postimg.cc/R0PswDX3/94576839-00-E1-4-DD1-BEFB-85-B1-B43-D4-B13.png)
-
This morning, Sat 2022-03-19, after an engineers visit to sort out line #2, we now have the following picture of beauty restored:
(https://i.postimg.cc/xd5L567H/A48-D97-B7-06-C6-4-E15-A66-E-A3-C8-C0-BB3-C16.png)
-
Much better.
Does A&A have an opinion on my thoughts that the lack of AC ringing current being applied to the lines may be behind these regular "defective joint" problems? With just the -52 to -48 VDC being present, I wonder if "iffy joints" are slowly being conditioned into semi-conductive states.
-
Could we write something together that AA will understand?
-
Paragraph three, Reply #1 and paragraph two, Reply #5. I've probably typed similar in some of your other threads on the "HCD" phenomenon.
Perhaps you would like to merge them all together and then edit the result to use the sort of words that A&A will understand. ;)
-
Have written to AA explaining Burakkucat’s good theory.
Update on line 4. Down to 1kbps downstream at 6.6 dB (should be ~3Mbps).
(https://i.postimg.cc/dQBv1YNt/4724-A778-C890-4053-81-D4-5-E9-CF2281-D29.png)
(https://i.postimg.cc/SskH9ggb/00-CEF134-0875-401-D-9-C2-C-624-C17-A3-A83-D.png)
(https://i.postimg.cc/wjhpsR9J/F4-ACC3-C6-680-B-4376-9-A17-F25-D4-C28-DC9-A.png)
Notice the packet loss (‘dripping blood’ - red from the top down) building after 13:00:
(https://i.postimg.cc/gjDMWdpx/135139-A9-E2-BE-4-DC9-8-AB0-889409903-CB9.jpg)
-
Update on line 4. Down to 1kbps downstream at 6.6 dB (should be ~3Mbps).
Yuck. :yuck: :thumbdown:
The SNRM plot looks like just what has been seen with a HR (and/or semi-conductive) joint. A good "blast" of AC (~95V) would probably clear it.
A static -50V DC plus 95V AC would result in a potential swing from -145V to +45V at the iffy joint.
-
@burakkucat
One thing I've never been sure about is whether, when ringing a line to clear dodgy joints, it needs or is more effective if there is an actual handset on the end of the line?
-
Do we know if it is possible to ‘ring’ a line with no active voice service if you are an ordinary human or maybe you have to be a BT engineer with a magic wand? How might engineers be clearing faults? They always do - for a long time anyway.
Now this is the first time that we have definitively proven that it’s not the modems, because we have new modems that have never been used before and so we’re not exposed to the surge event of jan/feb 2020. So that’s good to know. But what puzzles me is that at times turning the modem off for a good while seems to help. - why? That made me think about guilt of modems.
Because of the dripping blood, I have now turned off modem 4 and it’s staying off unless AA need it. I sent them an extremely polite nudge today not to forget about me. AA must be so fed up with this going on and on. Do you think they’re getting charged by OR? I assume AA is supposed to moan at BTW to make it all good. Not entirely sure about the sequence of actions.
Do we think that an OR engineer should be able to find out where the HR or semi-conductive joint is?
-
@burakkucat
One thing I've never been sure about is whether, when ringing a line to clear dodgy joints, it needs or is more effective if there is an actual handset on the end of the line?
Ideally if the zap has been organised from a remote location (relative to the end-user), say the serving telephony exchange, then the best effect would be obtained with the pair looped (i.e. a hard short-circuit) at the subscriber's end. Without having the pair looped, the standard series connected capacitive-resistive shunt across the pair, provided by all master sockets (be it an LJU or an NTE5), will provide an attenuated AC path.
If the end-user owned an ex-GPO magneto connected to a standard BT plug, just plug it into the test-socket and crank the handle. That would (probably) temporarily clear the fault.
-
(FYI I think I was still adding to that last post, we may have crossed over slightly? )
I was asking about whether or not OR should be able to find the fault.
@Black_Sheep If you’re about, my friend, I’d love to hear your thoughts.
-
Do we know if it is possible to ‘ring’ a line with no active voice service . . .
I have tried that for you on three (or maybe four) occasions. The most recent occurrence being just moments ago --
Drop cable 1, Pair 1, 01471 nnnn70 engaged tone returned
Drop cable 1, Pair 2, 01471 nnnn14 (abandoned 2021-04) number unobtainable tone returned
Drop cable 2, Pair 1, 01471 nnnn44 engaged tone returned
Drop cable 2, Pair 2, 01471 nnnn75 engaged tone returned
. . . maybe you have to be a BT engineer with a magic wand? How might engineers be clearing faults? They always do - for a long time anyway.
Via the TAM, maybe? Black Sheep might be able to assist our search for knowledge.
Now this is the first time that we have definitively proven that it’s not the modems, because we have new modems that have never been used before and so we’re not exposed to the surge event of jan/feb 2020. So that’s good to know. But what puzzles me is that at times turning the modem off for a good while seems to help. - why? That made me think about guilt of modems.
Agreed. Twice over.
Because of the dripping blood, I have now turned off modem 4 and it’s staying off unless AA need it. I sent them an extremely polite nudge today not to forget about me. AA must be so fed up with this going on and on. Do you think they’re getting charged by OR? I assume AA is supposed to moan at BTW to make it all good. Not entirely sure about the sequence of actions.
Your guess is as good as mine, if not better!
Do we think that an OR engineer should be able to find out where the HR or semi-conductive joint is?
With a lot of effort, yes. I'll guess it is somewhere out on the high moor, in a joint-closure that is half-buried in mud or peat at the side of a water-course.
-
You forgot to mention the sheep droppings scattered all over the top of the enclosure! :P
-
> I have tried that for you on three (or maybe four) occasions. The most recent occurrence being just moments ago --
:-[ ??? I remember now. Stupid of me. Do we think the effect of the call operation gets as far as generating a ring signal though ?
@Jelv :lol:
Can our OR engineer do distance measurement (eg like reflectometry) to find faults of this type? Because otherwise it’s not much fun at all examining every joint in turn - or something ?
-
... Do we think the effect of the call operation gets as far as generating a ring signal though ? ...
If an engaged tone was being received then the answer is no.
-
Can our OR engineer do distance measurement (eg like reflectometry) to find faults of this type?
If s/he has a time-domain reflectometer (that does not go into "nanny knows best mode" when voltage in excess of the normal, quiescent line, voltage (-48V to -52V) is present) and is able to apply ringing current to the line (from the TAM, possibly) then, when looking from your end towards the exchange "battery", any artefact seen in the trace that "jumps up and down" in synchronism with the standard ringing cadences is likely to be due to a defective joint.
-
Clueless.aa.net.uk CQM picture of line 4 just now :
(https://i.postimg.cc/Z5DKD1fN/E01-CBA9-F-3-B75-4-B3-C-8208-1-B1366502-A84.jpg)
-
Isn't there supposed to be a warning before horror shows are broadcast?
-
Hi Weaver
Thanks for letting me know about this topic.
From what you said in the other topic, you're absolutely right about Openreach never permanently applying a fix. It's always a temporary one or one that enables the line to continue operating 'within the parameters'. I understand that one of the parameters may be that the user experiences up to three disconnections per day, but what that these disconnections occur in the most annoying moments (rather than in the middle of the night) Openreach can't be challenged upon.
-
@Jelv ;D ;D
-
Openreach did apply a permanent fix today though. Fixed line 4 for good. Water ingress into a joint up on the moor somewhere. Cured my very serious hollow curve disease.
-
Openreach did apply a permanent fix today though. Fixed line 4 for good.
That's good to know. Let's hope it is for good . . . For good until another defect occurs, of course. :)
Water ingress into a joint up on the moor somewhere.
That doesn't surprise me. (The haggis' teeth can gnaw through anything, if they are given enough time.)
-
BT Fault 5-884480236198 Customers Equipment, Error or Misoperation;Fault Not Found Cleared Whilst Localising
What did the end customer describe as the primary issue ?: 3762.03-Customer Report: End customer informed us that broadband is not stable / intermittent.
Where did you perform the initial PQT ?: 3743.01-\n\n Initial test results: PQT test performed at NTE back plate.
What was the result ?: 3763.02-The test passed with amber parameters on 2022-03-28T15:17:21.
Have you identified any other issue(s) in the OR network ?: 6357.08-\n\n What was found: Other Openreach network fault.
Where was the fault located ?: 743.02-\n\n Actions to resolve: Fault located in the D side underground network.
What did you do to fix it ?: 745.42-The fault was fixed by re-terminating pair in joint.
Have you given any general advice to EC ?: 6159.06-\n\n Advice given to end customer: Optimal speed already achieved.
Where did you perform the final PQT ?: 3835.01-\n\n Final test results: Final PQT performed at the NTE back plate.
What was the result ?: 3836.02-The test passed with amber parameters on 2022-03-28T17:59:01.
What was your final test ?: 3417.03-Final FastTest completed.
What was the result ?: 4000.01-The test passed on 28/03/2022 18:03:58.
-
I have two comments to make about the above quotation --
- The "customer" is A&A.
- The "end customer" is yourself.
It would have been interesting to know the location of faulty joint. If you remember back to last July, 2021, just "three words (https://forum.kitz.co.uk/index.php/topic,26128.msg438962.html#msg438962)" would have been sufficient.
-
I’m afraid that’s all I have, so I published it as is, many warts and all. I can’t see a "Location:" field in that lot that uses the three words system (something that was new to me, my wife then explained it).
Initially I saw the following, with delight:
Where was the fault located ?: 743.02
I thought that the number was some kind of TDR-returned distance along the cable. But then I noticed similar numbers associated with adjacent records, so it seems my initial thought about cable distance was hopelessly over-optimistic.
Burakkucat wrote:
> Let's hope it is for good . . . For good until another defect occurs, of course. :)
I am cautiously optimistic this time because at least (i) the engineer found something non-trivial and ‘real’, and (ii) actually made a change - he fixed it. The many other call-outs to HCD complaints have most often been cured by an engineer seemingly doing nothing at all. Hence theories about how- the fix could be due to phoning the problem line, thus applying ring voltage, or
- the fix could be to do with turning modems off for a while then back on again - this requires bad modems. However it would seem the bad modem theory is ruled out now following a thorough modem swapout.
[digression]When confronted by problems that are very very hard to debug, many years of experience has taught me to consider multi-bug scenario, with different problems simultaneously presenting different appearances so that rule-outs can mislead the investigator completely. "It can’t be x, because…”-type ‘negative’ reasoning.[/digression]
-
Unfortunately, here we go again, but with line #2 this time. I can let AA know what’s going on. Hollow curve disease detected when I wasn’t even looking for it, spotted it on Monday, with a below-chord-depth of around 2dB later increasing to 4dB. Is it better to let it develop into full badness with very low synch rate before calling OR in ?
The downstream synch rate was at ~3 Mbps but with a downstream SNRM of < 1dB so it was hanging on in there until it wasn’t, lost sync and dropped to 2.5Mbps downstream yesterday.
-
If you can tolerate it, let it fully "ripen" before asking A&A for help.
-
Agreed. I suggested to AA today that we both keep an eye on the ripening, and give it time.
A while ago I improved the reporting in my HCD detection code so that it now outputs the vertical drop distance in dB from the midpoint of a chord to the graph below. It also indicates which of the two tested chords it’s referring to; that is either chord #1, the chord between tones 40-85, or chord #2, between tones 40-60. It now gives me a concrete badness figure and saves me having to squint and use a ruler.
-
Maybe OR should hire you to rewrite the ADSL DLM? ::) Or at least the fault detection code.
-
I hear you Alex. ;) Has anyone apart from me ever seen this anywhere? I never saw this before Jan/Feb 2020 (can’t remember when it was exactly).
Stupid question of mine, getting confused. Does the DLSAM/MSAN have any access to the SNRM-vs-tones data that I can see in my modem at this end?
-
Has anyone apart from me ever seen this anywhere?
No, not I.
Does the DLSAM/MSAN have any access to the SNRM-vs-tones data that I can see in my modem at this end?
The xTU-C and xTU-R do exchange data. Exactly "when, how and what", I know not. (Or if I did know, I've now forgotten. :-[ )
-
I hear you Alex. ;) Has anyone apart from me ever seen this anywhere? I never saw this before Jan/Feb 2020 (can’t remember when it was exactly).
No, not I.
Before Weaver posted about HCD, I wonder if you would have noticed? Now I guess, being aware, it's something you'll (maybe subconsciously) look out for.
-
Before Weaver posted about HCD, I wonder if you would have noticed?
A good question. My answer has to be probably not.
Now I guess, being aware, it's something you'll (maybe subconsciously) look out for.
Now with a typical VDSL2 circuit, rather like that of your own, it's not something that could be easily seen.
-
Indeed, it would be hard to translate this concept into VDSL2 with its multiple d and u bands. I don’t know why my experience of it is limited to tones 40 - 85 or maybe just possible 40 - 100.
Have agreed with AA that we will keep an eye on it as the fault develops.
The HCD detection code reports:
*❗ DSL link #2 defect detected: so-called ‘hollow curve disease’ in the downstream bit-loading. (-5.149 dB at tone 62 below chord 1: tones 40-85) ❗ Serious fault 🔺
I see two overall measures of how bad the fault has become. One is the above 5 dB, the drop below the chosen approx ‘midpoint’ of a certain chord.
The other measure compares what is with what used to be, as follows: (I’ve actually cheated and compared the afflicted line 2 with the health line 4) The total hollow curve disease peak-to hollow-depth is now 14dB; that is the amount that the SNRM of the former best downstream tone has gone down, forming the bottom of the ‘hollow curve’ of SNRM vs tones. The best tone was approx tone 44 at around 41dB SNRM; now the local minimum is somewhere around tone 62 at 27dB. So either the noise has gone up on those frequencies, or the susceptibility to noise has, or the signal strength has gone down, or the line attenuation has gone up on those tones.
-
Yesterday, the downstream CRC rate and ES rate were both steadily rising, also being higher at night as might be expected. The SNRM downstream was a mere 1.8 dB rather than the target of 3dB. In order to cool the rate of errors, which had risen to roughly one CRC error ever two seconds, I forced a resynch. The period after the resynch is also visible at the end of the graph below:
(https://i.postimg.cc/XJhxLxnh/FD4-ECD9-B-955-E-48-EB-91-AF-DE2132-B7-F7-D3.png)
Here is the usual HCD picture in its current state of ‘ripening’ :
(https://i.postimg.cc/ncd59KDy/2-E10-CCEB-A8-E5-49-E0-9968-FAD7460-F6584.png)
Summary of DSL links’ wellbeing and error counts
────────────────────────
* There are 3 modems in total. They are: #1, #2 and #4
* The active, contactable modems are: #1, #2 and #4
* The modems successfully queried are: #1, #2 and #4
───────────────────────
*** ***
*** There is some SERIOUS BADNESS ! ***
*** All is not well ! 😦 ***
*** ***
───────────────────────
--
* Sync rate: The following link has a really low downstream sync rate, below min:
Link #2 downstream: current sync rate 2380 kbps is below minimum expected downstream rate (2800 kbps) ❗Line #2 fault 🔺
--
*❗Link #2 defect detected: so-called ‘hollow curve disease’ in the downstream bit-loading. (-5.426 dB at tone 62 below chord 1: tones 40-85) ❗ Serious fault 🔺
--
* ES (less serious): The following links have a few CRC errors, at lower error rates, where the ES rate < 60 ES / hr (†):
Link #2 downstream: latest period: ES per hr: 13.02, mean time between errors: 276.5 s, collection duration: 553 s
Link #2 downstream: 'previous' period: ES per hr: 24.00, mean time between errors: 150 s, collection duration: 900 s
──────────────────────────────
(†) The duration of the ’latest‘ errored seconds (ES) collection
bucket is variable, with a _maximum_ of 15 mins. The buckets’
start times are always 15 mins apart. A ‘previous’ bucket's
duration is a fixed 15 mins. An ES is a 1 s time period in which one
or more CRC errors are detected. A CRC error is a Reed-Solomon
coding-uncorrectable error, ie. corrupted data is received that
cannot be recovered.
──────────────────────────────
◅ ◅ ◅◊▻ ▻ ▻
-
xDSL needs nuking from orbit, just to be sure. What a horrendous waste of time, energy and effort chasing down these constant faults over 7km of rusting wires precariously jointed is.
Anything further on being rid of this nonsense and being able to replace it with something that just works? There is certainly no reason why a remote pizza box OLT cannot be installed in the area.
-
It wasn’t like this before early 2020. Yes there were plenty of faults but not the same fault again and again, never getting fixed. Line 1 hasn’t failed in I don’t know how long. This forum serves as a record I suppose, but it would be a pain going back through old threads trying to find the most recent line 1 fault. I could go back through AA’s records more easily but there’s a lot of noise in there.
As for nuking from orbit don’t I wish. I don’t have the money nor the physical abilities to be able to take up some of the options available that would truly nuke the problem from orbit. FTTP is getting close now after 18 years of waiting, and the system as a whole is indeed highly reliable because the bonding covers up all known faults. Even with one line down it’s still fine and my wife is able to work with no problems. We also have 3G as a transparent failover should three copper lines fail and 4G is available to us too. If I didn’t monitor the system closely I wouldn’t even know that a line has gone down/up, or down and staying down. My wife gets an SMS (email optional too) to tell us that a line has gone down. Ten years ago, before I spent money in starting up the business, I definitely could have nuked the thing from orbit.
My aims in boring you all with documenting the history of this are (i) to use your collective brain power considering how weird it all is (ii) to serve as a historical record for me too, (iii) to be of interest to a small minority.
As for your point about the waste of time, you would thing Openreach would have some escalation department for repeat faults. They ought to know about this, and if not then they could be reminded and then they should either review the entire cable bundle or simply replace the whole lot with fresh copper as part of decent customer service.
I could ask AA to try and escalate this within OR and/or BTW. Presumably if I email OR they’ll tell me to contact my ISP.
Could you put your thinking caps on (or keep them on) and please, electronic engineering people, please let me know how we can have either a frequency-selective noise susceptibility and/or signal attenuation problem so that it amounts to reduced SNRM that isn’t truly narrow-band and which can somehow get fixed without us knowing how.
-
There is certainly no reason why a remote pizza box OLT cannot be installed in the area.
I would suspect that either the garage or the stable at "The Weaving Shed" could be made available to Openreach for such a purpose. All of Heasta could then be provisioned from that OLT.
The free provision of a fibre from that OLT to an ONT located in Senior Management's office would be a fair exchange. ("Senior Management" == "Finance Director" == Mrs Weaver.)
-
As for your point about the waste of time, you would thing Openreach would have some escalation department for repeat faults. They ought to know about this, and if not then they could be reminded and then they should either review the entire cable bundle or simply replace the whole lot with fresh copper as part of decent customer service.
sed 's/lot with fresh copper as part/lot with an optical fibre link as part/' :angel:
-
Indeed. Agrees with your sed. OR could save themselves money or maybe it’s BTW or AA paying for all the stupid engineer call-outs.
Before 2020, when a fault was fixed that was an improvement in the quality of the line. A fault fixed stayed fixed. It was one less, and enough call-outs and we would ultimately end up with a line with zero bugs. Twenty+ years ago there was a situation like this with repeat problems and it was fixed by replacing a piece of copper that was lying in the road and being driven over by vehicles every day. Instead the cable was replaced and buried properly. End of faults.
In my opinion, ideally when a fault is ‘fixed’ it ought to remain fixed. Clearly OR can’t fix faults that vanish the moment they arrive, but in that case they need to capture the badness without disturbing the system too much. Ghostbusters. A tall order for them. I need to find out whether or not the HCD faults survive the action of unplugging a modem from the line and then plugging it back in, the modem remaining powered up.
I wish AA had their own engineers, now I come to think of it. Don’t know how the practicalities or financial side of that would work but I’m going to ignore that for the moment. That would be fantastic. My own Black Sheep. Either on the end of a phone line or here in Heasta in person or someone remotely accessing kit over the internet (over 4G in this case).
-
Replacing the copper would be a waste of money and time. Almost certainly not going to happen. Overbuild with fibre delivers a faster, more reliable and better service and is happening sooner or later.
I need to make some calls tomorrow and see what I can find out. Nostalgic as it is seeing weird faults on really long copper loops it is 2022 and we've the technology to do far better.
-
We have the technology, but I don’t have the cash, nor the physical abilities that most of us have (as I’m stuck in bed much of the time btw).
Agreed that replacing the copper would be a waste of money and time. BT has spent 15 years saying that, while they continue to waste more money and my time on repairs. They are expected to have a commitment to a certain level of customer service.
[A polite reminder to all. ;) Alternative link technologies have been discussed elsewhere already and are off-topic for this thread. This thread is about documenting faults and diagnosis musings.]
Good point about my garage. The stable isn’t suitable, and nor is the garage as there was some rodent damage in the garage some years ago, maybe before Thomas turned up though. Might be much better to use a different building, one that is imaginatively called "the Big Shed" which I used to use as a warehouse for electronics hardware and is now a residents’ lounge for Shepherd Hut guests. Has mains, but has no advantage over just using Janet’s office though, the latter being much much more secure and has much more room.
-
I think there’s a slight trend upwards:
(https://i.postimg.cc/8PSm5vy3/FDC4-B49-A-6-C7-C-4235-A429-CD0-F59-ACFF31.png)
Have forced a resync twice to keep the error rate down. This has resulted in a further downstream sync rate reduction, to 2.310 Mbps, from an original 2.9 Mbps; upstream unchanged at 573 kbps
-
After I forced the target downstream SNR up to 6dB, in order to control the rising error rate, it only remained like that for a few days before DLM pushed it back down to 3dB at around 06:30 this morning. The HCD and the downstream sync rate remained stable apart from this DLM event. The downstream sync rate is now around 2.2 Mbps @3dB.
-
The deterioration continues slowly. The HCD hollow is now about 8dB deep[!]. Yesterday I checked the stats and the CRC error rate was ~2k per hr. I decided this was completely silly and increased the downstream target SNRM from 3dB to 9dB. This knocked the downstream synch rate right down to 1.191 Mbps, as is to be expected. When all is well it’s 2.9 Mbps downstream at 3dB with no errors.
My thinking is to definitely let it get ridiculously bad before OR are called in, and this is what I proposed to AA, asking them to keep an eye on it, to check it regularly and to decide when they think it’s bad enough to give an OR engineer a really, really good chance of finding the fault. It just means living with a rubbish line for longer than is perhaps necessary.
Looking at the framing parameters, I can’t seem to get any downstream interleaving, downstream D=1. I have it set to downstream interleaving=ON in clueless.aa.net.uk, but that doesn’t seem to be working. Don’t understand what’s going on. A few questions:
- I don’t know how to tell how beneficial this would be - can’t see what type of noise I’m getting. Surely more interleaving is never a bad thing, if you don’t care much about latency.
- There’s plenty of interleaving on the upstream, D=8.
- Perhaps this is something to do with the presence of downstream L2ReTx in the form of PhyR? Maybe the modem just decides it doesn’t need to have interleave if it can do retransmissions, and if so, it then overrides any instructions from outskde? Does that make sense or seem doable?
- In FTTC, what happens about interleave depth D vs G.INP on/off?
-
Deterioration continues. I’ve now forced the downstream target SNRM up to 15dB which has brought the downstream sync rate down to 611kbps. This is the best I can do now, and any worse and I’ll have to turn the modem off in order to prevent it from polluting the whole bonded link with CRC errors. It had also got to the stage where it was dropping sync sometimes (at 12dB downstream). I’ve updated AA and asked them to decide if now is the time to finally give up and call OR in, given that we’re waiting until the fault is likely to be successfully located and properly fixed, because it’s really really bad and shows up.
-
A target SNRM of 15 dB is incredibly severe and the fault seems to be as hard as it can be (other than a disconnection). :(
-
Indeed. It has now completely failed to synch and the link has been down for many hours. I emailed AA at the weekend so they will get it on Tuesday morning.
-
AA has booked an OR engineer for Thursday. Here are some pictures for those who don’t frighten easily:
(https://i.postimg.cc/SN2n9Rwf/5-F2491-DF-4355-496-C-AE2-C-1-A0-F0-CD0566-E.png)
(https://i.postimg.cc/zBrCtK1N/6-B086679-5-B0-C-4-E39-8-EC5-7-EB06303004-F.png)
(https://i.postimg.cc/c4rncX8Y/144399-E0-4-BC1-44-EB-A42-F-54-FABDA31823.png)
-
Full stats sampled on Wednesday afternoon. I don’t quite know what has been going on here; Things have improved a lot in the past hour or so. The ES have dropped to 0 at last. This is presumably because of the high RS error correction combined with the recent arrival of a small amount of interleaving, D=2, not enough but finally better than zero. The payload of a downstream RS FEC frame is small at only 6*2=12 bytes (plus one sync byte), with R=16 bytes of RS CRC appended. I’m wondering if it’s possible that this was caused by the intervention of BT DLM. I don’t know if BT DLM is even turned on for this line. Can BT DLM be turned off on a BT 21CN ADSL2 line? I could ask AA.
If it stays like this then at least it’s usable, reliable, albeit at this painful downstream speed of ~10% of what it should be. An interesting investigation into a sick line at its very slowest.
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 439 Kbps, Downstream rate = 1324 Kbps
Bearer: 0, Upstream rate = 522 Kbps, Downstream rate = 288 Kbps
Link Power State: L0
Mode: ADSL2 Annex A
TPS-TC: ATM Mode(0x0)
Trellis: U:ON /D:ON
Line Status: No Defect
Training Status: Showtime
Down Up
SNR (dB): 10.5 6.0
Attn(dB): 67.5 41.5
Pwr(dBm): 16.7 12.3
ADSL2 framing
Bearer 0
MSGc: 91 11
B: 6 62
M: 2 1
T: 1 1
R: 16 12
S: 1.2903 3.7975
L: 186 158
D: 2 8
Counters
Bearer 0
SF: 561802 574176
SFErr: 12443 0
RS: 27247410 1166457
RSCorr: 414017 858
RSUnCorr: 37103 0
ReXmt: 540802 0
ReXmtCorr: 527585 0
ReXmtUnCorr: 37235 0
Bearer 0
HEC: 6510 0
OCD: 0 0
LCD: 0 0
Total Cells: 6169219 11410815
Data Cells: 300046 297485
Drop Cells: 0
Bit Errors: 881256 0
ES: 15505 12
SES: 4771 3
UAS: 154729 151302
AS: 9266
Bearer 0
INP: 28.00 2.00
INPRein: 0.00 0.00
delay: 8 8
PER: 15.64 16.13
OR: 49.60 8.42
AgR: 321.14 528.81
Bitswap: 13/13 6/6
Total time = 2 days 5 hours 43 min 59 sec
FEC: 1090244 1276
CRC: 67862 0
ES: 15505 12
SES: 4771 3
UAS: 154729 151302
LOS: 298 0
LOF: 888 0
LOM: 3 0
Latest 15 minutes time = 13 min 59 sec
FEC: 2171 315
CRC: 0 0
ES: 0 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 15 minutes time = 15 min 0 sec
FEC: 2642 51
CRC: 0 0
ES: 0 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Latest 1 day time = 5 hours 43 min 59 sec
FEC: 789678 978
CRC: 39210 0
ES: 5857 3
SES: 1253 0
UAS: 9323 8524
LOS: 21 0
LOF: 189 0
LOM: 0 0
Previous 1 day time = 24 hours 0 sec
FEC: 300566 298
CRC: 28652 0
ES: 7570 4
SES: 2169 0
UAS: 74173 72594
LOS: 151 0
LOF: 444 0
LOM: 1 0
Since Link time = 2 hours 34 min 25 sec
FEC: 414017 858
CRC: 12443 0
ES: 3227 0
SES: 12 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
NTR: mipsCntAtNtr=0 ncoCntAtNtr=0
-
AA clueless detailed CQM graphs. Look how much of the time it has been down. Note the blue spike at around 13:40; I think this is a resynch, and I think it’s due to BT DLM possibly. There was a lot of packet loss at the start of the synched period before this but then things improved a lot. Is it possible that the earlier error rate was high enough to trigger the intervention of BT DLM, forcing a resynch.
(https://i.postimg.cc/NMb4YyDT/CA224-E41-58-DD-4-D8-D-8-F38-C3805-C2-C1-B24.jpg)
-
I can't find the documents/charts but isn't that pretty much the lowest sync rate ADSL is capable of? Not really surprising its unstable.
Actually scratch that, I see you were even lower before, I didn't know ADSL could go below around 240kbit.
-
I thought 288kbps downstream was the slowest, but as you see on the previous day’s summary then it was 189kbps downstream, and I didn’t know that was possible. It’s been a nuisance having to wait so long with the line being only semi-usable but I was determined to get the best chance of letting an OR engineer see something significant.
-
Yeah when I saw 288 I was thinking the same thing, though possibly slightly lower but definitely not below 240.
If its the same as VDSL though I remember on Digital Region it was weird as the DSLAM would report a different sync rate to the modem. I was told its something to do with which value it reports, something about with or without overheads I think.
-
An engineer came at about 17:00 today, Thursday, and did some work on a joint at a pole close to the house, the lower pole, right by the road (not the very closest pole to the house). Line cwcc@a.2 is absolutely perfect now. ;D Saga over !! ;D Notes are not back yet, so we’ll hopefully see some answers.
-
An engineer came at about 17:00 today, Thursday, and did some work on a joint at a pole close to the house, the lower pole, right by the road (not the very closest pole to the house).
From the various images I've seen over the years, there are no joints in the cables at the upper (house) pole. It is purely a carrier pole.
At the lower level (road side) pole there are two sets of joints per pair. The first joint (in each pair) is at ground level, where the cable which runs up to the top of the pole is connected to the passing D-side cable. The second joint (in each pair) are gel-crimps, housed in the pole top BT66. The lower level (road side) pole does not have a traditional DP "block and tail".
Line cwcc@a.2 is absolutely perfect now. ;D Saga over !! ;D Notes are not back yet, so we’ll hopefully see some answers.
That's good to know. :) (I am assuming that the fault was water ingress.)
-
With any luck the notes will come back tomorrow. This could in theory have been fixed a month ago but of course there might have been nothing visible to an engineer back then that could have been located.
-
b*cat nods his head in agreement.
-
For the first time in a couple of months we see:
Summary of DSL links’ wellbeing and error counts
────────────────────────────────────
* There are 3 modems in total. They are: #1, #2 and #4
* The active, contactable modems are: #1, #2 and #4
* The modems successfully queried are: #1, #2 and #4
──────────────────────────────────────────
*** ***
*** All is well ! No errors or problems detected. ✅ ***
*** ***
──────────────────────────────────────────
◅ ◅ ◅◊▻ ▻ ▻
-
Normality restored.
(https://i.postimg.cc/fyBf0rfB/2179-E09-E-D8-DF-4-CCE-86-CF-F5-D5-D0-B7-CEA3.png)
-
Purrfect. :)
-
Status of each modem
───────────────────
* There are 3 modems in total. They are: #1, #2 and #4
* The active, contactable modems are: #1, #2 and #4
* The modems successfully queried are: #1, #2 and #4
--
Link #1 sync rate: downstream: 2.783 Mbps at SNRM 2.9 dB; upstream: 653 kbps at SNRM 5.9 dB
--
Link #2 sync rate: downstream: 2.854 Mbps at SNRM 3.2 dB; upstream: 525 kbps at SNRM 5.8 dB
--
Link #4 sync rate: downstream: 2.876 Mbps at SNRM 2.9 dB; upstream: 386 kbps at SNRM 5.9 dB
◅ ◅ ◅◊▻ ▻ ▻
As discussed in an earlier thread (https://forum.kitz.co.uk/index.php/topic,23434.msg397232.html#msg397232), note how the downstream sync values are 100k - 300kbps less than in the depths of winter, so over 10% speed loss downstream!
OR engineer’s notes:
BT Fault 5-887769895622 Right When Tested; End User Equipment;All BT tests completed ok, diagnostics show no fault condition
Did all the connections, extensions and DSL filters meet the minimum standards ?: Yes
Have you conducted a sync test (modem light on) ?: Yes
Did you complete the Frames module ?: No
Did you complete the Base module ?: Yes
Did you complete the end customer Premises Wiring Module ?: No
Did you complete the end customer Premises Equipment Module ?: No
What was the result of the initial pair quality test ?: Fail
Did you complete the Network module ?: Yes
Have you successfully demonstrated connectivity to the NTE ?: Yes
2022-05-05 18:05:07 Engineer: The engineer has now completed the visit. BT
2022-05-05 18:01:16 Yesterday 18:01:46 BT Fault 5-887769895622 Right When Tested; End User Equipment;All BT tests completed ok, diagnostics show no fault condition
What did the end customer describe as the primary issue ?: 3762.03-Customer Report: End customer informed us that broadband is not stable / intermittent.
Where did you perform the initial PQT ?: 3743.01-
Initial test results: PQT test performed at NTE back plate.
What was the result ?: 3763.02-The test passed with amber parameters on 2022-05-05T16:15:47.
Have you identified any other issue(s) in the OR network ?: 6357.03-
What was found: The line was noisy.
Where was the fault located ?: 743.01-
Actions to resolve: Fault located in the D side overhead network.
What did you do to fix it ?: 745.21-Resolved by re-terminating drop wire at external block.
Have you given any general advice to EC ?: 6159.07-
Advice given to end customer: Stop changing SNR to 6dB. Line too long for this setting. .
Please specify: 6160.01-
Where did you perform the final PQT ?: 3835.01-
Final test results: Final PQT performed at the NTE back plate.
What was the result ?: 3836.02-The test passed with amber parameters on 2022-05-05T16:20:59.
What was your final test ?: 3417.03-Final FastTest completed.
What was the result ?: 4000.01-The test passed on 05/05/2022 17:55:33.
A few remarks:- Recent OR engineers have been obsessed with downstream SNRM, whilst having absolutely zero clue about it.
- What is this engineer thinking when he made these notes about 6dB? "Stop changing SNR to 6dB. Line too long for this setting."
- He said no such thing to us, I had done no such thing,
- nor was it at 6dB downstream, it was at 10-11dB with a target of 12dB as a desperation measure.
- The line is usually set to 3dB downstream.
What on earth are the numbers that are sprinkled throughout the report? eg: 3417.03
I asked AA, who said ‘no idea’ and ‘presumably some BT thing’.