Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

Here we go again, line 4, this time. HCD just like before, unfortunately the first signs of the same disease are showing in the SNRM vs tones graph measured by the modem, with slight loss of downstream sync rate. The very distinctive shape of a dent in the SNRM vs tones graph is what I termed before ‘hollow curve disease’. No real problem yet except for the slight speed loss, and can’t call OR out yet to find the incipient fault. As before I just have to wait until it gets to a really really bad speed loss and becomes unusable. Just ask if you would like any detailed pictures. God, roll on FTTP / FTTPoD.

Asked AA to keep an eye on this line for me as it develops, and asked if anything funny visible in their tests yet.


Line 4, SNRM vs tones:




Line 4 detailed stats and errors:

Code: [Select]

xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 308 Kbps, Downstream rate = 2996 Kbps
Bearer: 0, Upstream rate = 355 Kbps, Downstream rate = 2691 Kbps

Link Power State: L0
Mode: ADSL2 Annex A
TPS-TC: ATM Mode(0x0)
Trellis: U:ON /D:ON
Line Status: No Defect
Training Status: Showtime
Down Up
SNR (dB): 2.9 5.8
Attn(dB): 65.5 41.4
Pwr(dBm): 18.2 12.4

ADSL2 framing
Bearer 0
MSGc: 52 11
B: 18 4
M: 8 16
T: 5 9
R: 16 16
S: 1.7778 7.0459
L: 756 109
D: 1 4

Counters
Bearer 0
SF: 5559084 369689
SFErr: 9 1
RS: 201516811 3849449
RSCorr: 15758 1377
RSUnCorr: 9 0

ReXmt: 3003 0
ReXmtCorr: 3003 0
ReXmtUnCorr: 9 0

Bearer 0
HEC: 7 1
OCD: 0 0
LCD: 0 0
Total Cells: 571851294 75488003
Data Cells: 45393609 5761466
Drop Cells: 0
Bit Errors: 477 20

ES: 829 11
SES: 25 0
UAS: 184 163
AS: 90113

Bearer 0
INP: 28.00 2.00
INPRein: 0.00 0.00
delay: 8 7
PER: 16.11 16.84
OR: 28.80 8.07
AgR: 2707.42 361.91

Bitswap: 9666/9667 319/319

Total time = 22 days 3 hours 58 min 43 sec
FEC: 411735 45157
CRC: 2931 13
ES: 829 11
SES: 25 0
UAS: 184 163
LOS: 1 0
LOF: 9 0
LOM: 0 0
Latest 15 minutes time = 13 min 43 sec
FEC: 93 6
CRC: 0 0
ES: 0 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 15 minutes time = 15 min 0 sec
FEC: 111 8
CRC: 0 0
ES: 0 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Latest 1 day time = 3 hours 58 min 43 sec
FEC: 2519 140
CRC: 1 0
ES: 1 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 1 day time = 24 hours 0 sec
FEC: 18380 1274
CRC: 63 1
ES: 42 1
SES: 0 0
UAS: 60 60
LOS: 0 0
LOF: 0 0
LOM: 0 0
Since Link time = 1 days 1 hours 1 min 53 sec
FEC: 15758 1377
CRC: 9 1
ES: 9 1
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
NTR: mipsCntAtNtr=0 ncoCntAtNtr=0


All modems:

                      Status of each modem
                     ─────────────────

* There are 3 modems in total. They are:     #1, #2 and #4
* The active, contactable modems are:         #1, #2 and #4
* The modems successfully queried are:       #1, #2 and #4

--
   Link #1  sync rate: downstream: 2.680 Mbps at SNRM 2.9 dB; upstream: 650 kbps at SNRM 5.9 dB
   Link #1  uptime: 5 days  13 hours  10 min 27 sec

--
   Link #2 sync rate: downstream: 2.773 Mbps at SNRM 2.9 dB; upstream: 519 kbps at SNRM 7.9 dB
   Link #2 uptime: 20 days  13 hours 43 min 41  sec

--
   Link #4 sync rate: downstream: 2.691  Mbps at SNRM 2.8 dB; upstream: 355 kbps at SNRM 6.1  dB
   Link #4 uptime:  1  day  1  hour 4 min 9 sec

--
                                              ◅ ◅ ◅◊▻ ▻ ▻

[burakkucat]

b*cat nods and thinks "Yes, once again. As before." 

So you are now in that wait and see mode, waiting for the fault to "ripen".

[Weaver]

Exactly. I’ve asked AA to just watch and decide when to spring. I hope they’ll remember about it and make the decision when to pounce for me. I wouldn’t have known anything without the HCD detector code firing off an alarm. It successfully detected HCD to begin with, but then the shape shifted and the code no longer picked the problem up, so the new algorithm was needed. I’ve noticed that the SNRM values in the curve do ripple and change quite frequently, by very small amounts.

So now the three chords have rhs endpoints at tone 60 / 70 / 80 approx values in each case, as some are dynamically adjusted slightly by stalactite detection and avoidance. In one case, the test condition is not just that the y-value (ie SNRM) of the curve at the test point / chord midpoint is below the chord, but it has to more than a certain minimum amount below, a small amount. This is to guard against bogus HCD reporting due to noise / ripple. It does reduce the effectiveness of the detector though and means it can miss HCD cases, so I have not used this technique unless experience suggests there’s a real need in a particular case. It’s not used with chord 3; I think only with chord 2, the mini-notch detector.

I need to update the documentation for this code accordingly now.

Now I remember - the very first indication that anything odd was happening was a retrain for no reason, which reduced both the downstream and upstream rates slightly. This made me run my health checker program, which includes full HCD detection, and that reported the alarm. Looking back, there was no large amount of errors that would perhaps force a retrain, so it was a mystery to me. If I recall correctly, there were zero CRC errors in both directions for a substantial number of hours before the retrain, even at downstream target SNRM= 3dB and upstream target SNRM= 6dB. And the line wasn’t in the BT ‘training period’ iirc.

[Weaver]

And while we’re waiting for the line 4 fault to brew, line 2 has now also suddenly gone spectacularly bad in one leap. Line 2 did a retrain on Sun evening, and lost a lot of downstream sync rate, down from 2.7 Mbps to ~1.95 Mbps with very very bad HCD, a 6dB deep hollow:

Links’ sync rates:
  link #1:    downstream: 2.680 Mbps, upstream: 650 kbps
  link #2:    downstream: 1.963 Mbps, upstream: 512 kbps
  link #4:    downstream: 2.691 Mbps, upstream: 355 kbps

                                              ◅ ◅ ◅◊▻ ▻ ▻

I’ll post up the picture of HCD horror if anyone is interested.

[burakkucat]

. . . line 2 has now also suddenly gone spectacularly bad in one leap.

   

[Weaver]

Here’s the latest line 2 horror show:




       Summary of DSL links’ wellbeing and error counts
      ─────────────────────────────────────

* There are 3 modems in total. They are:     #1, #2 and #4
* The active, contactable modems are:        #1, #2 and #4
* The modems successfully queried are:       #1, #2 and #4

       ───────────────────────────────────────
       ***                                                                    ***
       ***     There is some SERIOUS BADNESS !             ***
       ***                      All is not well !   😦                      ***
       ***                                                                    ***
       ───────────────────────────────────────

--
* Sync rate: The following link has a really low downstream sync rate, below min:
   Link #2 downstream: current sync rate 916 kbps is below minimum expected downstream rate (2700 kbps)  ❗Line #2 fault  🔺

--
* SNRM: The SNRM of the following link is out of range:
   Link #2 downstream: current SNRM: 8.8 dB is too high;  above the expected maximum SNRM ( 3.8 dB )

--
* HCD:❗So-called ‘hollow curve disease’ (HCD) defect detected in the downstream SNR-vs-tones curve. Serious fault. 🔺
   Link #2: Summary:  tone 62 is 0.55972 dB below chord  1: (tones 40-25.9375); ❗
   Link #2: Details: chord  1: [ tone 40: 22.5 dB; curve midpoint test tone 62:  16.5625 dB; tone 25.9375: 25.9375 dB ]; chord midpoint test tone 62:  17.12222 dB.

--
* HCD:❗So-called ‘hollow curve disease’ (HCD) defect detected in the downstream SNR-vs-tones curve. Serious fault. 🔺
   Link #4: Summary:  tone 62 is 2.225 dB below chord 3: (tones 40-70);❗
   Link #4: Details: chord 3: [ tone 40: 37.0625 dB; curve midpoint test tone 62: 34.1875 dB; tone 70: 34.625 dB ]; chord midpoint test tone 62: 36.41250 dB.

--
*❗ES (serious): Links with CRC errors at higher error rates, where the ES rate ≥ 60 ES / hr (†):
   Link #2 downstream: 'previous' period: ES per hr:  1664.00, mean time between errors:   2.16 s, collection duration: 900 s

──────────────────────────────
(†) The duration of the ’latest‘ errored seconds (ES) collection
bucket is variable, with a _maximum_ of 15 mins. The buckets’
start times are always 15 mins apart. A ‘previous’ bucket's duration
is a fixed 15 mins. An ES is a 1 s time period in which one or more
errors are detected ie. corrupted data is received that either can not
be recovered by back-calculation using the additional redundant
error correction information included with each message, or by 
being resent a certain maximum number of times without being
received corrupted.
──────────────────────────────      
                                     ◅ ◅ ◅◊▻ ▻ ▻


                 Status of each modem
                 ────────────────

* There are 3 modems in total. They are:     #1, #2 and #4
* The active, contactable modems are:        #1, #2 and #4
* The modems successfully queried are:       #1, #2 and #4

       ───────────────────────────────────────
       ***                                                                    ***
       ***     There is some SERIOUS BADNESS !             ***
       ***                      All is not well !   😦                      ***
       ***                                                                    ***
       ───────────────────────────────────────


--
   Link #1  sync rate: downstream: 2.680 Mbps at SNRM 2.9 dB; upstream: 650 kbps at SNRM 6.1  dB
   Link #1  uptime: 9 days 9 hours  17 min  11  sec

--
   Link #2 sync rate: downstream: 916 kbps at SNRM 8.2 dB; upstream: 512 kbps at SNRM 8.2 dB
   Link #2 uptime: 7 hours 2 min 49 sec

   Link #2 ES: downstream: latest  15 minutes time: ES  1610.53 per hr; MTBE 2.24 s; bucket duration 380 s
   Link #2 ES: downstream: previous  15 minutes time: ES 2280 per hr; MTBE  1.58 s; bucket duration 900 s

--
   Link #4 sync rate: downstream: 2.691  Mbps at SNRM 2.5 dB; upstream: 355 kbps at SNRM 6.1  dB
   Link #4 uptime: 4 days 21  hours  10 min 51  sec

--
                                              ◅ ◅ ◅◊▻ ▻ ▻

Line 4 latest graph. Classic early HCD::

[Chrysalis]

Interesting to know these Hollow Curves are a fault condition, I had them on my ADSL service for its entire duration which survived numerous pair swaps.  I might dig out some old DMT samples if I ever get bothered.

[Weaver]

Wow, Chrys - that says to me that they are some reasonably well-known electrical and possibly electro-mechanical fault conditions. I’d love to know exactly what the Openreach engineers did to repair things when they actually did something, as opposed to turning up and fixing the fault by the magical influence of their very arrival or presence.

Note about line 2: the line 2 downstream CRC error rate is so ridiculously high - see previous post - that it’s out of control and will, I assume [?], muck up the performance of the whole bonded set, so I might need to take line 2 down but I want AA to have an opportunity to see it obviously. So I’ve left it up for now but have increased the downstream target SNRM to 9dB from 3, and even that is nowhere enough, it’s that bad. Note that the figures posted are with 9dB downstream target SNRM set for line 2 and that’s going to need to go right up further.

[Chrysalis]

In my case weaver it was pretty much a wont fix line is within spec from openreach and this included my first stint on AAISP on that line, AAISP did get me moved to a different line card vendor as well as some D side pair swaps and the second vendor had better handling of line errors as well as a few more usable strong tones.  I eventually got an engineer to admit the fault needed very expensive roadworks on an extremely busy road around the city centre on the E side, the source of the fault "might" still exist today, but I am on VDSL now with fibre now been used E side it bypassed the issue.

My HCD was bad enough that 15db SNRM and interleaving couldnt control the stability (at the time was Openreach's most stable DLM profile).

[Weaver]

Amazing! Thanks for letting me know!

Update, I reported lines 2 and 4 to AA on Monday morning, OR engineer arrived Tuesday morning, booked for line 2 (only) pretty impressed by that. Both lines sorted straight away. Engineer, among other things, went up the pole by the roadside below my house - that trouble spot again. It seems that line 4 was fixed by OR magic, of the kind that has fixed faults at many a callout in the past, yet we have never understood how.


        ===========   F  I  X  E  D  !!   ==================


I told AA that both lines were looking splendid, apart from a slight loss of speed 50-100 kbps, so both tickets were to be closed. I noticed that the next morning AA added a note in clueless “Note LTOK Work Point: JB26”. I know what a JB26 is but I don’t understand how that note got put there as I can’t see it in the engineer’s notes.

The engineer was a known dreaded moaner, who has moaned and complained to my wife before. He shall not be named in public. He complains to my wife about [possibly inaccurate 3rd party account] "you keep on turning it up and it can’t handle it" (the warp engines cannae take it captain) "it" presumably being the downstream target SNRM which as we all know must not be turned up too high as it makes lines fail. My wife was very worried about "him coming again". I told her to turn the moaner over to me, or to AA, and Janet had the required AA contact number at the ready. On this occasion Janet handled him by saying "it’s no good telling me, I’m not the techie one - It means nothing to me", so all was well. I have told her if suchlike cause her any anxiety then I will ask AA to ‘have a word’, but I’m very worried about OR overreaction.

The line 4 has no HCD at all, so as I said, I have no idea what happened and the engineer was not booked to do anything to line 4 - there’s no mention of it in the notes below. I checked the clueless log for line 4 today and there’s nothing significant mentioned in it.

Here are the engineer’s notes as sent to AA and included below, taken from the clueless log for line 2:


Wednesday 11:21:16      Note LTOK Work Point: JB26   aa@a
Monday 11:37:13   Wednesday 10:15:05   ToDo Check Line since an SNR Reset was run   aa@a
Tuesday 16:10:53   Wednesday 10:14:57   BT Fault 5-894563588028 Please accept clear if fault resolved. If rejected an SFI appointment will be offered.
Local access network between Exchange and End User;Fault rectified in underground network   aa@a
Wednesday 00:56:13      SMS to janet: Delivered to handset 2022-07-20 00:56:10   SMS
Wednesday 00:56:11      Sent KCI sms janet 2022-07-19 23:08:40 DSL line up (down 80 seconds)   KCI
Tuesday 23:08:44      Deferred KCI sms 2022-07-19 23:08:40 DSL line up (down 80 seconds)   KCI
Tuesday 23:08:43      Tx rate (adjusted) 2092104 to 2338182 (rx 512000)   -Auto-
Tuesday 23:07:26      Cancelled KCI sms 2022-07-19 23:07:20 Line down (LostCarrier) [cancelled deferred KCI]   KCI
Tuesday 23:06:12   Tuesday 23:06:51   Test SNR reset: RateBandDS=160-24384 InterleaveLevelDS=On TargetMarginDS=3dB RateBandUS=32-Uncapped InterleaveLevelUS=On TargetMarginUS=6dB:SNR Recalculation initiated which will enable the line to reach optimum speed, stabilization retrains can be expected over the next couple of days.   weaver@a
Tuesday 23:06:07      Update Update RateBandDS=160-24384.6dB->160-24384.3dB   weaver@a
Tuesday 15:37:15   Tuesday 16:10:53   BT Fault 5-894563588028 Notes as it is received from OR
What did the end customer describe as the primary issue ?: 4012.04-Customer Report: End customer advised there is a broadband issue.
Specifically where did you resolve the fault ?: 1619.03-\n\n Actions to resolve: Engineer has resolved the fault located at the D-side including aerial cables / lead-in / block terminal.
What were the visible signs that caused the fault ?: 1613.36-The fault was located outside the end customer's curtilage and shown by damp / corrosion to wire / cable.
What did you do to fix the fault ?: 1614.33-The fault was fixed by clearing in joint.
Where / how did you perform the final Eclipse / FastTest / xDSL ?: 3839.03-\n\n Final alternate test results: Final FastTest performed from the customer premise.
What was the result of the final Eclipse / FastTest / xDSL ?: 3853.01-The test passed on 19/07/2022 15:30:15.   BT
Tuesday 15:37:00   Tuesday 15:37:15   BT Fault 5-894563588028 Notes as it is received from OR
What did the end customer describe as the primary issue ?: 4012.04-Customer Report: End customer advised there is a broadband issue.
Specifically where did you resolve the fault ?: 1619.03-\n\n Actions to resolve: Engineer has resolved the fault located at the D-side including aerial cables / lead-in / block terminal.
What were the visible signs that caused the fault ?: 1613.36-The fault was located outside the end customer's curtilage and shown by damp / corrosion to wire / cable.
What did you do to fix the fault ?: 1614.33-The fault was fixed by clearing in joint.
Where / how did you perform the final Eclipse / FastTest / xDSL ?: 3839.03-\n\n Final alternate test results: Final FastTest performed from the customer premise.
What was the result of the final Eclipse / FastTest / xDSL ?: 3853.01-The test passed on 19/07/2022 15:30:15.   BT
Tuesday 10:29:42   Tuesday 15:37:00   BT Fault 5-894563588028 The fault has reached PONR, amend/cancel currently can not be accepted   BT


Modem 2 SNRM-vs-tones:




Modem 4 SNRM-vs-tones:

[burakkucat]

Update, I reported lines 2 and 4 to AA on Monday morning, OR engineer arrived Tuesday morning, booked for line 2 (only) pretty impressed by that. Both lines sorted straight away. Engineer, among other things, went up the pole by the roadside below my house - that trouble spot again. It seems that line 4 was fixed by OR magic, of the kind that has fixed faults at many a callout in the past, yet we have never understood how.


        ===========   F  I  X  E  D  !!   ==================


I told AA that both lines were looking splendid, apart from a slight loss of speed 50-100 kbps, so both tickets were to be closed. I noticed that the next morning AA added a note in clueless “Note LTOK Work Point: JB26”. I know what a JB26 is but I don’t understand how that note got put there as I can’t see it in the engineer’s notes.

So you now have three working pairs, once again. Excellent. 

As for the "Note LTOK Work Point: JB26" perhaps A&A can enlighten you?

At the pole top, there is a BT66 in which the pairs from the two aerial drops are joined to a multipair cable. (Five pair? or ten pair?) That multipair cable then descends the pole and, at ground level, enters a joint closure (A standard, clip, joint closure (maybe)? What I describe as a "giant's thimble".) on one of the larger multipair cables (running from NSBRD to Heasta properties further down the hill). So, at the pole, there are two joints in each pair (separated by the height of the pole) on the way to Torr Gorm where water ingress could cause corrosion.

[burakkucat]

A further thought on the enigmatic: "Note LTOK Work Point: JB26" phrase.

A very reliable source has just sent me a link to a supplier for a "BT Joint Box 26". I can't imagine one of them being anywhere out on the high moor or at the foot of the lower (roadway) pole.

However there might well be one of those joint boxes either on the A87 before the turn onto the road to Heasta or on the early part of the Heasta road, by those last few houses at Harrapul, before the start of the moor. If that is the case, perhaps it was the last point of easy access to the pairs before the lower (roadway) pole is reached and, at that point, "LTOK" - only becoming "LTNOK" in your office.

[Weaver]

I did ask A&A what LTOK means and they simply said "line test OK". I’ve just remembered - line 2 is my "fourth line" concerning which there was some fuss with OR trying to find additional E-side pairs (was it?) and something involving the green cab by the Claymore.

[burakkucat]

I’ve just remembered - line 2 is my "fourth line" concerning which there was some fuss with OR trying to find additional E-side pairs (was it?) and something involving the green cab by the Claymore.

It was probably the E-side. I recall that there needed to be some rearrangement for the provision of the fourth circuit but it was not clear exactly where the rearrangement was made.

[Weaver]

That didn’t last long. The so-called ‘repair’ failed on Sat afternoon. Line up-down, huge packet loss. Have let AA know.