Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

Amazing development: after almost exactly 24 hours (hmm) line 2 has fixed itself. The HCD has gone away! The slightest trace of it may be detectable to the keen eye, but it doesn’t satisfy the criterion required by my summary program to be reported as HCD. Its ’depth’ if any is probably less than 1 dB, judged by eye, so that is counted as ‘in the noise’ and not deemed suitable for a reliable warning report.

[burakkucat]

How odd. Could it be something environmental is taking place? (b*cat is puzzled.) 

[Weaver]

I just can’t think what a candidate cause might be. If it had started at exactly 11:00 then that would be very interesting, but iirc it was around 10:55. And it was very close to 24 hours’ duration. I’m befuddled here; could it have been a limited-band interference blast lasting 24 hours? And just covering those tones roughly 45-80 or probably a bit narrower than that.

I have now finally done what I’ve been wanting to do for ages; I’ve moved the code around in my modem wellness summary program and reordered the HCD per-chord tests so that they are now in the order: test chord 1, then chord 2, then chord 3. I immediately introduced a bad bug, false error report with chord 1. This was because when trying to move statements around by dragging, the Shortcuts code editor dropped a moved statement in completely the wrong place, and the editor seems to allow variables to be used before values are assigned to them. So simply moving the variable definition down to where it should be fixed the bug. The computed value of the y coordinate of the (approx) midpoint or test-point of the chord 1 diagonal line was very obviously wrong, and it turned out that this was caused by the use of the undefined variable, the x-coordinate of the midpoint, in the equation of a straight line, whose value was I think defaulting to zero.


Janet has just pointed out something very significant, which I had forgotten about. The line went down at Fri 10:55:22 and (as best I can tell, because there was a lot of chaos in the log) came back up at Sat 10:56:33. So pretty close to 24 hours. But that up-time was during the time period the OR engineer was investigating line 4. So that is suspicious. Mind you, he was supposed to be working on line 4, not line 2.

[burakkucat]

Janet has just pointed out something very significant, which I had forgotten about. The line went down at Fri 10:55:22 and (as best I can tell, because there was a lot of chaos in the log) came back up at Sat 10:56:33. So pretty close to 24 hours. But that up-time was during the time period the OR engineer was investigating line 4. So that is suspicious. Mind you, he was supposed to be working on line 4, not line 2.

Ah, ha! That is interesting. Assuming all four of your lines are in the same cable, which is very likely to be the case, opening a joint closure and poking the contents with an Openreach magic-wand could perturb all of the pairs in the cable. (Now if the joint being worked upon was back up on the high moor and the engineer left the joint in an open state, whilst getting some gubbins from the back of his van, an evil haggis could have pee'd in the joint without being noticed.)

[Weaver]

Janet told me that the engineer just tested line 4, told us both the results and left without going out onto the high moor to fix anything. This was because it looked all good, having got mysteriously fixed by who knows what mechanism as has often been observed, and because her was only booked to do ‘telephony’-type work, and told Janet something about a menu of job prices for different activities, something which I couldn’t hear. He said that he had been told that the line had crackles on it, and put this ‘end-user reported crackles’ in his notes that were sent back to AA. This was an outright lie from someone; I certainly never said any such thing, not even having a phone plugged in. One of my favourite sayings: “Mas breug bhuam, bu bhreug chugam e” - if it is a lie from me, it was a lie to me. I don’t like the fact that someone is saying that we said things that we did not. Very odd.

[Weaver]

Line 2 current picture:



The line 2 HCD depth or ‘drop’ has now increased to 2.16 dB. Still not enough to cause any major problems, but the error rate is now non-zero.

       Summary of DSL links’ wellbeing and error counts
     ────────────────────────

* There are 4 modems in total. They are:     #1, #2, #3 and #4
* The active, contactable modems are:        #1, #2, #3 and #4
* The modems successfully queried are:       #1, #2, #3 and #4

       ───────────────────────
       ***                                                                           ***
       ***     There is some SERIOUS BADNESS !          ***
       ***                      All is not well !   😦                       ***
       ***                                                                           ***
       ───────────────────────

* HCD:❗So-called ‘hollow curve disease’ (HCD) defect detected in the downstream SNR-vs-tones curve of link #2. Serious fault. 🔺
   Link #2: Summary: curve midpoint test tone 59 is 2.15938 dB below chord  1: (tones 40-80); ❗
   Link #2: Details: chord  1: [ tone 40: 35.0625 dB; chord midpoint test tone 59: 33.22188 dB; tone 80: 31.1875 dB ]; curve midpoint test tone 59: 31.0625 dB

--
* ES (not so serious): The following links have a few CRC errors, at lower error rates, where the ES rate < 60 ES / hr (†):
   Link #2: downstream:  most recent period: ES per hr: 32.00, mean time between errors:  112.5 s, collection duration: 450 s, SNRM: 2.399 dB
   Link #2: downstream:  penultimate period: ES per hr:  12.00, mean time between errors: 300 s, collection duration: 900 s, SNRM: 2.399 dB
   
                                     ◅ ◅ ◅◊▻ ▻ ▻

[Weaver]

Latest analysis. The error rate has really picked up, so I will have to adjust the downstream SNRM now. This is before any such adjustment.

Code: [Select]

       Summary of DSL links’ wellbeing and error counts
     ────────────────────────

* There are 4 modems in total. They are:  #1, #2, #3 and #4
* The active, contactable modems are:      #1, #2, #3 and #4
* The modems successfully queried are:    #1, #2, #3 and #4

       ───────────────────────
       ***                                                                           ***
       ***     There is some SERIOUS BADNESS !          ***
       ***                      All is not well !   😦                       ***
       ***                                                                           ***
       ───────────────────────

* HCD:❗So-called ‘hollow curve disease’ (HCD) defect detected in the downstream SNR-vs-tones curve of link #2. Serious fault. 🔺
   Link #2: Summary: curve midpoint test tone 59 is 1.78716 dB below chord 1: (tones 43-80); ❗
   Link #2: Details: chord 1: [ tone 43: 35.3125 dB; chord midpoint test tone 59: 33.47466 dB; tone 80: 31.0625 dB ]; curve midpoint test tone 59: 31.6875 dB

--
*❗ES (serious): Links with CRC errors at higher error rates, where the ES rate ≥ 60 ES / hr (†):
   Link #2: downstream: most recent period: ES per hr: 65.78, mean time between errors: 54.73 s, collection duration: 602 s, SNRM: 2.399 dB
   Link #2: downstream: penultimate period: ES per hr: 64.00, mean time between errors: 56.25 s, collection duration: 900 s, SNRM: 2.399 dB

--
* ES (not so serious): The following link has a few CRC errors, at a lower error rate, where the ES rate < 60 ES / hr (†):
   Link #1: upstream:       penultimate period: ES per hr: 48.00, mean time between errors: 75 s, collection duration: 900 s, SNRM: 5.799 dB

────────────────────────────────────
(†) The duration of a most ‘recent’ errored seconds (ES) collection bucket is
variable, with a _maximum_ of 15 mins. The ‘most recent’ buckets’ start times
are always 15 mins apart. A ‘penultimate’ bucket is a fixed 15 mins. An ES is a 1 s
time period during which corrupt data is received by a DSL modem that not be
recovered by the error correction methods of the two DSL modems. (This is not
to be confused with the error recovery method used in ‘transport layer’ software
such as QUIC, TCP etc that uses whole-path, end-to-end packet retransmission.
Here we are only concerned with error recovery over one single DSL link.)
────────────────────────────────────
                                     ◅ ◅ ◅◊▻ ▻ ▻


[Weaver]

Here’s a recent picture of line 2, Thursday morning 2022-09-08T09:20:




And then, an amazing spontaneous recovery, Thursday afternoon 2022-09-08 when there was a period of instability, with many retrains including one where the downstream sync rate dropped quite low, maybe to something like 1.4 Mbps. Then the downstream sync rate came right back up to roughly where it should have been and the HCD was completely gone.



This is the summary of wellness of all lines after I put the line 2 downstream target SNRM back to 3 dB. It isn’t happy with that target at all as you see. So I shall go back to line 2 6 dB downstream target SNRM, and adjust config expectation settings accordingly. But otherwise all is good.

Code: [Select]

       Summary of DSL links’ wellbeing and error counts
     ────────────────────────

* There are 4 modems in total. They are:  #1, #2, #3 and #4
* The active, contactable modems are:      #1, #2, #3 and #4
* The modems successfully queried are:    #1, #2, #3 and #4

       ───────────────────────
       ***                                                                           ***
       ***     There is some SERIOUS BADNESS !          ***
       ***                      All is not well !   😦                       ***
       ***                                                                           ***
       ───────────────────────

*❗ES (serious): Links with CRC errors at higher error rates, where the ES rate ≥ 60 ES / hr (†):
   Link #2: downstream:  most recent period: ES per hr: 60.10, mean time between errors: 59.9 s, collection duration: 599 s, SNRM: 3.10 dB

────────────────────────────────────
(†) The duration of a ‘most recent’ errored seconds (ES) collection bucket is
variable, with a _maximum_ of 15 mins. The ‘most recent’ buckets’ start times
are always 15 mins apart, and the end time is now. A ‘penultimate’ bucket is a
fixed 15 mins long. An ES is a 1 s time period in which a DSL modem receives
corrupt data that can not be recovered by the two modems’ error correction
capabilities. (This is not to be confused with the whole-path, end-to-end packet
retransmission-based error correction used in ‘transport layer’ software such as
QUIC, TCP etc. Here we are only concerned with data corruption on a single,
direct, modem-to-modem DSL link, and its correction by modems.)
────────────────────────────────────
                                     ◅ ◅ ◅◊▻ ▻ ▻


Here are the full stats for all lines just now:

Code: [Select]

                    Status of each link
                    ──────────

* There are 4 modems in total. They are:  #1, #2, #3 and #4
* The active, contactable modems are:      #1, #2, #3 and #4
* The modems successfully queried are:    #1, #2, #3 and #4


   Link #1: sync rate: downstream: 2.641 Mbps at SNRM 2.9 dB; upstream: 646 kbps at SNRM 5.8 dB
   Link #1: uptime: 15 days 20 hours 44 min 39 sec

--
   Link #2: sync rate: downstream: 2.684 Mbps at SNRM 3.1 dB; upstream: 515 kbps at SNRM 6.2 dB
   Link #2: uptime: 1 hour 3 min 27 sec

--
   Link #3: sync rate: downstream: 2.968 Mbps at SNRM 3.1 dB; upstream: 627 kbps at SNRM 5.6 dB
   Link #3: uptime: 13 days 10 hours 57 min 47 sec

--
   Link #4: sync rate: downstream: 2.741 Mbps at SNRM 3.1 dB; upstream: 355 kbps at SNRM 5.8 dB
   Link #4: uptime: 5 days 2 hours 27 min 27 sec


                                              ◅ ◅ ◅◊▻ ▻ ▻

[tubaman]

I suspect there is a less than perfect joint somewhere along the vastness of your lines and that weather or other variable conditions are affecting it.
Always a pain to find when intermittent like this.

[Weaver]

This latest plus, last Saturday’s are the only two so-far known instances of spontaneous recovery, without some kind of at least possible engineer intervention in the form of a visit, even though it is not known what action or effect the visits have had.

[Weaver]

Yesterday we should have had an SFI about HCD (900 kbps downstream) in line 2. No show, again, no phone call.

[Weaver]

Line 2 is bad again, and line 4 is showing the very faintest sign of HCD on the threshold of reliable detection.

There were two (!) Openreach no-shows last week for SFI appointments. That makes three or maybe four no-shows. AA has escalated it within BT and I’m leaving them to it. They are asking BT what’s up with the absent engineers and also, I hope, asking what the plan is to get my line 2 finally fixed for good. I’m once again thinking about getting an additional line and then ceasing this one.

Line 2 suddenly became perfect in the middle of the illness and then jumped straight back to badness after two days of being fine. So now it’s becoming more intermittent, which is a new development.

Down to 288 kbps downstream at 12 dB SNRM, upstream is fine.

FYI The new line line 3 is 3.048 Mbps downstream sync @ 3dB target SNRM and literally zero ES/CRCs, and the upstream is good at 630 kbps @ 6dB. So it shows you what it could be like.