Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

The line 2 HCD has developed into a full blown thing as predicted. In the night there was a resynch event and after that the downstream sync rate was down to ~1.7 Mbps and the hollow curve was deep, so I missed most of the early development.



From this picture I have two thoughts. The (lhs_x=40, lhs_y=y40-1) y40 has dropped so much that the ymid is quite low, especially when you consider that I’m actually working on a lhs_y of y40-1 but luckily the hcd detection still works. The lhs_y could be set to be equal to the rhs_y in this case! That would give a nice high ymid with a deep drop to the curve at y[xmid]. But if it works then all good enough I suppose. The diagonal line thing is really meant to detect the early phase.

Secondly, looking at (x_rhs, y_rhs) it seems x_rhs could be best moved up from x_rhs=85 to 90.
—
       Summary of DSL links’ wellbeing and error counts
     ────────────────────────


* There are 3 modems in total. They are:     #1, #2 and #4
* The active, contactable modems are:        #1, #2 and #4
* The modems successfully queried are:       #1, #2 and #4


       ───────────────────────
       ***                                                                           ***
       ***     There is some SERIOUS BADNESS !          ***
       ***                      All is not well !   😦                       ***
       ***                                                                           ***
       ───────────────────────


--
* Sync rate: The following link has a really low downstream sync rate, below min:
   Link #2 downstream:    current sync rate  1749 kbps is below minimum expected (2900 kbps)  ❗Line #2 fault  🔺

   ❗Link #2 defect detected: so-called ‘hollow curve phenomenon’ in the downstream bit-loading 🔺
--
*❗ES (more serious): Links with CRC errors at higher error rates, where the ES rate ≥ 60 ES / hr (†):
   Link #1  upstream:             latest period: ES per hr: 68.57, mean time between errors: 52.5 s, collection duration: 210 s
   Link #2 downstream:        latest period: ES per hr: 205.71, mean time between errors:  17.5 s, collection duration: 245 s
   Link #2 downstream: 'previous' period: ES per hr: 244.00, mean time between errors:  14.75 s, collection duration: 900 s


──────────────────────────────
(†) The duration of the ’latest‘ errored seconds (ES) collection
bucket is variable, with a _maximum_ of 15 mins. The buckets’
start times are always 15 mins apart. A ‘previous’ bucket's
duration is a fixed 15 mins. An ES is a 1 s time period in which one
or more CRC errors are detected. A CRC error is a Reed-Solomon
coding-uncorrectable error, ie. corrupted data is received that
cannot be recovered.
──────────────────────────────      
                                              ◅ ◅ ◅◊▻ ▻ ▻

[burakkucat]

I have been giving this some more thought.

I wonder if you could arrange for your utility to look at data for both the bit loading and the SNR plots? As you have shown, the SNR plot will show the early stages of the development of the phenomenon and the bit loading plot will "ring the big alarm bell". 

[Weaver]

It would be easy to do both, but due to the difficulty of writing subroutines that share variables with each other - awkward but not impossible as all the variables you need could be written into a struct and then that could be passed in  as a parameter - I would probably just be better off duplicating all the code twice. Could put it in a two-times-called loop but that would be very awkward too in this case.

Makes me think, if I process the SNR data, do I even need the bitloading "bits" data at all? What do we think?

The pilot tone isn’t a consideration in the SNR graph, so it actually simplifies the code if I just change over to use the smooth curve SNR dataset! If I go that way, it would massively simplify the algorithm at https://forum.kitz.co.uk/index.php/topic,26716.msg449098.html#msg449098 down to just a couple of lines of code approx as the if-else tests for pilot tones would be gone completely. Makes me think I made a huge error in using the wrong dataset to begin with, no?

[burakkucat]

Makes me think, if I process the SNR data, do I even need the bitloading "bits" data at all? What do we think?

A good question. Hmm . . . 

The pilot tone isn’t a consideration in the SNR graph, so it actually simplifies the code if I just change over to use the smooth curve SNR dataset! If I go that way, it would massively simplify the algorithm at https://forum.kitz.co.uk/index.php/topic,26716.msg449098.html#msg449098 down to just one line of code approx as all the if-else tests before the final boolean assignment would be gone completely.

It would be helpful to see some psuedo-code, please.

Makes me think I made a huge error in using the wrong dataset to begin with, no?

But only with my encouragement.

[Weaver]

What’s your feeling about the SNR-only option?

I did a trial implementation of this which I will call algorithm III (forthcoming v shortly), and my first question is about the minimum noise stalactite height which is a "- 2" in "<y> - 2", would need to decide what the SNR curve equivalent height is. The result with a bad guess at the answer to this question worked really well on the current ‘fully-developed HCD’ dataset.

I will post up the pseudo code asap.

[burakkucat]

What’s your feeling about the SNR-only option?

I'm dithering (and thinking about finding my bed).

I did a trial implementation of this which I will call algorithm III (forthcoming v shortly), and my first question is about the minimum noise stalactite height which is a "- 2" in "<y> - 2", would need to decide what the SNR curve equivalent height is. The result with a bad guess at the answer to this question worked really well on the current ‘fully-developed HCD’ dataset.

This may be a silly suggestion but could you perform a first pass, looking to find the absolute minimum. Then, in a second pass, add or subtract the value found in first pass such that you will always work with a data set that has zero as its absolute minimum, thus all y-data points are then positive.

[Weaver]

Updated pseudo-code; current running version (old bits version archived)

ALGORITHM III.0.1 - applies now to the y[]=SNR[] table

    alias y = SNR;
    uint half_width = 3;
    double min_spike_height = 2.0;
    bool y60_is_bogus = (y[60] < y[60 + half_width] - min_spike_height )
                    && ( y[60] < y[60 - half_width] - min_spike_height );
    double y6x = y60_is_bogus ? y[60 + half_width] : y[60];
    double ymid = y[40] + (y[85] - y[40])*((60 - 40.0)/(85.0 - 40.0));
    bool hollow_curve = y6x < ymid;

Works well also even after the fault was cured by power-cycling modem 2, ie in the no-fault case gives the right answer of ‘all good’.

[burakkucat]

Has a spurious "4" found its way into the penultimate line?

double ymid = y4[40] + . . .

Other than that, it appears to make sense (without applying it to test data).

[Weaver]

typo, well spotted!

I’m not at all sure about the correct value of the min spike height parameter. This detects large spikes which cause the relevant test point to be shifted in the x dimension if it lands on a stalactite of a certain height or greater. Need to ignore these as they could be seen as bogus indicators of hollow curve disease.

I’m thinking that I’ve made a big mistake in simplifying this algorithm III as I have. I dumped the tests for pilot tones but now I suspect that I need a similar kind of test on the x==40 and x==85 points but to detect bogus stalactites not the pilot tone. Am I right in thinking this way ?

I’ve seen some stalactites on new healthy SNR curves that are dangerously close to the lhs and rhs x-coordinates, so if we go this way I’d have to apply the same kind of test as for x==60 currently. However I can’t do the double-sided test at x==40 as it’s way too close to the ramp up from nothing at the junction between upstream and downstream tones, so a single sided version with the y[40] < y[43] - 2.0 test would be the only test. And for x==x_rhs==85, I’m hoping that a double-sided test is overkill as that amount of code would be a bit of a pain in Shortcuts just from the point of view of finger-wear. What about the single-sided compromise there?

[burakkucat]

Looking at the real-world data for my own circuit, there is a rapid up-ramp from x == 33 to about x == 65 and a stalactite at about x == 47.

I'm not sure that you can absolutely depend upon the presence of a stalactite, for when looking at your example --



-- it really is quite clean.

[Weaver]

ALGORITHM III.1 - here’s the new currently running algorithm, with full bogus stalactite detection x 3; cheap stalactite detector by necessity at x==40 and cheap out of plain laziness at x==85. I should make some of the x values into variables rather than fixed numbers in the division below, but I don’t think the extra code is essential because the gradient error in the straight line expression is not going to be enormous.

    alias y = SNR;
    const uint half_width = 3;
    const double min_stalactite_height = 2.0;
    const bool y40_is_bogus = y[40] < y[40 + half_width] - min_stalactite_height;
    const double y4x = y40_is_bogus ? y[40 + half_width] : y[40];

    const bool y60_is_bogus = (y[60] < y[60 + half_width] - min_stalactite_height )
                    && ( y[60] < y[60 - half_width] - min_stalactite_height );
    const double y6x = y60_is_bogus ? y[60 + half_width] : y[60];

    const bool y85_is_bogus = y[85] < y[85 + half_width] - min_stalactite_height;
    const double y8x = y85_is_bogus ? y[85 + half_width] : y[85];

    const double ymid = y4x + (y8x - y4x)*((60 - 40.0)/(85.0 - 40.0));
    const bool hollow_curve = y6x < ymid;

[Weaver]

> I'm not sure that you can absolutely depend upon the presence of a stalactite, for when looking at your example --

I presume that you meant ‘absence’ rather than presence? Otherwise I’m being stupid and have misunderstood completely

[Weaver]

The modified algorithm is (III.2) - currently running:
    alias y = SNR;
    const uint half_width = 3;
    const double min_stalactite_height = 2.0;
    const bool y40_is_bogus = y[40] < y[40 + half_width] - min_stalactite_height;
    const uint x4x = y40_is_bogus ? 40 + half_width : 40;
    const double y4x = y[x4x];

    const bool y60_is_bogus = (y[60] < y[60 + half_width] - min_stalactite_height )
                    && ( y[60] < y[60 - half_width] - min_stalactite_height );
    const uint x6x = y60_is_bogus ? 60 + half_width :  60;
    const double y6x = y[x6x];

    const bool y85_is_bogus = y[85] < y[85 + half_width] - min_stalactite_height;
    const uint x8x = y85_is_bogus ? 85 + half_width : 85;
    const double y8x = y[x8x];

    const double ymid = y4x + (y8x - y4x)*((x6x - x4x)/(x8x - x4x));
    const bool hollow_curve = y6x < ymid;

[Weaver]

Here are some pictures of the very healthy lines 2 and 4. They feature some nasty little stalactites and I would appreciate some advice on who to deal with them. The min stalactite height parameter is set to 2.0 and this you will recall is used in the filter in algorithm III.2 which will cause the test point x-coord to shift ahead (increased x) to avoid the stalactite and move on to a known good new point, known good because it is at least 2.0 dB higher.

Line 4:




Line 2:

[Weaver]

A question, referring to the new algorithm III.2, I wonder if the "<" signs in the stalactite height tests should be "≤" instead? For example in
   y40_is_bogus = y[40] < y[40 + half_width] - min_stalactite_height;
and the other two similar tests.

Notice in the latest SNR graphs above the various stalactites, all big enough to mess up the HCD algorithm without anti-down-spike filters. Line 4 shows three stalactites, two of which are in the critical region of tones 40-90. There’s also an iffy-looking dent in the top, which looks a bit like a mini hcd, but the shape is strange. The detector algorithm does not flag this feature as incipient HCD, which is a questionable decision but on balance I think correct, because the downstream speed is an all-time record high, so how can you ask for even better performance out of the tones? The line 4 stalactite at tone 74 is actually the pilot tone, so a pilot tone does show up on an SNR curve, but it seems that it’s only sometimes.    On the other hand, line 2’s SNR vs tones graph doesn’t show the pilot tone at all.

Line 2 shows just one stalactite but in a potentially awkward place, approx tone 46, which could have hit the lhs detection point at tone 40 (or its secondary, shifted point x==40+3).