Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[burakkucat]

[b*cat has been to sleep since he last typed anything . . . ]

I've probably confused myself with the "mites" and "tites". I'm not sure what I meant in my last post.

As for using "<=" rather than "<", how about testing both by single-stepping through the code to see how they perform?

Your current "Line 2" I would state is normal. Your current "Line 4" I would state is abnormal, suspect, tending to show the start of a hollow curve. 

[Weaver]

I agree about the incipient hollow curve thing, but the hollow is in the ‘wrong’ place in terms of x-cord, too far to the left compared with earlier incipient faults and as I said the sync rate is hardly such that I can say it’s already missing a few bits from a top-class bitloading. If I were to make a change to detect shapes like this, it might need to be a second complete test, with a different diagonal line. That’s not a bad thing though, it’s just a pain because I can’t easily make this into a subroutine for more than one reason. I’m also worried about introducing false positives by detecting noise as hollow curves. I feel I might have to make the detector oversensitive to get it to detect this, if you follow me. And I feel that any such false reports would be a very bad failing. As you can see I’m trying to talk myself into not doing it. And I’ll admit that I’m also lazy and it would be a lot of work, which you wouldn’t have in a better programming framework where it was easier to implement subroutines. (Can create subroutines and pass parameters, but sharing data is a bit awkward, and here the second killer is that I’m not really using array indexing as in the pseudo code. I’m extracting a per-tone ‘y’ data point value by using a regex instead to get the line. Doing that with literal constant index values is easy, but doing so with variable indices is a bit more of a fiddle, would have to inject a variable’s value into the regex, a bit awkward. Doing anything with iOS Shortcuts is massively harder than with C, but in some areas Shortcuts has these incredibly powerful high-level library functions, so it isn’t all one-way.)

[Weaver]

My chemistry teacher, Mrs Johnson, used to say, "when the mites go up, the tights come down".

I’m thinking about the case of a candidate possible ‘stalactite’ exactly 2.0 dB high. That hits the == condition. I wanted your opinion on what we should classify such a feature as. If you follow me, it’s not really a debugging issue, more of a ‘choosing the spec issue’. `it’s hardly crucial; if something is exactly 2.0 dB high, I skip past it (ignore it) or don’t and move to the right, that is increase x and return the y of a new increased x instead, as you see from the pseudo-code. In my view it’s all about the significance of the particular height value 2.0, and in truth there isn’t any so it’s difficult to know what to do and since any hit has to be an exact one on the y-difference value 2.0, then we ask ourselves what do we do if the y-difference were 1.999 say and that counts as pretty much exactly the same thing as the number 2.0 has no special significance. I’m inclined to leave it alone, but I do wonder if I would be missing anything by passing over the question?


Latest picture of line 4. Has this changed? This has to be an early HCD, but the test point at x==60 is to far to the right to pick it up. Can change that, of course. Or have two such test points. I’m wondering if this is just a slightly different type of early HCD shape to the ones that I’ve seen before?

[burakkucat]

Understood. 

Latest picture of line 4. Has this changed? This has to be an early HCD, but the test point at x==60 is to far to the right to pick it up. Can change that, of course. Or have two such test points. I’m wondering if this is just a slightly different type of early HCD shape to the ones that I’ve seen before?

All those plots that we have previously seen with a "hollow curve" have never been critically analysed to extract the coordinates of the significant points. As such, I don't think I can comment on your last postulation.

[Weaver]

Agreed.

The reasoning behind that straight line equation algorithm is that a chord is already very slightly ‘hollow’, that is a real hollow curve must be lower at its midpoint than a chord is. If we knew the true equation of the expected ideal SNR curve then we could measure a downward deviation from the expected curve and that counts as accurate hollowness. So the algorithm is good imho, but we don’t know the coordinates of the chord endpoints to use; they were just obtained by observation of an extremely limited range of data. The best point for measurement of the downward deviation is always the midpoint of the chord assuming that the chord is well-chosen. In this latest case the midpoint is in the wrong place and therefore the rhs is in the wrong place too. The midpoint/test point is too far to the right.

Using the current chord and test point, the ‘drop’ at the midpoint, ie the y deviation below the test point is ~0.9 dB (very approx) by my calculations. I would say that’s too small to allow reliable detection because it could be in the noise.

[burakkucat]

Ah, I now see the logic behind your technique. Thank you for that explanation.

I keep coming back to the idea of looking for a local maximum, followed by a local minimum, followed by a local maximum. (Which, of course, implies differentiation . . .  )

[Weaver]

You’re right of course about the local maxima, minima. The noise in the data would I think scupper any attempt, because the derivative values would be messy, would need a smoothing function to be applied first then. One reason for my hesitation would be the difficulty of implementation in iOS Shortcuts, as many-times loops are a bit slow and the interpreter engine is painfully slow now.

I am thinking about testing the chord from (40, y_lhs) to (60, y_rhs) and then measuring the ‘drop’ at the midpoint. That would be quite cheap as it would require only one additional spike-filtered test. The condition would be as usual, and then the final boolean would be an OR of the two existing boolean < tests.

Here’s the detailed data set from the problematic region of line 4, showing our candidate small HC. The third column and the chart on the right show the deviations from the value of y at x==40. You can see the shape of the hollow in that nicely. It’s just that its dip is in the wrong place for the current algorithm’s parametrisation.



See also attached zip file containing spreadsheet in Excel format and in iOS Numbers format

[Weaver]

A small note. I believe that theoretically you could have a ‘lopsided’ early / incipient HCD curve. This would be one tilted down to the right, so that the rhs is monotonically decreasing, but has a point where dy/dx is instantaneously 0, ie not a local extremum, but a turning point, is that the term? I forget after forty years. Of course since it’s diseased, the first derivative does not have to be continuous, but could go sharply from zero to a new negative value at our point.

[burakkucat]

Hmm . . . If I am remembering correctly (from over 50 years ago), the critical (or turning) points of a curve are those points when the first differential equals zero.

Nothing else (currently) comes to mind.

[Weaver]

Agreed.

The modified algorithm currently running is as follows. Has two chords and has an associated test for each one:
    /* ALGORITHM IV */
    const double SNR[ 256 ];
    alias y = SNR;
    const uint half_width = 3;
    const double min_stalactite_height = 2.0;
    const bool y40_is_bogus = y[40] < y[40 + half_width] - min_stalactite_height;
    const uint x4 = y40_is_bogus ? 40 + half_width : 40;
    const double y4 = y[x4];
   
    const bool y5_is_bogus = (y[52] < y[52 - half_width] - min_stalactite_height )
                          && (y[52] < y[52 + half_width] - min_stalactite_height );
    const uint x5 = y5_is_bogus ? 52 + half_width : 52;
    const double y5 = y[x5];

    const bool y60_is_bogus = (y[60] < y[60 - half_width] - min_stalactite_height )
                           && (y[60] < y[60 + half_width] - min_stalactite_height );
    const uint x6 = y60_is_bogus ? 60 + half_width :  60;
    const double y6 = y[x6];

    const bool y85_is_bogus = (y[85] < y[85 - half_width] - min_stalactite_height)
                           && (y[85] < y[85 + half_width] - min_stalactite_height);
    const uint x8 = y85_is_bogus ? 85 + half_width : 85;
    const double y8 = y[x8];

    /* line chord1 [(x4,y4)-(x8,y8)]; 'approx midpoint' is at (x6, ymid1); data point is at (x6, y6) */
    const double ymid1 = y4 + ((y8 - y4)*(x6 - x4))/(x8 - x4); /* fp div required */
    const bool hollow_curve1 = y6 < ymid1;
   
    /* line chord2 [(x4,y4)-(x6,y6)]; 'approx midpoint' is at (x5, ymid2); data point is at (x5, y5) */
    const double ymid2 = y4 + ((y6 - y4)*(x5 - x4))/(x6 - x4); /* fp div required */
    const bool hollow_curve2 = y5 < ymid2;

    const bool hollow_curve = hollow_curve1 || hollow_curve2;

Implementation note: (1) the boolean OR is not available in iOS Shortcuts, although there is an extension package that has an if-or function or something, haven’t investigated this yet. Anyway, because of this, I implemented the OR as a pair of if-statements, quite ugly.

(2) The brackets surrounding the division in the chord equations shouldn’t be there. This is a bug because the bracket triggers integer division, the numerator and denominator bring uint’s, and floating point division is required. I’m assuming that the multiplication gets done first and that converts the rhs op uint to double so then the result of the multiplication is double and then so is the division operation.

* The new algorithm works fine , now detects HCD for modem 4 (only), which of course is the correct answer.

[burakkucat]

* The new algorithm works fine , now detects HCD for modem 4 (only), which of course is the correct answer.

That's good to know. Congratulations. 

Let's see how things are reported over the coming days / weeks / months / etc.

[Weaver]

There is a mini stalactite developing slightly at x=58-59, quite deep now.


I am not sure about that, maybe random variation?

I powered the modems down for several hours overnight because of a lightning scare. When they came back up the notch - supposed incipient hollow curve was still exactly the same. By the chosen geometric definition of hollow curve, it definitely qualifies, but I’m not sure it qualifies as a disease that develops; could be a permanent quirk or permanent disease though. We will have to see if it does grow.


I added an additional parameter ‘chord2 notch min height’ set to 1.5 dB. This is subtracted from the drop height below the midpoint as in
      bool hollow_curve2 = y5 < ymid2 - chord2_notch_min_height;
This means that small notches are ignored when the parameter’s value is set large enough and the definition of ‘notch’ is that detected by chord2, that is the chord x=40 to x=60.

This worked perfectly, blocking the detection of small chord2 notches if they are regarded as not genuine ‘disease’, although this classification is questionable. I would say that notches are only bogus if they do not go on to develop into something larger.


After a resynch on Tue afternoon, it seems the chord2 notch phenomenon has gone away. Who knows how long for though. At least the flexible twin-chord test and the filter ‘chord2 notch min height’ parameter are available as a future flexible guard.

Current line 4, Tue afternoon after the notch went away:

Code: [Select]


   39 39.2500
   40 40.0625
   41 40.6250
   42 40.8125
   43 41.0625
   44 41.2500
   45 41.5000
   46 35.3125
   47 41.1875
   48 41.5625
   49 41.5000
   50 38.6250
   51 41.6250
   52 41.1250
   53 41.4375
   54 41.3125
   55 41.0000
   56 40.8750
   57 40.8125
   58 39.0625
   59 39.0000
   60 39.9375
   61 40.0000
   62 40.2500
   63 40.0625
   64 39.4375
   65 39.5000
   66 39.0625


[Weaver]

Lightning all over the western isles and the west coast last night. Port Rìgh electricity substation got struck by lightning at around 8:00 am-ish and the electricity supply was off when residents got up. I was watching the lightning warnings on the internet for half the night because I couldn’t sleep due to pain.

Lightning + ‘thundersnow[?]’ is forecast here for tomorrow (Thursday) with ‘yellow warnings’.

It seems to me that the downhill slope is more or less a straight line after x=65, and that is somewhat different to the impression that I initially had. Does anyone have any old ADSL SNR-vs-tones graph pictures at all, or the dataset?

[Weaver]

Final summary and up-to-date algorithm.

The algorithm currently running is as follows. Has two chords chord1 x=40 to x=85 and chord2 x=40 to x=60, and has an associated test for each one:
    /* ALGORITHM IV.1 */
    const double SNR[ 256 ];
    alias y = SNR;
    const uint half_width = 3;
    const double min_stalactite_height = 2.0;
    const double chord2_notch_min_height = 1.5; /* adjustment of sensitivity of ‘notch’-detector, detector for smaller hollow in chord2 x=40 to x=60 */

    const bool y40_is_bogus = y[40] < y[40 + half_width] - min_stalactite_height;
    const uint x4 = y40_is_bogus ? 40 + half_width : 40;
    const double y4 = y[x4];
   
    const bool y5_is_bogus = (y[52] < y[52 - half_width] - min_stalactite_height )
                          && (y[52] < y[52 + half_width] - min_stalactite_height );
    const uint x5 = y5_is_bogus ? 52 + half_width : 52;
    const double y5 = y[x5];

    const bool y60_is_bogus = (y[60] < y[60 - half_width] - min_stalactite_height )
                           && (y[60] < y[60 + half_width] - min_stalactite_height );
    const uint x6 = y60_is_bogus ? 60 + half_width :  60;
    const double y6 = y[x6];

    const bool y85_is_bogus = (y[85] < y[85 - half_width] - min_stalactite_height)
                           && (y[85] < y[85 + half_width] - min_stalactite_height);
    const uint x8 = y85_is_bogus ? 85 + half_width : 85;
    const double y8 = y[x8];

    /* line chord1 [(x4,y4)-(x8,y8)]; 'approx midpoint' is at (x6, ymid1); data point is at (x6, y6) */
    const double ymid1 = y4 + ((y8 - y4)*(x6 - x4))/(x8 - x4); /* fp div required */
    const bool hollow_curve1 = y6 < ymid1;
   
    /* line chord2 [(x4,y4)-(x6,y6)]; 'approx midpoint' is at (x5, ymid2); data point is at (x5, y5) */
    const double ymid2 = y4 + ((y6 - y4)*(x5 - x4))/(x6 - x4); /* fp div required */
    const bool hollow_curve2 = y5 < ymid2 - chord2_notch_min_height;

    const bool hollow_curve = hollow_curve1 || hollow_curve2;

As mentioned earlier, I added an additional parameter ‘chord2_notch_min_height’ set to 1.5 dB. This is subtracted from the drop height below the midpoint as in
      bool hollow_curve2 = y5 < ymid2 - chord2_notch_min_height;
This means that small ‘notches’ are ignored when the parameter’s value is set large enough and the definition of ‘notch’ is that detected by chord2, that is the chord x=40 to x=60.

This has worked perfectly, blocking the detection of small chord2 notches if they are regarded as not genuine ‘disease’, although this classification is questionable. I would say that ‘notches’ are only bogus if they do not go on to develop into something larger.

[burakkucat]

Final summary and up-to-date algorithm.

Looks good. And hopefully it also works well. 

As mentioned earlier, I added an additional parameter ‘chord2_notch_min_height’ set to 1.5 dB.

<snip>

This has worked perfectly, blocking the detection of small chord2 notches if they are regarded as not genuine ‘disease’, although this classification is questionable. I would say that ‘notches’ are only bogus if they do not go on to develop into something larger.

As for the latter, I agree with you. The bogus status can only be resolved with the aid of either a crystal ball or a time machine.