Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

The code looks quite encouraging now. A bit more robust. As Burakkucat mentioned, I keep getting drawn to differentiate the whole thing n times, where n is (at least) 2, but

• I have avoided any loops so far, which is good, and
• I must remember that we’re dealing with finite differences here, not smooth curves - the latter are just pretty pictures, secondary to the real integer discrete data points, and
• the noise in the data is very loud and unpredictable and will completely mess up attempts involving calculus, and
• methods of filtering the noise out have been put to use all ready but I don’t think they’re particularly sophisticated yet somehow.

There’s a mixture of quantisation and other low-level noise and occasional serious spikes, including radio stations potentially, and the pilot tone, which is a killer, has already been discussed and can’t be included in a system involving differencing.

[burakkucat]

I have once again altered the core algorithm, now using algorithm II, combined with general sharp dip detection (not just specific to the pilot tone) at the ‘mid’ test point. The x==40 test point has a specific test for being a pilot tone.

    y40 = ( y[40] <= 2 ) ? y[41] : y[40];
    y60_is_bogus = (y[60] < y[60 + 2] - 2) && ( y[60] < y[60 - 2] - 2);
    y6x = y60_is_bogus ? y[62] : y[60];
    ymid = y40 - 1 + (y85 - y40)*(60 - 40.0)/(85.0 - 40.0);
    hollow_curve = ( y6x < ymid );

That certainly looks simpler and can be easily understood.

I keep going back to your example of bad and example of good . . . Recently I looked at the former and asked myself "How do I know it shows the ailment?" The answer is that we visually take in the shape of the bulk image and mentally compare it to what we have seen in the past. No precise mathematical transformations are required . . .

Suppose we consider only part of the plot, the domain where

120 > X > 32

Now consider the "stripes", first

1 >= Y >= 0

then

2 >= Y >=1

then

3 >= Y >= 2

until the stripes contain no data. Would that give a fingerprint which could be compared with that archived from "bad" and "good" examples? I'm not sure. 

And now my brain is starting to hurt. 

[Weaver]

You’re quite right. But now my brain hurts. The right hand portion of the diagram is generally fallling, tilting downwards. You are slicing it horizontally. isn’t that true? I’m looking at a right-down-tilting diagonal in my equation of a straight line, so a bit like your scheme apart from my tilt.

[burakkucat]

I really think you need some data on which to test the various algorithms. And now I am thinking how I could recreate the data set which, when plotted, produces your example of bad.

[Weaver]

I have found some more old examples, but they all predate the occurrence of bad. I’ll have to pray for the arrival of more badness. I just wish I had saved all the many recent data snapshots of bad that I have seen but not recorded.

I’ve also find a lot of old data from the DLink modem, but it was ‘being good’ though, no danger of hollow curve. The shape of the bitloading distribution is very different, much more jagged at the top, very bad for my code which wants a smooth sugar-loaf mountain (is it?) shaped distribution without to many jumps in it. My code cannot talk to a DLink, luckily, because the code requires the availability of Johnson firmware and its "Easy Stats"-HTTP protocol, as I have named it, so fortunately for me it will never be presented with the embarrassing DLink. I have the data if anyone is interested.

[Weaver]

asked myself "How do I know it shows the ailment?" The answer is that we visually take in the shape of the bulk image and mentally compare it to what we have seen in the past. No precise mathematical transformations are required…

I am using that philosophy and I reckon that the answer is in the core of the new Algorithm II - the droop below a straight line as defined by the straight line equation, cutting a chord across the top of a good distribution, but in the case of a ‘bad’ distribution, the ‘midpoint’ (approx) is hanging down below the straight line, a ‘sag’. And the vertical distance of that droop is measured in the code above.

When I was a small boy, I visited a farm across the valley, ‘Thorn Top’, and sat in the kitchen. While the adults were talking, I was miles away, as ever, staring at a lead pipe on the kitchen wall. The pipe had support clips every so far along it, but in one place the lead had flowed and sagged into a deep dip, miles below the horizontal straight line. I thought this thing was strange and wondrous. My first real-life meeting with the hollow curve?

Anyway, that’s my attempt at turning the feeling about the character of the shape into a precise mathematical test, only reliable though after downward spikes and quantisation noise have been removed as far as possible, something that is not at all easy and is the part I’m not confident about. I think I mentioned some while back that the usual filters to remove short-wavelength glitches could be very problematic; need to make that more precise, but I’m too tired just now.

[burakkucat]

I am using that philosophy and I reckon that the answer is in the core of the new Algorithm II - the droop below a straight line as defined by the straight line equation, cutting a chord across the top of a good distribution, but in the case of a ‘bad’ distribution, the ‘midpoint’ (approx) is hanging down below the straight line, a ‘sag’. And the vertical distance of that droop is measured in the code above.

Yes, you have expressed it, in words, quite nicely.

When I was a small boy, I visited a farm across the valley, ‘Thorn Top’, and sat in the kitchen. While the adults were talking, I was miles away, as ever, staring at a lead pipe on the kitchen wall. The pipe had support clips every so far along it, but in one place the lead had flowed and sagged into a deep dip, miles below the horizontal straight line. I thought this thing was strange and wondrous.

I, too, have memories, from sixty plus years ago, of a horizontal lead pipe with sags between each support. 

[Weaver]

God only knows what all that lead did to them. Doesn’t lead form an insulating barrier compound ?an oxide? which prevents further dissolving into the water? I’m trying to remember Mrs Johnson’s chemistry lessons from 1976. Luckily we had none in our farmhouse.

Referring to the line
    y6x = y60_is_bogus ? y[62] : y[60];
taking the "? :" expression apart and turning it into an if-statement, we are not fully examining the conditions surrounding our new sideways-x-shifted choice of x=62; is there any problem with that? laxity?

[burakkucat]

God only knows what all that lead did to them. Doesn’t lead form an insulating barrier compound ?an oxide? which prevents further dissolving into the water?

Old lead pipes in a hard water area are quite safe, as the dissolved minerals will have created a passive barrier by precipitation. Soft water or slightly acidic water will rapidly dissolve lead and that was the problem.

Referring to the line
    y6x = y60_is_bogus ? y[62] : y[60];
taking the "? :" expression apart and turning it into an if-statement, we are not fully examining the conditions surrounding our new sideways-x-shifted choice of x=62; is there any problem with that? laxity?

I see

if (y60_is_bogus)
        y6x = y[62];
else
        y6x = y[60];

where y60_is_bogus is a boolean.

[Weaver]

Agreed about the if-statement. I was wondering if there are potential problems due to not applying the full double-sided test to the y[62] - that is testing y[60] - which is already done - and y[64] which isn’t. I am a bit lost there. I note that this is just a one-off rightward x-shift x+=2 and an if-statement; this could evolve into a loop, instead of a one-off; examining data points and shifting right until we find a good point.

[burakkucat]

I see what you mean. Would it really be necessary? Hmm . . . 

How about just enhancing the pseudo code to see how complex it becomes?

[Weaver]

Agreed, I don’t think it would be necessary, I don’t propose to fix the above code where it isn’t broken.

I wish I could find a picture and dataset of the earliest stages of the disease. There the first stage is a very subtle dent whose rhs is _horizontal_, so the two sides of the hollow are not symmetrical in the earliest stages. This is the reason for the ‘diagonal line’ (potentially) y=mx+c technology employed, because I feel it crucial to detect the earliest warning stages of the disease so as to warn the user soon. In truth though, the app will probably sound an alarm early because a check on the minimum downstream sync rate will fail as the highest bitloadings decrease in the formation of the hollow. I have now set the app’s minimum d/s sync rates to be a tight constraint.

So draw a chord, slightly sloping downward to the right, small in length, just chopping off the peak and reflect the curve in the line so that it is below the line instead of above it. That’s another way of picturing stage one of the disease. In this stage one or two bit high noise will really mess up detection. Our brains have neural network technology literally which can recognise this departure from the expected rounded hump shape.


I could generate some fake data using a spreadsheet. I need to write a quadratic or cubic polynomial to take care of drawing the ‘bite out of’ the good curve, below the ‘chord’ that is tilted in the decreasing y with increasing x direction.

[Weaver]

Here’s a presumably ‘good’ picture, no hollow curve. Modem 4.



If I draw a diagonal straight line from (x=40, y=12) -> (x=85, y=10), and then I inspect the (x=60, y), you see that I’m in danger of hitting that small dip down to y=10 at (x=~58?, y=10) approx. with the diagonal.

Both fortunately and unfortunately, depending on how you look at it, I don’t have any hollow curve data. If I do see an iffy picture, I can immediately save the bitloading numbers by fetching the file : http://modem:8000/data/Bits.

Code: [Select]

xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 321 Kbps, Downstream rate = 3304 Kbps
Bearer: 0, Upstream rate = 396 Kbps, Downstream rate = 3154 Kbps

Tone number      Bit Allocation
   0 0
   1 0
   2 0
   3 0
   4 0
   5 0
   6 0
   7 8
   8 9
   9 9
   10 10
   11 9
   12 9
   13 9
   14 8
   15 7
   16 7
   17 7
   18 6
   19 6
   20 5
   21 5
   22 5
   23 4
   24 4
   25 4
   26 4
   27 4
   28 3
   29 3
   30 3
   31 3
   32 0
   33 8
   34 9
   35 10
   36 11
   37 11
   38 11
   39 12
   40 12
   41 12
   42 12
   43 12
   44 12
   45 12
   46 7
   47 12
   48 2
   49 12
   50 12
   51 11
   52 11
   53 11
   54 11
   55 11
   56 11
   57 11
   58 10
   59 10
   60 11
   61 11
   62 12
   63 12
   64 12
   65 12
   66 12
   67 11
   68 12
   69 11
   70 11
   71 11
   72 11
   73 11
   74 11
   75 11
   76 11
   77 11
   78 11
   79 10
   80 10
   81 10
   82 10
   83 10
   84 10
   85 10
   86 10
   87 9
   88 9
   89 9
   90 9
   91 9
   92 9
   93 9
   94 9
   95 9
   96 8
   97 8
   98 8
   99 8
   100 8
   101 8
   102 8
   103 7
   104 7
   105 7
   106 7
   107 7
   108 7
   109 7
   110 7
   111 7
   112 6
   113 6
   114 6
   115 6
   116 6
   117 6
   118 6
   119 5
   120 4
   121 3
   122 2
   123 4
   124 4
   125 3
   126 4
   127 3
   128 4
   129 3
   130 3
   131 4
   132 4
   133 3
   134 3
   135 3
   136 2
   137 3
   138 3
   139 3
   140 3
   141 3
   142 2
   143 3
   144 3
   145 3
   146 2
   147 2
   148 0
   149 1
   150 0
   151 1
   152 0
   153 1
   154 1
   155 0
   156 0
   157 0
   158 0
   159 0
   160 0
   161 0
   162 0
   163 0
   164 0
   165 0
   166 0
   167 0
   168 0
   169 0
   170 0
   171 0
   172 0
   173 0
   174 0
   175 0
   176 0
   177 0
   178 0
   179 0
   180 0
   181 0
   182 0
   183 0
   184 0
   185 0
   186 0
   187 0
   188 0
   189 0
   190 0

From x-46 to x=58 the y-quantisation says to me that y is firmly down at 11, or even less than half a bit lower, say 10.8 in places because there is presumably one point where y dips under y<=10.49. Does this departure from y=12 down to y=11 or even slightly less count as the merest hint of incipient hollow curve? We can’t draw any conclusions based on ‘curves’ of depth y+/- 0.5 as we’re right in the noise. And what are we to make of the solid y=12 region 62 <= x <= 66 ?

[burakkucat]

It's difficult to say when looking at that "step-wise" (rather low resolution) plot. 

I'm still fiddling with things to try and generate a bit-loading v sub-carrier index "hollow curve" plot for you to use. 

[Weaver]

I’ve now included the dataset, not just a pretty picture. Posts possibly crossing

For some reason line 1 is a bit slower than line 4. Line 4 is bonkers, don’t know why.