Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

I have been getting too many CRC errors on my line 1, upstream only. That was at the usual 6dB upstream target SNRM, so I increased the target SNRM to 9dB but I’m still getting a few errors. Hardly World War III and probably more of a mild case of stats-obsession.

Why upstream only? And why line 1 only? A target SNRM of 9dB upstream is the maximum that I can set, in the range of ADSL 2/2+-specific choices.

Is there anything I can do about it ?

[burakkucat]

Did you actually notice a degraded performance? Would you have known about those CRC errors if you didn't go looking?

Is there anything I can do about it ?

I don't think so. 

[Weaver]

No, hence the diagnosis of stats-obsession. I’m not sure about Zoom; that could be the killer, because it may not be using TCP. I don’t know for certain, but I believe some live streaming video apps don’t use TCP but rather use their own unreliable protocols because they want new data to always arrive on time, and to hell with any corrupt data in frames that will soon be out of date, and so worthless.

I used to get all kinds of horribleness in Zoom - see earlier threads. I went on a CRC + ES elimination rampage, even if it cost a small amount of speed here and there, having decided that I did not want to be having errors covered up by TCP. Now with my new wellness overview app (written in ~500 lines of iOS Shortcuts code), a ‘good’ verdict means zero ES on all modems in the last 30 mins.

So seeing as the Zoom badness has completely gone away now - no audio corruption any more - it maybe that my anti-ES campaign is the reason for the improvement. Of course, in this current particular case, since this is corruption of the upstream, it would be the audio/video of me as heard+seen by others that would be affected.

So it’s possible that my interest in zero ES could have some sane reasoning behind it.

I wonder if it would help if I force a resynch in the early hours when RF conditions are at their noisiest ? Then SNRM would rise from there in the daytime.

[burakkucat]

I wonder if it would help if I force a resynch in the early hours when RF conditions are at their noisiest ? Then SNRM would rise from there in the daytime.

Yes, you could certainly try that.

[Chrysalis]

How many errors are we talking about here?

Is zero ES a reasonable target for a ADSL line as long as yours?  Maybe consider only alerting if is a SES?

Resyncing during the noisiest period will probably get the most stable outcome in my opinion.

Dont think I have ever seen a DSL line without a single ES.

For reference on my VDSL with over 12db SNRM on the upstream and FEC enabled albeit weak FEC with no interleaving, it still registers some ES daily.

[Weaver]

Here’s the current report with the upstream target SNRM raised to 9 dB :

* ES (not so serious): The following link has a few CRC errors, at a lower error rate, where the ES rate < 60 ES / hr (†):
   Link #1  upstream:  latest period: ES per hr: 45.06, mean time between errors: 79.9 s, collection duration: 799 s

I report all non-zero ES counts in the latest 30-min (explanation for threshold of zero follows). That’s a low-significance warning. Serious error alarm is raised if ES count in latest or penultimate max-15 mins collection periods exceeds the equivalent when rescaled of 60 ES per hr. (This would be 15 ES per full 15 mins, but the latest collection bucket’s duration is variable as the collection start time is quantised on 15 min boundaries.)

I’m open to suggestions as to what this ‘serious ES count’ threshold should be. Please contribute.

I usually get zero ES for my other lines all the time, and for line 1 I’m getting zero on the downstream. There are a number of reasons for this. Firstly I have no neighbours nor any human electrical activity for the nearest and most downstream-attenuated 3.5 miles. So the line is eerily quiet apart from crosstalk and RF from long-range narrowband sources such as radio stations which are a very long way away. The second is that the line is potentially good in the sense that it has very thick copper according to Black Sheep, so modest attenuation for its extreme length. The most important thing of all is PhyR, which means that I can run at 3dB downstream target SNRM (as opposed to 6dB upstream). Without PhyR I would have to run at 9dB downstream to get low corruption. (One of the OR engineers, "the moaner", constantly goes on about how the line has been broken by running at 3dB downstream (or something similar and equally ill-formed) and then goes on to find joint faults and fixes them.)

When I first got an exchange upgrade to 21CN back in 2015, I discussed running at the newly available 3dB downstream. AA rightly not to expect this to be stable and we agreed to give it a try and see how it went. Back then I had no easy means of checking stats apart from remotely requesting a stats report via BT mechanisms initiated from clueless.aa.net.uk. I think that I was getting regular ES all the time downstream but they were covered up by TCP. I was also using a much inferior modem compared to my current superb ZyXEL, and that was the DLink DSL-320B-Z1. It hung on in there always despite any downstream errors though.

Then I upgraded to the current ZyXEL VMG 1312-B10A modems, thanks to Burakkucat’s recommendation amongst others, and the example of many other ZyXEL-loving Kitizens. Finally Mr Johnson, with generosity beyond words, wrote the custom firmware that I now enjoy, so I can now monitor modem stats, and here we are, brought up to the present.

According to a Broadcom document that I dug up, Broadcom developed PhyR in order to make IPTV 100% reliable, as it needs to be, without using TCP and without the 9 dB downstream target SNRM which would have further slowed down the then already slow ADSL2 speeds users were getting 15 years ago. So it does the intended job of approaching 100% reliability downstream for me. Shame that it isn’t available upstream too, as then I wouldn’t be getting the current minor upstream problem.

[Weaver]

It’s early days, but it seems that the wee small hours resynch idea has been a success.

An aside: I would like to also assess the error count over whole uptime that the modems report. I want to give far greater priority to recent history, but alerting the user to bad patches that have come and gone is important too, as the user just simply missed those bad periods by being unlucky and not running the wellness check tool at the right time. However, for some reason I often seem to get a presumably semi-bogus burst of downstream errors reported at the time of the DSL-link-reconnect. Have you seen that early error burst effect?

If I could work out a way of discarding that initial collection bucket’s data, I could subtract its contributions from the since-link-up grand total error counts. I just can’t see any way of doing it. The modem doesn’t record that ‘first collection bucket after link-up’ statistic. If I had a service process running continuously, monitoring the modem, then I could perhaps address this myself if I could somehow detect link-up ‘showtime’ events and then set a timer for a short ‘ignore errors’ time window. I’ve no idea how to do the resynch event detection though. A service process would require a port to C (or even D, yay!) and installation on a server such as my Raspberry Pi.

[burakkucat]

However, for some reason I often seem to get a presumably semi-bogus burst of downstream errors reported at the time of the DSL-link-reconnect. Have you seen that early error burst effect?

Yes, others have noticed CRCs or UASes (or both) following a forced re-train of an xDSL circuit.

[Chrysalis]

I meant the actual SES counter weaver, Broadcom modems have an actual SES counter weaver, so no need to program your own threshold.

Snippet of my stats.

Code: [Select]

Since Link time = 33 days 16 hours 47 min 13 sec
FEC:            0               1523
CRC:            2905            178
ES:             2312            177
SES:            0               0
See the SES ? last line.

Thats over 33 days and the lowest ES rates I have had on VDSL. 2300 may seem big but its under a 100 per day on average.  Alerting every time you get an ES, is going to be drawing attention to what is likely a non issue I think. 177 over 33 days on a 12db SNRM + FEC.

So my suggestion would be either only alert on SES, or alert on ES if its at least 300% above your average rate, instead of above zero.

Another potential idea is looking at the CRC to ES ratio, if its under 2:1 the line is likely very stable, whilst something like 20:1 would indicate a burst of noise.

[Weaver]

I’m aware of SES and have seen it in the stats, but I didn’t check its value in the code as in my experience it’s always zero for every line. These are really great suggestions, very helpful and very much welcomed.

I’m reporting low count ES as ‘minor problem’ so it’s not quite so guilty of drawing much attention as you say. The current ‘minor problem threshold’ is 60 ES per hours measured in any 15 min (max) ZyXEL most recent or penultimate collection bucket. More than that and it triggers a major problem state and that’s what I need help with choosing.

I understand the point about having the minor problem state reporting threshold set at zero ES. Non-zero ES is usual for me, as my lines are rather different than yours. I ought to make that ‘ignore’ threshold a configurable parameter, not just hard coded as zero.

> So my suggestion would be either only alert on SES, or alert on ES if its at least 300% above your average rate, instead of above zero.

I can’t do your second suggestion because I can’t get a decent average, because I’m not running continuously, and if I used the ‘total since showtime’ counter value (or whatever it’s called, ‘since link up time’ ?) then, as I think I mentioned earlier, this is polluted by the burst of bogus CRC events at the time when the link enters the showtime state. And I haven’t worked out how to filter out that initial burst yet.

Could you help me understand the rationale behind the CRC to ES ratio thing ? That is certainly a thing I am able to make use if. Is it because ES assesses the grouping of error events in any single one second window? That would allow me to make an assessment of the ‘type of badness’ going on currently, because as you say, it will detect noise bursts as opposed to continuous spread-out noise. This may be wrong, but I see it as a technique for characterising a problem state rather than detecting one. Is that unfair ?

About the count per-day units thing. You were only using that as a basis for quoting rate units, I understand. I wouldn’t (solely) average any of these rate values over long periods as a primary reporting measure, as I want to point out transient short-terms bad problems and not allow them to be averaged down to a lower total by long periods of good behaviour and quietness. But there absolutely has to be a use for long-term average tests, in a secondary role, with an if (badness_source_1 OR badness_source_2 test OR …)-type test. Or something, words fail me.

[Chrysalis]

My rationale is you have basic background noise on DSL, which would usually be fairly consistent in its pattern but at a low rate on healthy lines.  This I would expect to have a low CRC to ES ratio.  So when a second has at least one error, its usually only a very small amount of errors, with 1 been the minimum.

However if you get something like a burst of noise, from a faulty nearby device or something like a sudden shift of cross talk, I would expect in that situation to get a lot of CRC in one ES, which would more likely trigger visible instability.

Of course I am not a telecoms expert, and can only go on my own experiences and reports I seen on the internet, but thats a big part of my reasoning.  Whenever I have had instances of instability on my line, at the very least it will have a much higher CRC to ES ratio, and in very powerful bursts of noise, SES will accumulate.

I had another look at my stats after this post and ironically I had 2 SES earlier today. 

[Weaver]

Got it. Thanks, the penny has dropped. I wonder if, in the case where you have CRC errors grouped inside a second, more interleave depth would fix it. Remember though that since I have PhyR, most CRC errors get fixed by L2 retransmissions.

An aside: I notice that my downstream interleave depth tends to always be 1, even though I’ve set my preferences in clueless.aa.net.uk to ds_interleave = max (or what ever it is), so the modem seems to be determined to ignore that. And I suspect that that is something to do with PhyR. I wonder if it’s the same with G.INP - forces interleave to be off?

So to get an ES, maybe the link has to overwhelm PhyR ? Don’t know what the implications are here.

I’m going to start noting the ES / CRC ratio and looking out to see if I ever do get an SES.

[Chrysalis]

Yes g.inp isnt used alongside normal interleaving and I thought phyr itself is an alternative g.inp so would be the same?

Do you mind posting some of your error data please, I am curious how well phyr works for you on your long adsl line's.

Also yes, if you have some kind of error correction, the crc would be if that has been overwhelmed.

[Weaver]

Here’s an overall report from my wellness summary tool. Includes ES if any. Shows some problems that I didn’t know about and which I need to investigate. We had a nearby lightning storm today, so modems were turned off and then back on.

Code: [Select]

       Summary of DSL links’ wellbeing and error counts
     ────────────────────────

* There are 4 modems in total. They are:  #1, #2, #3 and #4
* The active, contactable modems are:      #1, #2 and #4
* The modem skip-list declared in config:  #3 was skipped over and not queried; its state is unknown
* The modems successfully queried are:    #1, #2 and #4

       ───────────────────────
       ***                                                                           
       ***     There is some SERIOUS BADNESS !          
       ***                      All is not well !   😦                       
       ***                                                                           
       ───────────────────────

*** Note that modem #3 was skipped over and not queried; its state is unknown. ***

--
* HCD:❗So-called ‘hollow curve disease’ (HCD) defect detected in the downstream SNR-vs-tones curve of link #4. Serious fault. 🔺
   Link #4: Summary:  tone 52 is  1.83088 dB below chord 2: (tones 45-62);  ❗
   Link #4: Details: chord 2: [ tone 45: 39.125 dB; curve midpoint test tone 52: 36.625 dB; tone 62: 37.5 dB ]; chord midpoint test tone 52: 38.45588 dB.

--
*❗ES (serious): Links with CRC errors at higher error rates, where the ES rate ≥ 60 ES / hr (†):
   Link #2 downstream:        latest period: ES per hr: 64.00, mean time between errors: 56.25 s, collection duration: 225 s
   Link #2 downstream: 'previous' period: ES per hr:  100.00, mean time between errors: 36 s, collection duration: 900 s

--
* ES (not so serious): The following links have a few CRC errors, at lower error rates, where the ES rate < 60 ES / hr (†):
   Link #1  downstream:        latest period: ES per hr: 36.18, mean time between errors: 99.5 s, collection duration:  199 s
   Link #1  downstream: 'previous' period: ES per hr: 24.00, mean time between errors:  150 s, collection duration: 900 s
   Link #4 downstream:        latest period: ES per hr:  18.18, mean time between errors:  198 s, collection duration:  198 s
   Link #4 downstream: 'previous' period: ES per hr:   4.00, mean time between errors: 900 s, collection duration: 900 s

──────────────────────────────
(†) The duration of the ’latest‘ errored seconds (ES) collection
bucket is variable, with a _maximum_ of 15 mins. The buckets’
start times are always 15 mins apart. A ‘previous’ bucket's duration
is a fixed 15 mins. An ES is a 1 s time period in which one or more
errors are detected ie. corrupted data is received that either can not
be recovered by back-calculation using the additional redundant
error correction information included with each message, or by 
being resent a certain maximum number of times, without being
received corrupted.
──────────────────────────────
                                     ◅ ◅ ◅◊▻ ▻ ▻


Detailed status of all modems, including ES:

Code: [Select]

         Detailed status of each modem
--
   Link #1  sync rate: downstream: 2.666 Mbps at SNRM 2.9 dB; upstream: 550 kbps at SNRM 8.7 dB
   Link #1  uptime:  10 hours 43 min 41  sec

   Link #1  ES: downstream: latest  15 minutes time: ES  16.67 per hr; MTBE 216 s; bucket duration 648 s
   Link #1  ES: downstream: previous  15 minutes time: ES 24 per hr; MTBE  150 s; bucket duration 900 s

--
   Link #2 sync rate: downstream: 2.645 Mbps at SNRM 3.1  dB; upstream: 512 kbps at SNRM 8.2 dB
   Link #2 uptime:  10 hours 43 min 57 sec

   Link #2 ES: downstream: latest  15 minutes time: ES 48.29 per hr; MTBE 74.56 s; bucket duration 671  s
   Link #2 ES: downstream: previous  15 minutes time: ES  100 per hr; MTBE 36 s; bucket duration 900 s

--
   Link #4 sync rate: downstream: 2.676 Mbps at SNRM 3.1  dB; upstream: 352 kbps at SNRM 6.0 dB
   Link #4 uptime:  10 hours 43 min 51  sec

   Link #4 ES: downstream: latest  15 minutes time: ES 5.6 per hr; MTBE 643 s; bucket duration 643 s
   Link #4 ES: downstream: previous  15 minutes time: ES 4 per hr; MTBE 900 s; bucket duration 900 s

--
                                              ◅ ◅ ◅◊▻ ▻ ▻



Here’s raw historical data for line 4:

Code: [Select]

xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 496 Kbps, Downstream rate = 2964 Kbps
Bearer: 0, Upstream rate = 550 Kbps, Downstream rate = 2666 Kbps

Link Power State: L0
Mode: ADSL2 Annex A
TPS-TC: ATM Mode(0x0)
Trellis: U:ON /D:ON
Line Status: No Defect
Training Status: Showtime
Down Up
SNR (dB): 2.9 8.7
Attn(dB): 66.0 41.5
Pwr(dBm): 18.1 12.4

ADSL2 framing
Bearer 0
MSGc: 51 12
B: 18 62
M: 8 1
T: 5 1
R: 16 14
S: 1.7944 3.6023
L: 749 171
D: 1 8

Counters
Bearer 0
SF: 2361893 273388
SFErr: 155 41
RS: 84142450 3508924
RSCorr: 52170 3317
RSUnCorr: 222 0

ReXmt: 19965 0
ReXmtCorr: 19910 0
ReXmtUnCorr: 223 0

Bearer 0
HEC: 356 62
OCD: 2 0
LCD: 2 0
Total Cells: 238773754 49330362
Data Cells: 6730560 3496136
Drop Cells: 0
Bit Errors: 12208 5923

ES: 858 1503
SES: 101 0
UAS: 6334 6282
AS: 37987

Bearer 0
INP: 28.00 2.50
INPRein: 0.00 0.00
delay: 8 7
PER: 15.98 16.21
OR: 28.53 8.88
AgR: 2682.35 557.45

Bitswap: 9365/9380 4/4

Total time = 19 days 9 hours 13 sec
FEC: 951166 694238
CRC: 6180 2262
ES: 858 1503
SES: 101 0
UAS: 6334 6282
LOS: 5 0
LOF: 36 0
LOM: 30 0
Latest 15 minutes time = 13 sec
FEC: 6 0
CRC: 0 0
ES: 0 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 15 minutes time = 15 min 0 sec
FEC: 983 7
CRC: 7 0
ES: 6 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Latest 1 day time = 9 hours 13 sec
FEC: 36444 2848
CRC: 31 41
ES: 25 19
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 1 day time = 24 hours 0 sec
FEC: 54315 18745
CRC: 3144 65
ES: 517 38
SES: 47 0
UAS: 6037 6026
LOS: 2 0
LOF: 9 0
LOM: 28 0
Since Link time = 10 hours 33 min 5 sec
FEC: 52170 3317
CRC: 155 41
ES: 117 19
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
NTR: mipsCntAtNtr=0 ncoCntAtNtr=0


[Weaver]

This below is much more typical, and I’m thinking that this is the difference between nighttime and daytime, given that all the modems were turned on during daytime, following the period of being unplugged during the thunderstorm near-miss.

Code: [Select]

      Summary of DSL links’ wellbeing and error counts
     ────────────────────────

* There are 4 modems in total. They are:  #1, #2, #3 and #4
* The active, contactable modems are:      #1, #2 and #4
* The modem skip-list declared in config:  #3 was skipped over and not queried; its state is unknown
* The modems successfully queried are:    #1, #2 and #4

       ───────────────────────
       ***                                                                           ***
       ***     There is some SERIOUS BADNESS !          ***
       ***                      All is not well !   😦                       ***
       ***                                                                           ***
       ───────────────────────

*** Note that modem #3 was skipped over and not queried; its state is unknown. ***

--
* HCD:❗So-called ‘hollow curve disease’ (HCD) defect detected in the downstream SNR-vs-tones curve of link #4. Serious fault. 🔺
   Link #4: Summary:  tone 52 is 2.31562 dB below chord 2: (tones 45-65);  ❗
   Link #4: Details: chord 2: [ tone 45: 39.5625 dB; curve midpoint test tone 52: 36.4375 dB; tone 65: 37.25 dB ]; chord midpoint test tone 52: 38.75312 dB.

--
* ES (not so serious): The following link has a few CRC errors, at a lower error rate, where the ES rate < 60 ES / hr (†):
   Link #1  upstream:      'previous' period: ES per hr:   8.00, mean time between errors: 450 s, collection duration: 900 s

──────────────────────────────
(†) The duration of the ’latest‘ errored seconds (ES) collection
bucket is variable, with a _maximum_ of 15 mins. The buckets’
start times are always 15 mins apart. A ‘previous’ bucket's duration
is a fixed 15 mins. An ES is a 1 s time period in which one or more
errors are detected ie. corrupted data is received that either can not
be recovered by back-calculation using the additional redundant
error correction information included with each message, or else by 
being resent a certain maximum number of times.
──────────────────────────────
                                     ◅ ◅ ◅◊▻ ▻ ▻