Topic: Weird pattern in stats related to error count if successive 24hr periods (Read 887 times)

Weaver · « **on:** August 01, 2019, 05:50:08 PM »

Tell me whether or not you see the same as I do in the stats from your modem, such as from my Broadcom CLI-equipped ZyXEL modems?

If I look at the CRC and ES counts for different time periods 15 min, the most recent part-day (less than 24 hr) and then the full 24 hr period before that I see that there are very few CRCs and ES for the more recent time periods. But after rescaling if necessary to account for any difference in the length of the time periods, the older periods might have ten or a hundred times more CRC or ES per unit time!

This is true for all my four modems right now and it always is true, it’s not a one off. What in earth is going on?

— Modem #1:: xdslctl: ADSL driver and PHY status Status: Showtime Last Retrain Reason: 8000 Last initialization procedure status: 0 Max: Upstream rate = 436 Kbps, Downstream rate = 3052 Kbps Bearer: 0, Upstream rate = 515 Kbps, Downstream rate = 2716 Kbps Link Power State: L0 Mode: ADSL2 Annex A TPS-TC: ATM Mode(0x0) Trellis: U:ON /D:ON Line Status: No Defect Training Status: Showtime Down Up SNR (dB): 3.0 5.9 Attn(dB): 66.0 41.6 Pwr(dBm): 18.3 12.4 ADSL2 framing Bearer 0 MSGc: 52 11 B: 18 62 M: 8 1 T: 5 1 R: 16 12 S: 1.7615 3.8462 L: 763 156 D: 1 8 Counters Bearer 0 SF: 37041611 474103 SFErr: 921 163 RS: 1342758416 273950 RSCorr: 515772 105917 RSUnCorr: 2958 0 ReXmt: 57257 0 ReXmtCorr: 56787 0 ReXmtUnCorr: 2958 0 Bearer 0 HEC: 3793 190 OCD: 74 0 LCD: 74 0 Total Cells: 3810389687 723694246 Data Cells: 127041550 27509836 Drop Cells: 0 Bit Errors: 116124 23259 ES: 443 111 SES: 0 0 UAS: 52 52 AS: 594951 Bearer 0 INP: 29.00 2.00 INPRein: 0.00 0.00 delay: 8 8 PER: 15.96 16.34 OR: 29.06 8.32 AgR: 2732.49 522.12 Bitswap: 188145/188148 6129/6129 Total time = 6 days 21 hours 16 min 43 sec FEC: 515772 105917 CRC: 921 163 ES: 443 111 SES: 0 0 UAS: 52 52 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 15 minutes time = 1 min 43 sec FEC: 48 21 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 15 minutes time = 15 min 0 sec FEC: 502 177 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 1 day time = 21 hours 16 min 43 sec FEC: 74856 13819 CRC: 1 1 ES: 1 1 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 1 day time = 24 hours 0 sec FEC: 78105 15010 CRC: 417 2 ES: 188 2 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Since Link time = 6 days 21 hours 15 min 51 sec FEC: 515772 105917 CRC: 921 163 ES: 443 111 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 NTR: mipsCntAtNtr=0 ncoCntAtNtr=0 — Modem #2:: xdslctl: ADSL driver and PHY status Status: Showtime Last Retrain Reason: 8000 Last initialization procedure status: 0 Max: Upstream rate = 440 Kbps, Downstream rate = 3020 Kbps Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2694 Kbps Link Power State: L0 Mode: ADSL2 Annex A TPS-TC: ATM Mode(0x0) Trellis: U:ON /D:ON Line Status: No Defect Training Status: Showtime Down Up SNR (dB): 3.1 6.4 Attn(dB): 66.0 42.1 Pwr(dBm): 18.2 12.4 ADSL2 framing Bearer 0 MSGc: 52 11 B: 18 62 M: 8 1 T: 5 1 R: 16 10 S: 1.7754 4.0000 L: 757 146 D: 1 8 Counters Bearer 0 SF: 36753032 458167 SFErr: 652 101 RS: 1332297445 2285919 RSCorr: 873009 39839 RSUnCorr: 1915 0 ReXmt: 36252 0 ReXmtCorr: 35964 0 ReXmtUnCorr: 1915 0 Bearer 0 HEC: 2626 108 OCD: 53 0 LCD: 53 0 Total Cells: 3780705843 695900800 Data Cells: 126110525 26051136 Drop Cells: 0 Bit Errors: 82435 14240 ES: 293 76 SES: 0 0 UAS: 62 62 AS: 594993 Bearer 0 INP: 28.00 2.00 INPRein: 0.00 0.00 delay: 8 8 PER: 16.08 17.00 OR: 28.83 8.00 AgR: 2711.00 502.03 Bitswap: 177577/178283 697/697 Total time = 6 days 21 hours 17 min 35 sec FEC: 873009 39839 CRC: 652 101 ES: 293 76 SES: 0 0 UAS: 62 62 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 15 minutes time = 2 min 35 sec FEC: 20 0 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 15 minutes time = 15 min 0 sec FEC: 90 175 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 1 day time = 21 hours 17 min 35 sec FEC: 12934 3375 CRC: 1 8 ES: 1 8 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 1 day time = 24 hours 0 sec FEC: 18221 8984 CRC: 292 30 ES: 153 26 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Since Link time = 6 days 21 hours 16 min 31 sec FEC: 873009 39839 CRC: 652 101 ES: 293 76 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 NTR: mipsCntAtNtr=0 ncoCntAtNtr=0 — Modem #3:: xdslctl: ADSL driver and PHY status Status: Showtime Last Retrain Reason: 8000 Last initialization procedure status: 0 Max: Upstream rate = 250 Kbps, Downstream rate = 3052 Kbps Bearer: 0, Upstream rate = 412 Kbps, Downstream rate = 2788 Kbps Link Power State: L0 Mode: ADSL2 Annex A TPS-TC: ATM Mode(0x0) Trellis: U:ON /D:ON Line Status: No Defect Training Status: Showtime Down Up SNR (dB): 3.1 2.7 Attn(dB): 65.5 41.4 Pwr(dBm): 18.1 12.4 ADSL2 framing Bearer 0 MSGc: 51 12 B: 32 5 M: 4 16 T: 3 8 R: 10 16 S: 1.4889 7.2846 L: 763 123 D: 2 4 Counters Bearer 0 SF: 52129533 819915 SFErr: 1603 47 RS: 2228537578 3433567 RSCorr: 88585 1191645 RSUnCorr: 10158 0 ReXmt: 111880 0 ReXmtCorr: 102275 0 ReXmtUnCorr: 10569 0 Bearer 0 HEC: 8579 50 OCD: 262 0 LCD: 262 0 Total Cells: 1199269702 813561053 Data Cells: 174421135 30254365 Drop Cells: 0 Bit Errors: 268076 5078 ES: 688 43 SES: 0 0 UAS: 57 57 AS: 835520 Bearer 0 INP: 26.00 2.00 INPRein: 0.00 0.00 delay: 8 7 PER: 15.91 16.39 OR: 28.65 8.78 AgR: 2804.62 420.07 Bitswap: 195562/195566 13008/13008 Total time = 9 days 16 hours 6 min 17 sec FEC: 88585 1191645 CRC: 1603 47 ES: 688 43 SES: 0 0 UAS: 57 57 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 15 minutes time = 6 min 17 sec FEC: 10 595 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 15 minutes time = 15 min 0 sec FEC: 43 1271 CRC: 0 2 ES: 0 1 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 1 day time = 16 hours 6 min 17 sec FEC: 1910 46504 CRC: 2 4 ES: 1 3 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 1 day time = 24 hours 0 sec FEC: 6474 152006 CRC: 95 6 ES: 58 6 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Since Link time = 9 days 16 hours 5 min 19 sec FEC: 88585 1191645 CRC: 1603 47 ES: 688 43 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 NTR: mipsCntAtNtr=0 ncoCntAtNtr=0 — Modem #4:: xdslctl: ADSL driver and PHY status Status: Showtime Last Retrain Reason: 8000 Last initialization procedure status: 0 Max: Upstream rate = 440 Kbps, Downstream rate = 3100 Kbps Bearer: 0, Upstream rate = 499 Kbps, Downstream rate = 2762 Kbps Link Power State: L0 Mode: ADSL2 Annex A TPS-TC: ATM Mode(0x0) Trellis: U:ON /D:ON Line Status: No Defect Training Status: Showtime Down Up SNR (dB): 3.0 5.8 Attn(dB): 66.0 41.9 Pwr(dBm): 18.2 12.4 ADSL2 framing Bearer 0 MSGc: 53 11 B: 18 62 M: 8 1 T: 5 1 R: 16 10 S: 1.7320 3.9728 L: 776 147 D: 1 8 Counters Bearer 0 SF: 37031279 455204 SFErr: 576 167 RS: 1365528430 1456563 RSCorr: 51865 135341 RSUnCorr: 2005 0 ReXmt: 27904 0 ReXmtCorr: 27618 0 ReXmtUnCorr: 2005 0 Bearer 0 HEC: 2356 139 OCD: 45 0 LCD: 45 0 Total Cells: 3875005557 700555148 Data Cells: 130105192 26528327 Drop Cells: 0 Bit Errors: 70728 25068 ES: 208 117 SES: 0 0 UAS: 54 54 AS: 594888 Bearer 0 INP: 29.00 2.00 INPRein: 0.00 0.00 delay: 8 8 PER: 15.96 16.88 OR: 29.56 8.05 AgR: 2779.04 505.47 Bitswap: 123016/123602 1798/1798 Total time = 6 days 21 hours 15 min 42 sec FEC: 51865 135341 CRC: 576 167 ES: 208 117 SES: 0 0 UAS: 54 54 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 15 minutes time = 42 sec FEC: 1 9 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 15 minutes time = 15 min 0 sec FEC: 25 254 CRC: 0 0 ES: 0 0 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Latest 1 day time = 21 hours 15 min 42 sec FEC: 3313 21064 CRC: 0 29 ES: 0 23 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Previous 1 day time = 24 hours 0 sec FEC: 8410 25669 CRC: 336 24 ES: 123 18 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 Since Link time = 6 days 21 hours 14 min 47 sec FEC: 51865 135341 CRC: 576 167 ES: 208 117 SES: 0 0 UAS: 0 0 LOS: 0 0 LOF: 0 0 LOM: 0 0 NTR: mipsCntAtNtr=0 ncoCntAtNtr=0

There is also something interesting in the comparison between the recent full 24hr counts and the entire lifetime count as well. When you rescale the two dividing by the durations, the error count per unit time values do not match, there are not enough errors in the full lifetime period. To take just one example: ES = 123 in 24 hr period but only 208 for a whole-lifetime period of 6 days 21 hours.

I wonder if (a) I am going mad, (b) just failing to read the stats properly (c) there’s a serious bug in the results-display code (d) can’t think what else. There can’t really have been a one-off ‘peak’ in error rate that occupies a certain time period, because it keeps on happening over and over again. If there were pulses such as that, why do I nit sometimes see that the most recent period is bad say and the preceding time period is good, which is what would be seen if I happened to start observing just when the error rate peak was still in progress and earlier on there could have been a quiet time. That would be what you would expect to see sometimes.

kitz · « **Reply #1 on:** August 01, 2019, 11:55:13 PM »

Without looking at the stats in detail... I will say that there is some sort of bug obtained from the full telnet stats that we've been aware of for a while in that it doesn't seem to record total uptime data correctly and the link time counter loses track of the correct amount of days.

Mine is showing as follows

Code: [Select]

Total time = 1 days 5 hours 59 min 9 sec
FEC:            0               34750
CRC:            143193          3134
ES:             8106            2714
SES:            263             0
UAS:            28              28
LOS:            0               0
LOF:            0               0
LOM:            0               0

I know for a fact that my modem has been up for well over 3 months without resyncing.

Weaver · « **Reply #2 on:** August 03, 2019, 11:46:12 AM »

The individual per-period numbers seem to make little sense.

I haven’t checked particular individual entries repeatedly to see if they are monotonic or not, or if they have some sort of suspicious pattern to them.

Thanks to Kitz.

Would anyone else be willing to take a quick peek at their own stats just to see if they get the same general picture as I do, regarding the relationship between the numbers of ES in the different time periods ?

News:

Author Topic: Weird pattern in stats related to error count if successive 24hr periods (Read 887 times)

Weaver

Weird pattern in stats related to error count if successive 24hr periods

kitz

Re: Weird pattern in stats related to error count if successive 24hr periods

Weaver

Re: Weird pattern in stats related to error count if successive 24hr periods