Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[kitz]

Following on from broadstairs and daveesh1 threads, thought I'd open my own "Strange happenings" for yet another ECI cab.   These events actually took place on the 12th, but today is first chance I've had to report it.

On Tues my internet became useless.   Pages were taking several minutes to load and many pages would either fail to load or only partially load.   I had a quick look at my basic line state (sync and SNR) which looked ok, and then seeing a page on the PN forums about routing issues, perhaps decided it was that.  I had other things on so never bothered to look in depth. 

Speedtest.net failed to load, but after a few attempts I did manage to get a speed test from TBB



I did a quick PPP drop to ensure it wasnt a gateway issue, but seeing the connection was still rubbish I left it to concentrate on other things.

Then on Wed morning I noticed this email.

This is to let you know that a resync/restart occurred on your line at 13-07-2016 06:41 local time (+/- 1 minute).

Reason: 0 Loss of Service

Current Downstream Sync Rate is now 75592kbps Current Upstream Sync Rate is now 20000kbps

Hmmm so the DLM has clobbered me for something, so I go look at my errors...





Whatever it was started at 10:30 am and righted itself at 10pm without any interaction from me.
Despite the line being stable since Tues eve, DLM as yet hasnt shown any sign of relenting. 

----
I'm suspecting that this was cab related. Like the others with weird events Im on an ECI.
My cab got the xx206 f/w update this week. 
Ive mentioned in the past about how the cabs seems to be busy sending some sort of data between the hours of 10-11 each morning and night because it also triggers what Im assuming is the G.INP error spikes. 

Strictly speaking because the line is good and had no recent previous DLM history I was hoping that it should have decided to go back to normal on Friday or at least by this morning

[NewtronStar]

Yeah seen this happen on your line on MDWS a few days ago moved from fastpath to interleaving as would be expected with 31000 errored seconds in 24 hours 10 times over the limit of 2880.

From my experience your looking at up to 14 days for the DLM to switch you back to fastpath

[j0hn]

wow, that's some amount of errors. The crc/es seems to have went solid for 12 hours. looks like they went on and off as quick as flicking a switch. The ses only peaked for the 1st hour.

not surprised dlm took action as you broke the mtbe threshold within the 1st hour. that's quite a chunk of interleaving to add for a 1st step. is this the 1st time you've had this pattern of errors happen?

[kitz]

Yep.   The line is normally ILQ green and has quite a bit of surplus SNRm. 

I have been DLM'd before after a line fault several years ago and then once again about 18m ago when I was beta testing a modem that turned out not to be bitswapping correctly.   In both cases DLM started recovery after one full day without errors, which is why I was expecting perhaps something maybe as early as Thurs. 


It stopped all by itself, but whilst that huge amount of errors were going on it was impossible to surf the internet.   You can also see what happened to my speeds.  The downspike at ~6pm was me doing a manual restart just to eliminate ISP gateway.

[NewtronStar]

In both cases DLM started recovery after one full day without errors, which is why I was expecting perhaps something maybe as early as Thurs. 

It has been a while since my last DLM intervention which was banding with higher than normal INT: depth, you should start to get resyncs when the DLM is ready to adjust your line back to were it was in small decrements the INT: level should decrease during each event and then resync to fastpath

Day 5 should be the start of this DLM process hopefully but you and I know it never goes to plan

[burakkucat]

There certainly appears to be something strange occurring with circuits connected via ECI Hi-FOCuS M41 equipped cabinets. 

[NewtronStar]

What are your seeing that we can't ?

[kitz]

Thanks NS (et al)  for your input

AFAIK, the previously quoted 5 days, is 'old hat' and they moved over to some sort of doubler method.   It is exceedingly difficult to know what is going on though as they may have changed the rules after the ASSIA court case for FTTC lines.

It was about Oct/Nov 2014 that iirc the DLM last had anything to do with my line.  It certainly appeared to be using the 'doubler method' back then.
That was through no fault of my own, but some weird thing that appeared to be backhaul/cab related.

What concerns me atm is according to my ISP yesterday I am currently this profile:-

Profile Name 0.128M-80M Downstream, Error Protection Off - 00.128M-20M Upstream, Error Protection Off

However it should be noted that I do not have a ticket open on this issue as a fault.... and that the info comes in relation to something else entirely separate.

If that is my supposed profile, it certainly is not what I am seeing live. 

[WWWombat]

More bizarreness from ECI, eh?

What I can see, from perusing the stats on MDWS, is:
- CRCs/min jumps to a pretty constant 18,000 from 10:44 to 21:49
- Bitswaps/min jumps to a pretty constant 52 per minute from 10:45 to 21:48
- ESs: I can only view the numbers in "whole hour" values, but every "whole hour" from 11:00 to 20:00 shows ES's at 2,800 per hour, which is more than 75% of the seconds per hour. The 10:00 total is pro-rata for 16 minutes; the 21:00 is pro-rata for 50 minutes.

Those are all pretty consistent. However SES's (again, only "whole hour" totals) happen in a very non-consistent way:
- The 10am total for SES (637) shows about 82% of that hour's ES's (772) were severe. A high proportion.
- The 11am total for SES (2291) shows about 81% of that hour's ES's (2819) were severe. A high proportion again.
- The 12pm total for SES (581) shows only 20% of that hour's ES's (2819) were severe. A much lower proprotion!
- For 1pm and beyond, SESs run at approximately 40 per hour, or about 1%. That's a significant change from the earlier hours - but no different profile in terms of ES, CRC or bitswaps.

Just like the previous events, where we saw ECI craziness for precisely an hour, these statistics look more like anomalies of the modems, rather than real errors caused by noise on the line. I can't tell if they have become real errors (ie real CRC failures), or they are errors in generation of the statistics:
- On one hand, the slowdown during a speedtest suggests that real blocks are going missing.
- 18,000 CRC errors per minute sounds awfully high to (a) only trigger ESs for about 75% of the seconds, and (b) to not trigger SES for much beyond 12:00.

A few days ago, I pointed out an ADSL superframe rate of 58 per second, and that I believed that ADSL could therefore generate a maximum CRC count of 58/second, or 3,500 per minute. 18,000 per minute in ADSL would be a huge, impossible number.

I'm not sure that VDSL2 works with the same limitations. Some modems report counts for "B (# of bytes in Mux Data Frame)", "M (# of Mux Data Frames in an RS codeword)" and "T (# of Mux Data Frames in an OH sub-frame)". Right now, @kitz's line is reporting B=51, M=1, T=64", though it was surely different when the errors happened.

Is an "OH sub-frame" the same block size as protected by one CRC? If so, the CRC block is currently 3,264 bytes long, so there are roughly 2,750 blocks per second on the 72Mbps line - far more than are needed to generate 300 CRC's per second.

So, why does the SES pattern look very different to ES and CRC?

I wonder if the fault is limited to the DSP, and its handling of certain bits and certain tones. I wonder if the fault started out affecting a lot of bits within each CRC block (but, heh, you can only count a corrupted block once no matter how corrupted it got). Then, over a couple of hours, the bitswap process managed to move away from most of the faulty sections. That the process changed from a profile of "many errors within one second" to the same number of errors, but better spread out over time.

I'm not sure I can believe that such a shift manages to keep both the CRC and ES rate identical, even if it might explain why the SES rate tales off. Hmmm 

[pooclah]

I had a similar event recently, nowhere near as bad as yours but the difference is that I’m on a Huawei cabinet.
 
Probably totally unrelated but odd all the same

[Chrysalis]

damn.

Newt's 14 day suggestion may be right, I remember a couple of years back he told me to wait 14 days and bam at day 14-15 I got reverted by DLM.

[kitz]

Something has gone decidedly wrong with DLM.

According to Plusnet this is my line profile since last Friday (15th) July.

Profile Name 0128M-80M Downstream, Error Protection Off  - 0128M-20M Upstream, Error Protection Off.

Since then Ive had several resyncs
The backhaul has been down since 01:58 last night but came back up this afternoon. (BT35812)
Ive tried putting another router on, but I'm still interleaved and have INP 3.
Plusnet ran another GEA test today (before BT issued the MSO) and my line profile is still 80/20.

Yet these are my line stats which quite clearly show INP 3 and interleaved.   If DLM thinks that since last Friday that Interleave & INP isnt in place then I dont think its now going to clear by itself.  It appears stuck.

Code: [Select]

adsl info --stats
adsl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    0
Last initialization procedure status:   0
Max:    Upstream rate = 30709 Kbps, Downstream rate = 91284 Kbps
Bearer: 0, Upstream rate = 20000 Kbps, Downstream rate = 75928 Kbps

Link Power State:       L0
Mode:                   VDSL2 Annex B
VDSL2 Profile:          Profile 17a
TPS-TC:                 PTM Mode
Trellis:                U:ON /D:ON
Line Status:            No Defect
Training Status:        Showtime
                Down            Up
SNR (dB):        6.1             12.6
Attn(dB):        0.0             0.0
Pwr(dBm):        5.6             5.6

                        VDSL2 framing
                        Bearer 0
MSGc:           18              150
B:              51              236
M:              1               1
T:              64              5
R:              12              16
S:              0.0218          0.3771
L:              23496           5410
D:              1487            1
I:              64              255
N:              64              255

                        Counters
                        Bearer 0
OHF:            12794371                186201
OHFErr:         0               1
RS:             3275177836              1074482
RSCorr:         19693           37
RSUnCorr:       0               0

                        Bearer 0
HEC:            0               0
OCD:            0               0
LCD:            0               0
Total Cells:    2615562472              0
Data Cells:     4526041         0
Drop Cells:     0
Bit Errors:     0               0

ES:             0               1
SES:            0               0
UAS:            29              29
AS:             17918

                        Bearer 0
INP:            3.00            0.00
INPRein:        0.00            0.00
delay:          8               0
PER:            1.40            6.15
OR:             137.13          202.87
AgR:            76064.87        20203.27

Bitswap:        24/24           0/0

Total time = 4 hours 59 min 7 sec
FEC:            19693           37
CRC:            0               1
ES:             0               1
SES:            0               0
UAS:            29              29
LOS:            0               0
LOF:            0               0
LOM:            0               0
Latest 15 minutes time = 14 min 7 sec
FEC:            17              0
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Previous 15 minutes time = 15 min 0 sec
FEC:            1               0
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Latest 1 day time = 4 hours 59 min 7 sec
FEC:            19693           37
CRC:            0               1
ES:             0               1
SES:            0               0
UAS:            29              29
LOS:            0               0
LOF:            0               0
LOM:            0               0
Previous 1 day time = 0 sec
FEC:            0               0
CRC:            0               0
ES:             0               0
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
Since Link time = 4 hours 58 min 37 sec
FEC:            19693           37
CRC:            0               1
ES:             0               1
SES:            0               0
UAS:            0               0
LOS:            0               0
LOF:            0               0
LOM:            0               0
 >

[NewtronStar]

Only time will tell whether your line has become stuck on this interleaved profile I would wait until the 27th of July and if no change raise a ticket with Plusnet.

I take it your seeing higher than normal pings which would suggest your on the interleaving profile even your bloated attainable sync says so.

Are you putting this down to the XXX206 firmware ?

[kitz]

I already raised a ticket earlier this evening.    I may have waited a bit longer under normal circumstances but finding out on Friday that my DLM profile appeared to have reverted back to how it should be yet it didn't.

I rang Plusnet this morning at about 8:30 re my Internet being down.   At that time MSO BT35812 hadn't been released, so they ran a GEA service test which again showed that my DLM profile should be 0128M-80M Downstream, Error Protection Off  - 0128M-20M Upstream, Error Protection Off.

In order to satisfy PN and ensure that me not having internet connectivity (I had sync) I put another router on, but the results were practically the same.   Using the TP-link, I could see that I was still interleaved and also the lower sync speed would suggest that Error Protection was indeed on.

The guy at PN refreshed my line to ensure there wasnt a stale session and I left it disconnected for a good hour or so whilst I went out.   The guy at PN also noticed the differential and suggested I raise it as a fault.  (Separate online process) which I have now done.

Yes the increased latency is definitely there.  Normally I can ping the BBC in 12ms

Code: [Select]

Tracing route to www.bbc.net.uk [212.58.246.54]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    22 ms    22 ms    22 ms  lo0.12.central12.pcl-bng01.plus.net [195.166.130.140]
  3    22 ms    22 ms    22 ms  irb.12.pcl-cr01.plus.net [84.93.249.113]
  4    22 ms    23 ms    22 ms  ae1.ptw-cr01.plus.net [195.166.129.0]
  5    22 ms    23 ms    22 ms  kingston-gw.thdo.bbc.co.uk [212.58.239.6]
  6     *        *        *     Request timed out.
  7    23 ms    23 ms    23 ms  ae0.er01.cwwtf.bbc.co.uk [132.185.254.93]
  8    24 ms    24 ms    24 ms  132.185.255.165
  9    23 ms    23 ms    23 ms  bbc-vip045.cwwtf.bbc.co.uk [212.58.246.54]

Are you putting this down to the XXX206 firmware ?

There is something else going on.   I don't think the f/w would cause that huge amount of Err/Secs last week.  Thats likely hardware somewhere in the BT network.   Last week I would have guessed between the cab and the head-end exchange. 

The fault at 01:58 this morning suggests hardware again.   I later found this failed maintenance fault reference BT35765 which implies its on the BTw backhaul as it says 21CN.  They obviously had planned maintenance scheduled but something broke completely.
Also judging from a post on a friends facebook wall and the amount of 'me too''s and 'Im ok', that I saw later,  it was only those using BT and PN reporting they were down.  Those on Sky/TT said they were ok. *

So could it be whatever broke last night/this morning be what generated all the Err/Secs?   I guess I'll never know.


----
ETA

* It was only those on BT/PN FTTC.    Someone on BT ADSL2+ was fine.   But then again ADSL2+ would backhaul from the local exchange.  Cleveleys is the FTTC head-end exchange.
   

[kitz]

So, why does the SES pattern look very different to ES and CRC?

I know, all rather strange.

I can't tell if they have become real errors (ie real CRC failures), or they are errors in generation of the statistics:

They were real errors, the Internet was pretty broken for me.   
Web pages wouldn't load or pages would only partially load.  Those that did would take about 2 mins.   I tried doing a test at speedtest.net but the page just sat there.   The TBB one worked and you can see on there the effect on throughput.   I tried downloading mail at one point and it took ages despite there not being any substantial attachments.