Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Bald_Eagle1]

FWIW, I/we have today started to remotely monitor another Huawei HG612 modem connected to an ECI DSLAM.

US QLN AND Hlog data can indeed be seen & plotted:-

I have been dabbling with my scripts & it appears that US SNR data is also provided when connected to an ECI DSLAM.

Another example Showing US SNR is attached.



@ JoshShep & BlackEagle,

If you would like to email a recent Plink log or two to me, I could check this out a bit further.

[JoshShep]

7 day graph here.

I can't figure out any patterns of these errors;

they just seem to be random.

Cheers,

Josh

[burakkucat]

Josh -- A simple request . . . When posting a montage of your line statistics for our examination, please post them in the portrait format rather than landscape.

You have posted line_stats_L-20120502-1243.png, above, whereas line_stats_P-20120502-1243.png would allow more convenient viewing. 

[JoshShep]

Josh -- A simple request . . . When posting a montage of your line statistics for our examination, please post them in the portrait format rather than landscape.

You have posted line_stats_L-20120502-1243.png, above, whereas line_stats_P-20120502-1243.png would allow more convenient viewing. 

Thanks for that, will do next time!

[asbokid]

7 day graph here.  I can't figure out any patterns of these errors;

they just seem to be random.

Cheers, Josh

The spikes of errors could be misleading.   They primarily record the packet flow on your connection as a function of time. The spikes are indicative of your internet usage pattern as much as they illustrate noise interference at any point in time.   There would be no packets in error if there was no packet flow, IYSWIM.   More important is the proportion of packets (or cells or frames) in error, measured as a percentage of those transmitted and received. That metric would capture the relative noise on the line along the time domain.  The task then is to identify any variance in the distribution of errors across time.

cheers, a

[Bald_Eagle1]

The spikes of errors could be misleading.   They primarily record the packet flow on your connection as a function of time. The spikes are indicative of your internet usage pattern as much as they illustrate noise interference at any point in time.   There would be no packets in error if there was no packet flow, IYSWIM.   More important is the proportion of packets (or cells or frames) in error, measured as a percentage of those transmitted and received. That metric would capture the relative noise on the line along the time domain.  The task then is to identify any variance in the distribution of errors across time.

So, out of this lot, what data would be more beneficial in trouble-shooting, instead of or as well as the data currently being plotted:-

Code: [Select]

# xdslcmd info --stats
xdslcmd: ADSL driver and PHY status
Status: Showtime
Retrain Reason: 2
Max: Upstream rate = 5654 Kbps, Downstream rate = 32148 Kbps
Path: 0, Upstream rate = 5795 Kbps, Downstream rate = 29624 Kbps

Link Power State: L0
Mode: VDSL2 Annex B
VDSL2 Profile: Profile 17a
TPS-TC: PTM Mode
Trellis: U:ON /D:ON
Line Status: No Defect
Training Status: Showtime
Down Up
SNR (dB): 4.1 5.7
Attn(dB): 0.0 0.0
Pwr(dBm): 12.3 6.3
VDSL2 framing
Path 0
B: 63 175
M: 1 1
T: 64 39
R: 16 12
S: 0.0687 0.9635
L: 9312 1561
D: 469 1
I: 80 94
N: 80 188
Counters
Path 0
OHF: 68852985 1057147
OHFErr: 3887 587
RS: 334919681 1664228
RSCorr: 4855931 273
RSUnCorr: 140410 0

Path 0
HEC: 39871 0
OCD: 0 0
LCD: 0 0
Total Cells: 92094052 0
Data Cells: 24616994 0
Drop Cells: 0
Bit Errors: 0 0

ES: 605 921
SES: 46 0
UAS: 65 65
AS: 228032

Path 0
INP: 3.00 0.00
PER: 3.29 9.39
delay: 8.00 0.00
OR: 58.20 27.25

Bitswap: 66317 10130

Total time = 1 days 14 hours 24 min 55 sec
FEC: 7935219 484
CRC: 11750 978
ES: 605 921
SES: 46 0
UAS: 65 65
LOS: 4 0
LOF: 5 0
Latest 15 minutes time = 9 min 55 sec
FEC: 2970 2
CRC: 5 2
ES: 2 2
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Previous 15 minutes time = 15 min 0 sec
FEC: 3950 1
CRC: 1 3
ES: 1 3
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Latest 1 day time = 14 hours 24 min 55 sec
FEC: 1464425 69
CRC: 283 144
ES: 76 142
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Previous 1 day time = 24 hours 0 sec
FEC: 1756226 86
CRC: 1197 242
ES: 155 230
SES: 3 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Since Link time = 2 days 15 hours 20 min 31 sec
FEC: 4855931 273
CRC: 3887 587
ES: 358 561
SES: 12 0
UAS: 0 0
LOS: 0 0
LOF: 0 0


The montage of the graphs using some of that data is attached

[asbokid]

EDIT:

Oops, sorry, I was wrong.. the cell count (at least for my ATM connection) increases when the link is up but not being used (from the transmission of idle cells).. So the various error counters should be increasing too..  So your graphs do capture what is needed. 

The Bit Error Rate (BER) would be the more widely accepted measure of the quality of a transmission channel.  If it's not available directly then an estimation could probably be derived from other statistics.  The three graphs (CRC from GUI, HEC from GUI and RSUnCorr from telnet) and two other graphs (FEC from GUI and OHFErr from telnet) are basically identical.  Could they be combined to save screen space?

Something that makes it difficult to spot the distribution of errors in the time domain is the tick frequency used on the x-axis for the noise graphs. Every seven hours has an x-tick, whereas a tick on a factor of 24 (hours) would be more intuitive.  i.e. an x-tick every 4, 6 or 8 hours.  That way it would be easier to compare the error distributions across different days of the week.

cheers, a

The spikes of errors could be misleading.   They primarily record the packet flow on your connection as a function of time. The spikes are indicative of your internet usage pattern as much as they illustrate noise interference at any point in time.   There would be no packets in error if there was no packet flow, IYSWIM.   More important is the proportion of packets (or cells or frames) in error, measured as a percentage of those transmitted and received. That metric would capture the relative noise on the line along the time domain.  The task then is to identify any variance in the distribution of errors across time.

So, out of this lot, what data would be more beneficial in trouble-shooting, instead of or as well as the data currently being plotted:-

Code: [Select]

# xdslcmd info --stats
xdslcmd: ADSL driver and PHY status
Status: Showtime
Retrain Reason: 2
Max: Upstream rate = 5654 Kbps, Downstream rate = 32148 Kbps
Path: 0, Upstream rate = 5795 Kbps, Downstream rate = 29624 Kbps

Link Power State: L0
Mode: VDSL2 Annex B
VDSL2 Profile: Profile 17a
TPS-TC: PTM Mode
Trellis: U:ON /D:ON
Line Status: No Defect
Training Status: Showtime
Down Up
SNR (dB): 4.1 5.7
Attn(dB): 0.0 0.0
Pwr(dBm): 12.3 6.3
VDSL2 framing
Path 0
B: 63 175
M: 1 1
T: 64 39
R: 16 12
S: 0.0687 0.9635
L: 9312 1561
D: 469 1
I: 80 94
N: 80 188
Counters
Path 0
OHF: 68852985 1057147
OHFErr: 3887 587
RS: 334919681 1664228
RSCorr: 4855931 273
RSUnCorr: 140410 0

Path 0
HEC: 39871 0
OCD: 0 0
LCD: 0 0
Total Cells: 92094052 0
Data Cells: 24616994 0
Drop Cells: 0
Bit Errors: 0 0

ES: 605 921
SES: 46 0
UAS: 65 65
AS: 228032

Path 0
INP: 3.00 0.00
PER: 3.29 9.39
delay: 8.00 0.00
OR: 58.20 27.25

Bitswap: 66317 10130

Total time = 1 days 14 hours 24 min 55 sec
FEC: 7935219 484
CRC: 11750 978
ES: 605 921
SES: 46 0
UAS: 65 65
LOS: 4 0
LOF: 5 0
Latest 15 minutes time = 9 min 55 sec
FEC: 2970 2
CRC: 5 2
ES: 2 2
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Previous 15 minutes time = 15 min 0 sec
FEC: 3950 1
CRC: 1 3
ES: 1 3
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Latest 1 day time = 14 hours 24 min 55 sec
FEC: 1464425 69
CRC: 283 144
ES: 76 142
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Previous 1 day time = 24 hours 0 sec
FEC: 1756226 86
CRC: 1197 242
ES: 155 230
SES: 3 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
Since Link time = 2 days 15 hours 20 min 31 sec
FEC: 4855931 273
CRC: 3887 587
ES: 358 561
SES: 12 0
UAS: 0 0
LOS: 0 0
LOF: 0 0


A ratio is needed, describing the number of [packets|cells|frames|bits] sent and received, and the number of them arriving at their destination in error.

Which data is trustworthy though?   For your VDSL2 service, several of the crucial fields are unpopulated.

For an ADSL2+ connection, this is reported by xdslcmd..

Code: [Select]

Path 0
HEC: 64715 21
OCD: 0 0
LCD: 0 0
Total Cells: 2400611752 440438110
Data Cells: 13617798 3052399
Drop Cells: 0
Bit Errors: 5780721 1430

We can see that 2,400,611,752 (2.4e9) ATM cells were received. Each ATM cell has a 48 byte payload + 5 header bytes =  424 bit cell length. And there were 5,780,721 (5.8e6) bit errors.

5.8e6 / (2.4e9 * 424) = 5.7e-6 = Bit Error Rate (BER)

When talking about bit- or packet- error rates, the coefficient (in this case 5.7) is usually ignored since it's almost irrelevant when dealing with numbers of such magnitude. Generally only the exponent (10^-6) is quoted as the BER.

In common language, around six bits in every million bits received by my TalkTalk connection are in error (before any error correction is applied).

Monitoring on a minute-by-minute basis that BER ratio (5.7e-6 in the worked example above) may hold the key to tracking down time-dependent problems from impulse noise, IMVHO.

cheers, a

[Bald_Eagle1]

I have checcked a few PuTTy/Plink logs from the HG612 modem (not only my own) & in every case, Bit Errors has zero values.

I also had a look at xdslcmd bert --start 300 (5 minutes of monitoring):-

# xdslcmd bert --start 300

# xdslcmd bert --show
xdslcmd: BERT results:
BERT Status = RUNNING
BERT Total Time   = 300 sec
BERT Elapsed Time = 40 sec
BERT Bits Tested = 0x000000003F5CA480 bits
BERT Err Bits = 0x0000000000000000 bits
:
:
:
:
:
# xdslcmd bert --show
xdslcmd: BERT results:
BERT Status = RUNNING
BERT Total Time   = 300 sec
BERT Elapsed Time = 280 sec
BERT Bits Tested = 0x00000001BB885400 bits
BERT Err Bits = 0x0000000000000000 bits

# xdslcmd bert --show
xdslcmd: BERT results:
BERT Status = NOT RUNNING
BERT Total Time   = 300 sec
BERT Elapsed Time = 300 sec
BERT Bits Tested = 0x00000001DB369980 bits
BERT Err Bits = 0x0000000000000000 bits
#

In my (not yet released) script updates, Error Seconds & Serious Error Seconds data is harvested, for now only DS & US Error Seconds being plotted.

However, for monitoring noise levels, I imagine changes in SNRM values are perhaps the most relevant?

In general, stable & quiet connections seem to comprise more or less straight line graphs (even recently monitored low speed connections over much longer distances appear very stable).

My own connection currently seems stable(ish) at 29.5Mb sync speed.
I have no explanation why it was unstable for a few days & returned to relative stability again as no repair work was carried out (unless my connection simply prefers wet & cold weather).
Interleaving, INP & delay have been turned back on at fairly low levels so various error types are now visible (e.g. RSCorr), while other error types have diminished.

I see that both Josh's & BlackEagle's connections suffer from many "error" spikes, sudden changes in SNRM levels, stuck sync speeds & quite high levels of Interleaving etc.
For both those connections (as when mine plays up), no real timed pattern has been detected.
It is possible that bursts of interference/errors could occur anywhere between the 1 minute sampling, with only the cumulative error counts being visible in the harvested data.

I believe Routerstats can log data right down to 5 second intervals.
I can ALMOST get Routerstats working with the HG612, but it incorrectly alternates between reporting SNRM & sync speed values in the same SNRM graph.

If anyone wants to have a go at setting Routerstats up, the login URL for the HG612's GUI data is http://192.168.1.1/html/status/xdslStatus.asp
The SNRM & Output Power values need dividing by 10 to display correct values to 1 decimal point.

I might some time have a go at converting my scripts to run continuously, harvesting data at much shorter intervals, as an alternative to only running them every minute via Windows Task Scheduler.

[Blackeagle]

Hi BE, yeah I can ALMOST get RS logging correctly, apart from the bouncing issue !!

Telnet would seem the better option as the ini file can easily be modified to replace the adslctl command with xdslcmd.  Unfortunately, the HG612 logs into the atp shell so unless we can either persuade John to include an "sh" command to get to the correct shell, or find an atp command that returns the same data its a no go.

Unless its possible to embed a carriage return into the command line passed via telnet to the router. ?

[Bald_Eagle1]

Hi BE,

Hi BE, yeah I can ALMOST get RS logging correctly, apart from the bouncing issue !!

Telnet would seem the better option as the ini file can easily be modified to replace the adslctl command with xdslcmd.  Unfortunately, the HG612 logs into the atp shell so unless we can either persuade John to include an "sh" command to get to the correct shell, or find an atp command that returns the same data its a no go.

Unless its possible to embed a carriage return into the command line passed via telnet to the router. ?

"Echoing" commands from a batch file includes a carriage return, as does "Typing" a one line text file.

For whatever reason, "Typing" a text file seems more reliable than "echoing":-


rem *************** Login using PERMANENT login files ***************

(sleep 2^
 & type %LOGIN_FOLDER%\login1.txt & sleep 1^
 & type %LOGIN_FOLDER%\login2.txt & sleep 1^
 & type %LOGIN_FOLDER%\login3.txt & sleep 1^
 & echo xdslcmd info --stats & sleep 1^
 & echo exit & sleep 1 & echo exit & sleep 2)^
 ^| Plink -telnet -P 23 192.168.1.1>> %WORKING_FOLDER%\data$$



login1.txt just contains the one word admin
login2.txt just contains the one word admin
login3.txt just contains the one word sh


Is that anything like what you mean by embedding a carriage return?

[Blackeagle]

Sort of BE.  The ini file section contains this

Code: [Select]

LoginTelnet=1
TelnetUsername=admin
TelnetPrompt=#
TelnetCmd=adslctl info --stats
It depends on how RS sends this data over telnet. If you were to change the last line to sh then that would drop to busybox, but wouldn't issue a further command (RS must put a carriage return on the end itself).  If you could send the command sh chr(13) xdslcmd info --stats then that would work, it is how to get the chr(13) embedded and sent as RS appears to type the command to the telnet session, therefore the actual characters will be typed, not the <CR>.

[Blackeagle]

Haha. got it plotting an RX SNR graph via telnet !!! 

How to do it.........

1) setup telnet login on experimental page as login with Admin and password.
2) Tick "Wait for password request"
3) Untick "Enable telnet...."
4) Change adslctl info --stats in the drop down box to xdslcmd info --stats.
5) Apply and save.
6) Start routerstats logging.
7) Goto the telnet page and click on the terminal tab.
8) Change "user defined1" to sh.

Routerstats should now go to the busybox shell and start logging downstream SNR !!

[asbokid]

I have checked a few PuTTy/Plink logs from the HG612 modem (not only my own) & in every case, Bit Errors has zero values.

I also had a look at xdslcmd bert --start 300 (5 minutes of monitoring):-

Code: [Select]

# xdslcmd bert --start 300

# xdslcmd bert --show
xdslcmd: BERT results:
BERT Status = RUNNING
BERT Total Time   = 300 sec
BERT Elapsed Time = 40 sec
BERT Bits Tested = 0x000000003F5CA480 bits
BERT Err Bits = 0x0000000000000000 bits
:
# xdslcmd bert --show
xdslcmd: BERT results:
BERT Status = RUNNING
BERT Total Time   = 300 sec
BERT Elapsed Time = 280 sec
BERT Bits Tested = 0x00000001BB885400 bits
BERT Err Bits = 0x0000000000000000 bits

# xdslcmd bert --show
xdslcmd: BERT results:
BERT Status = NOT RUNNING
BERT Total Time   = 300 sec
BERT Elapsed Time = 300 sec
BERT Bits Tested = 0x00000001DB369980 bits
BERT Err Bits = 0x0000000000000000 bits
#


That's a shame.. The bit error counter field from the BERT tests is populated in xdslcmd output from ADSL connections, but not for VDSL2 connections. Yet another Broadcom quirk.

With some basic arithmetic we could derive something close to the raw bit error rate from the other error counters.

The trouble with the error counters returned by the Reed Solomon decoder is that they somewhat mask the extent of the errors on the line, especially bursts of errors, as you might expect from impulse noise.

Only when the error count for a codeword exceeds the correction capabilities of the decoder is the error state categorised differently (RSUnCorrectable instead of RSCorrectable).  This doesn't tell us the exact number of errors in a RS codeword, a number which we would need to calculate the Bit Error Rate.

At a guess, the Bit Errors counter has been deliberately disabled in a kludge by Broadcom to maintain performance in VDSL2.  Perhaps at VDSL2 speeds, maintaining a raw bit error count would place too great a load on the DSP core which could badly affect performance.

Discovering whether a codeword is correctable by the RS decoder, with an error count that is within the correction capabilities of the decoder  (i.e with no more than (n-k)/2 symbol errors)  wouldn't be as demanding on resources as actually counting every single bit in error.

As soon as ((n-k)/2) + 1 symbol errors are discovered in a codeword by the RS decoder, with Broadcom's kludge applied, the decoder simply abandons the decoding, and returns an unquantified 'uncorrectable error' message to the caller and the RSUnCorr counter is increased by one.

The thinking is that once the decoder has discovered (n-k)/2 errors in a codeword - the maximum number of errors it can correct - it is pointless wasting CPU resources counting any further errors beyond that point (since the decoder can't correct them any way).  The kludge, if that is what it is, would result in the error counters being blunted in precision to the length of the RS codeword (255 bytes?)

In my (not yet released) script updates, Error Seconds & Serious Error Seconds data is harvested, for now only DS & US Error Seconds being plotted.

However, for monitoring noise levels, I imagine changes in SNRM values are perhaps the most relevant?

In general, stable & quiet connections seem to comprise more or less straight line graphs (even recently monitored low speed connections over much longer distances appear very stable).

My own connection currently seems stable(ish) at 29.5Mb sync speed.
I have no explanation why it was unstable for a few days & returned to relative stability again as no repair work was carried out (unless my connection simply prefers wet & cold weather).
Interleaving, INP & delay have been turned back on at fairly low levels so various error types are now visible (e.g. RSCorr), while other error types have diminished.

I see that both Josh's & BlackEagle's connections suffer from many "error" spikes, sudden changes in SNRM levels, stuck sync speeds & quite high levels of Interleaving etc.

For both those connections (as when mine plays up), no real timed pattern has been detected.

Maybe if the error counter data from each 24 hour period was over-plotted (and plotted with a vector-based format that allows for zooming), it could help to identify whether the noise was confined to the day (or night), and during certain hours thereof.   We might expect to find that noise from RFI ingress wains during the wee hours (e.g. 1am-6am) since many domestic electrical appliances are powered off during that period.

It is possible that bursts of interference/errors could occur anywhere between the 1 minute sampling, with only the cumulative error counts being visible in the harvested data.

The error counters appear to be updated every second.   

I believe Routerstats can log data right down to 5 second intervals.

I can ALMOST get Routerstats working with the HG612, but it incorrectly alternates between reporting SNRM & sync speed values in the same SNRM graph.

If anyone wants to have a go at setting Routerstats up, the login URL for the HG612's GUI data is http://192.168.1.1/html/status/xdslStatus.asp

The SNRM & Output Power values need dividing by 10 to display correct values to 1 decimal point.

I might some time have a go at converting my scripts to run continuously, harvesting data at much shorter intervals, as an alternative to only running them every minute via Windows Task Scheduler.

The PhD thesis of Nedko Nedev (Univ of Edinburgh, 2003) at [1] is on the Analysis of the Impact of Impulse Noise in Digital Subscriber Line Systems.   Nedev speaks of VDSL in the future tense, but his findings are probably just as relevant today.

Nedev defines impulse noise as...

Impulse noise is a non-stationary stochastic electromagnetic interference which consists of random occurrences of energy spikes with random amplitude and spectral content. The causes of impulse noise on the telephone line are diverse and vary from telephone on/off-hook events, through noise from home, office, and industrial electrical appliances, and transport vehicles, to atmospheric noise from electrical discharges.

We could expect all of those 'events', except perhaps the last (natural atmospheric noise) to reduce during the night.   Plotting the error rate (however it is defined) against the 24 hour clock should highlight any correlation of impulse noise and the hour of the day.

Nedev goes on to characterise impulse noise as...

The salient statistics of impulse noise include inter-arrival times, impulse durations, impulse amplitudes, and frequency spectrum

We can't measure several of those statistics using just the diagnostic data from the modem.  We have no way of measuring the amplitudes of the noise impulses.   And although the QLN data provides us with a frequency analysis of line noise, it is only a spectral snapshot of line noise from the initialisation phase. While very useful, it is limited to gauging time-invariant interference such as noise from crosstalk.

However, to a somewhat blunted degree of precision, the error counters from the modem could be used to estimate the inter-arrival times of impulse noise, and the impulse durations (to a precision of a second or two).  Not sure how useful that could be.  Interesting topic really 

cheers, a

[1] http://www.era.lib.ed.ac.uk/bitstream/1842/1366/1/Nedev.pdf

[JoshShep]

Little update, the REIN Engineer came today.

He inspected the house with his RF 444B Tester, and found a problem with the power supply for the modem, I heard the noise and it wasn't normal.

He put his tester against the drop wire just before the NTE5 and we heard the radio, he said this was quite normal, but installed a RF3 filter to reduce the RF interference.

So he installed the Filter, changed the modem and power supply and gave it the all clear, he didn't click that I originally was supplied with the ECI modem but installed a new Rev.3 HG612 anyway, I had the Rev.1 which he said were known for overheating and loosing sync, he also ran some tests with his JDSU and everything came back clear,

Lastly he went to the cab and made sure everything was fine there, and off he went.

No change in sync, but I now need a profile reset, I have requested one so should be within 24 hours.

Hopefully the power supply is what was causing the problems and things improve, really nice guy I must add!

I didn't expect anyone coming today as my isp never informed me. hence why I still had the HG612 connected 

Watch this space.

Cheers,

Josh

[Bald_Eagle1]

No change in sync, but I now need a profile reset, I have requested one so should be within 24 hours.

Hopefully the power supply is what was causing the problems and things improve, really nice guy I must add!

That all sounds pretty positive, Josh.

I'm not too sure about the DLM reset within 24 hours though, unless the 'rules' have been changed.

I believe the DLM reset should have been done while the engineer was still on site to request it & also check the result of the reset.

You might need another engineer's visit just to organise the DLM reset.

I presume the first task after the engineer left was to unlock the new HG612?

It will be interesting to see how the connection performs over the next week or so (once the reset has been done).