Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2 3 ... 8

Author Topic: Getting extra DSL stats from TP-Link modem/routers  (Read 161999 times)

ejs

  • Kitizen
  • ****
  • Posts: 2078
Getting extra DSL stats from TP-Link modem/routers
« on: September 01, 2014, 08:38:33 PM »

It should be possible to get more ADSL stats out of the TP-Link TD-W8970 than are shown in the web interface. The same method may work on similar TP-Link models. It can't get any per tone data, but it should get a few extra error counts.

I observed in wireshark while accessing the router web interface, and even just in that, I think you can see it retrieves more figures than the web page displays. The required names for further stats can be found looking in /etc/reduced_data_model.xml extracted from the firmware with binwalk.

A script to retrieve the stats is essentially one line, to run curl to make the necessary HTTP POST request, with the names of each item of requested data stored in a separate text file (attached). The value for the Authorization cookie could be copied out of wireshark, but it's just the value from http basic authorization (username:password base64 encoded).

An example script, the values for IP and AUTH (example is admin:password) would need to be changed as appropriate:
Code: [Select]
IP="192.168.1.1"
AUTH="YWRtaW46cGFzc3dvcmQ="
curl --data-binary "@td-w8970-adsl-stats-post-all.txt" --referer=http://${IP}/ --header "Cookie: Authorization=Basic ${AUTH}" --header "Content-Type: text/plain; charset=UTF-8" "http://${IP}/cgi?1&5"

Some names for some of the numbers can be found in /web/js/lib.js (again extracted from the firmware by binwalk).
It should be possible to get any value, or set any writable setting, in a similar way by making the appropriate HTTP POST request.
A "500 Internal Server Error" may be returned at random, this seems to happen occasionally while browsing the web interface anyway, it will probably work if you try it again.

I haven't connected the TD-W8970 to my phone line yet, so the list of names and values returned just has zeroes for most of the values.

Admin:  Edited topic title slightly because this also now covers several other TP-link models.
« Last Edit: August 31, 2015, 10:09:23 PM by kitz »
Logged

HighBeta

  • Reg Member
  • ***
  • Posts: 175
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #1 on: September 01, 2014, 09:32:08 PM »

Thanks for that.

The 8970 is a good lantiq vr9 unit.  {Same chipset at the ECI/r}
It will work on vdsl via a  vr9_firmware_bin swap mod  [{shh... as I think TP-Link reps read kitz } :))

dsl_cpe_consol --c        should be were the full line stats including tones are hiding  > bbsg 1 / bbsg 0.
:)
« Last Edit: September 01, 2014, 09:44:29 PM by HighBeta »
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #2 on: September 01, 2014, 10:21:51 PM »

Mine was supplied with the 131114 firmware, so the only way to install custom firmware or access a busybox shell would be via a serial connection. I suppose if you could access the busybox shell, then you wouldn't necessarily need to flash custom firmware, you could load VDSL stuff from a USB stick instead.

This method doesn't involve flashing a modified firmware and/or opening up the case and getting at the serial pins - there's only looking in the firmware file to see what's possible. So it's much more limited in what it can achieve, but it should be safer.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #3 on: September 12, 2014, 09:52:02 PM »

Ive come over to this thread from the TD-W9980 for which TP-Link have b@stardised the firmware and removed access not only to the shell but telnet has also been disabled in the latest f/w version.

Im not really sure what Im doing here when it comes to what can be done next.

Ive run a wireshark capture and can see various data, but aside from a few things such as FECs and CRCs I cant see anything much else of use.

Heres a snippet of the best stuff

Code: [Select]
134
[1,0,0,0,0,0]0
status=Up
modulationType=VDSL2
X_TPLINK_AdslModulationCfg=Multimode
upstreamCurrRate=20000
downstreamCurrRate=79995
X_TPLINK_AnnexType=Annex A/B/L/M
upstreamMaxRate=28292
downstreamMaxRate=82665
upstreamNoiseMargin=98
downstreamNoiseMargin=66
upstreamAttenuation=162
downstreamAttenuation=164

b1
[2,0,0,0,0,0]1
ATUCCRCErrors=70
CRCErrors=619
ATUCFECErrors=1305
FECErrors=12809
severelyErroredSecs=0
X_TPLINK_US_SeverelyErroredSecs=0
erroredSecs=0
X_TPLINK_US_ErroredSecs=0

b1
[3,0,0,0,0,0]1
ATUCCRCErrors=70
CRCErrors=619
ATUCFECErrors=1305
FECErrors=12809
severelyErroredSecs=0
X_TPLINK_US_SeverelyErroredSecs=0
erroredSecs=0
X_TPLINK_US_ErroredSecs=0

There are a few other bits and pieces scattered through the file such as Up time and the connection details ie my ISP login details... and the firware version stuff, but nothing else of much interest that I can see.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #4 on: September 13, 2014, 06:50:39 AM »

I'll try to write a small Java program to get the few extra stats that are available via these HTTP POST methods, which should be easier than fiddling about with curl, text files and shell scripts / batch files.

I don't know what's happened to telnet in the latest 9980 firmware, the firmware still contains the telnetd program, which is still started by the main init script. Not that the "cli" program that you're locked into offers much anyway. The reduced_data_model.xml file does have a setting which looks like it enables telnet but the default setting indicates it's enabled, there is also some sort of access control list settings for allowing certain IP addresses access to certain management interfaces. Although just because things are in the reduced_data_model.xml file doesn't actually mean they're in use, e.g. the TD-W8970 has various QoS items in the reduced_data_model.xml and config web pages, for prioritising traffic based on DSCP markings, but they don't work, there's only the bandwidth restrictions.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #5 on: September 13, 2014, 11:52:30 AM »

Quote
the firmware still contains the telnetd program, which is still started by the main init script. Not that the "cli" program that you're locked into offers much anyway. The reduced_data_model.xml file does have a setting which looks like it enables telnet but the default setting indicates it's enabled, there is also some sort of access control list settings for allowing certain IP addresses access to certain management interfaces.

Now that is an interesting point.  I mentioned in the other thread that when I first updated the firmware I was able to get telnet access to the CLI.
But now I cant. 


Quote
I'll try to write a small Java program to get the few extra stats that are available via these HTTP POST methods, which should be easier than fiddling about with curl, text files and shell scripts / batch files.

That would be great if you could.  Im sure anyone with the TD-W8970 / TD-W8980 / TD-W9980 would appreciate it :)
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #6 on: September 13, 2014, 12:06:51 PM »

Quote
Now that is an interesting point.  I mentioned in the other thread that when I first updated the firmware I was able to get telnet access to the CLI.
But now I cant.

Woah...  Im in  :clap:

I thought about what I could have done differently and the only thing I could think was me swapping the ethernet cable from this PC in between the TPLink and the Zyxel.   

I know the diagnostic tool sent to me from TPLink only works via LAN1 so thought Id try swapping ports.  I was on LAN1 from which I cant get access.  Swapped to LAN4  and still no telnet access..   tried from LAN2 LAN3.   success!!!!

Quote
there is also some sort of access control list settings for allowing certain....
Could there actually be anything in there that says eth1 ?

---------
Edited to add.

I was just switching cables without looking at the tiny print underneath which actually says LAN3 - not 2. 
and having just seen this.  It could be any eth'x' that you may be looking for.
« Last Edit: September 13, 2014, 06:34:31 PM by kitz »
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

adslmax

  • Kitizen
  • ****
  • Posts: 1937
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #7 on: September 13, 2014, 02:11:26 PM »

well spot kitz amazing LAN 2 port will work! thanks for let me know. Where did u get diagnostic tool from?
Logged

adslmax

  • Kitizen
  • ****
  • Posts: 1937
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #8 on: September 13, 2014, 03:42:31 PM »

IP="192.168.1.1"
AUTH="YWRtaW46cGFzc3dvcmQ="
curl --data-binary "@td-w8970-adsl-stats-post-all.txt" --referer=http://${IP}/ --header "Cookie: Authorization=Basic ${AUTH}" --header "Content-Type: text/plain; charset=UTF-8" "http://${IP}/cgi?1&5"

Sorry don't understand that bit. How to access it?
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #9 on: September 13, 2014, 09:01:49 PM »

Rename StatPOSTer.zip to StatPOSTer.jar and then you can run it by double clicking it in Windows, or by running "java -jar StatPOSTer.jar" from a command prompt / Linux terminal.

If choosing TD-W9980 for the model doesn't actually work, since I can't test that, then choosing TD-W8970 might work, it'll only be missing things like the VDSL profile value.

If you get a HTTP 500 - Internal Server Error, just click the same button again.

To use the bit for reading or writing settings, you'll need to extract everything in the firmware .bin file with binwalk, and refer to the /etc/reduced_data_model.xml file.

Don't blame me if you break your router using this program. It's called StatPOSTer because it works by making HTTP POST requests, in a similar fashion as the router's own config web pages do.

The few lines of shell script that adslmax quoted above was far less effort to write, and does pretty much the same thing, if you are happy using a command line.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #10 on: September 13, 2014, 10:10:24 PM »

It worked first time - thank you :)

Code: [Select]
2014.09.13 22:05:37 xDSL Stats:
[1,0,0,0,0,0]0
status=Up
modulationType=VDSL2
X_TPLINK_CurrSupportLinkType=2
lineEncoding=
allowedProfiles=
currentProfile=
dataPath=
interleaveDepth=0
lineNumber=0
upstreamCurrRate=20000
downstreamCurrRate=79995
upstreamMaxRate=28383
downstreamMaxRate=82458
upstreamNoiseMargin=98
downstreamNoiseMargin=65
upstreamAttenuation=162
downstreamAttenuation=164
upstreamPower=143
downstreamPower=42
ATURVendor=
ATURCountry=
ATUCVendor=
ATUCCountry=
totalStart=265047
showtimeStart=336
quarterHourStart=0
X_TPLINK_Bitswap=On
X_TPLINK_SRA=On
X_TPLINK_AdslModulationCfg=Multimode
X_TPLINK_AnnexType=Annex A/B/L/M
X_TPLINK_SupportAdslMode=VDSL2:A,B,A/B;T1.413:A;G.dmt:A;ADSL2:A,A/L/M;ADSL2+:A,M,A/L/M;Auto Sync-up:A/B/L/M
X_TPLINK_PtmMainIfName=ptm0
[2,0,0,0,0,0]1
receiveBlocks=0
transmitBlocks=0
cellDelin=0
linkRetrain=0
initErrors=0
initTimeouts=0
lossOfFraming=0
erroredSecs=0
X_TPLINK_US_ErroredSecs=0
severelyErroredSecs=0
X_TPLINK_US_SeverelyErroredSecs=0
FECErrors=12961
ATUCFECErrors=1522
HECErrors=0
ATUCHECErrors=0
CRCErrors=624
ATUCCRCErrors=79
[3,0,0,0,0,0]1
receiveBlocks=0
transmitBlocks=0
cellDelin=0
linkRetrain=0
initErrors=0
initTimeouts=0
lossOfFraming=0
erroredSecs=0
X_TPLINK_US_ErroredSecs=0
severelyErroredSecs=0
X_TPLINK_US_SeverelyErroredSecs=0
FECErrors=12961
ATUCFECErrors=1522
HECErrors=0
ATUCHECErrors=0
CRCErrors=624
ATUCCRCErrors=79
[error]0


Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

adslmax

  • Kitizen
  • ****
  • Posts: 1937
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #11 on: September 13, 2014, 10:46:51 PM »

Many thanks ejs - it worked on TD-W9980 and here is mine (24 hours VDSL):

Code: [Select]
2014.09.13 22:45:49 xDSL Stats:
[1,0,0,0,0,0]0
status=Up
modulationType=VDSL2
lineEncoding=
dataPath=
interleaveDepth=0
lineNumber=0
upstreamCurrRate=19999
downstreamCurrRate=79995
X_TPLINK_AnnexType=Annex A
upstreamMaxRate=33799
downstreamMaxRate=98408
upstreamNoiseMargin=150
downstreamNoiseMargin=114
upstreamAttenuation=127
downstreamAttenuation=113
upstreamPower=124
downstreamPower=2
ATURVendor=
ATURCountry=
ATUCVendor=
ATUCCountry=
totalStart=129241
showtimeStart=58
quarterHourStart=0
X_TPLINK_Bitswap=On
X_TPLINK_SRA=On
X_TPLINK_AdslModulationCfg=VDSL2
X_TPLINK_AnnexType=Annex A
X_TPLINK_SupportAdslMode=VDSL2:A,B,A/B;T1.413:A;G.dmt:A;ADSL2:A,A/L/M;ADSL2+:A,M,A/L/M;Auto Sync-up:A/B/L/M
[2,0,0,0,0,0]1
receiveBlocks=0
transmitBlocks=0
cellDelin=0
linkRetrain=0
initErrors=0
initTimeouts=0
lossOfFraming=0
erroredSecs=0
X_TPLINK_US_ErroredSecs=0
severelyErroredSecs=0
X_TPLINK_US_SeverelyErroredSecs=0
FECErrors=2305
ATUCFECErrors=296
HECErrors=0
ATUCHECErrors=0
CRCErrors=123
ATUCCRCErrors=25
[3,0,0,0,0,0]1
receiveBlocks=0
transmitBlocks=0
cellDelin=0
linkRetrain=0
initErrors=0
initTimeouts=0
lossOfFraming=0
erroredSecs=0
X_TPLINK_US_ErroredSecs=0
severelyErroredSecs=0
X_TPLINK_US_SeverelyErroredSecs=0
FECErrors=2305
ATUCFECErrors=296
HECErrors=0
ATUCHECErrors=0
CRCErrors=123
ATUCCRCErrors=25
[error]0



One more question how can we tell if VDSL2 become interleaved? (from the stats grabber)
« Last Edit: September 13, 2014, 11:39:05 PM by adslmax »
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #12 on: September 14, 2014, 12:04:12 AM »

One way is the prescense of FEC's, but this doesnt always apply with FTTC, since we are seeing on several lines Forward Error Correction without interleaving.

These are your FECs

Code: [Select]
FECErrors=2305
ATUCFECErrors=296

... but this is the most telling.

Code: [Select]
interleaveDepth=0


Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

adslmax

  • Kitizen
  • ****
  • Posts: 1937
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #13 on: September 14, 2014, 12:34:09 AM »

How many FECErrors in 24 hours before DLM was hit? Was mine 2305 is ok for 24 hours errors?
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #14 on: September 14, 2014, 07:25:18 AM »

While experimenting, I presumably crashed httpd and had to reboot my TD-W8970 to restore access to the web interface. Could still access telnet, but without access to the shell, there's no way to restart the httpd process, so I had to reboot the device.

I was expecting the 2 groups with CRC/FEC/HEC counts to cover different time periods - there are actually 3 groups available, the next version will show all 3, although I won't be surprised if all 3 are exactly the same.
Logged
Pages: [1] 2 3 ... 8