Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 [2] 3 4 ... 8

Author Topic: Getting extra DSL stats from TP-Link modem/routers  (Read 162008 times)

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #15 on: September 14, 2014, 11:19:14 AM »

Thank you for doing this - your time and effort is appreciated. It's a handy lightweight app that is so much easier than capturing packets to read :)

Yes I'd noticed that the groups on mine were returning data that looked the same.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #16 on: September 14, 2014, 06:10:39 PM »

New version:
StatPOSTer-20140914.jar

The new version doesn't need a model list box anymore. It's also much more useful for viewing settings, because if you leave the "key" name blank, the "Get value" or "Get list" buttons will list all the key/value pairs in that object or list.

In terms of viewing the settings, error 9003 is "invalid argument" (all the error codes and messages are in err.js), and often means you need to use "Get list" instead of "Get value" or vice versa. Error 9804 is "object not found" and probably means that the feature for that object was not built into the firmware.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #17 on: September 14, 2014, 08:53:33 PM »

Thank you ejs.   :)

First time I ran it I got a java error  pop up on my screen.
I then ticked 'Enable settings' and it worked fine.

Ive not been able to re-create the error, but I notice scrolling up

Code: [Select]
2014.09.14 20:43:39 xDSL Stats:
java.io.IOException: Server returned HTTP response code: 500 for URL: http://192.168.1.1/cgi?1&5
2014.09.14 20:43:45 xDSL Stats:

Since everything now appears to be working fine, Im assuming it was just 'one of those things'.


Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

vs1

  • Just arrived
  • *
  • Posts: 7
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #18 on: September 24, 2014, 02:31:38 AM »

Some stats from my faulty line:
Code: [Select]
ActualDataRate: 9.999 Mbit/s 18 Mbit/s
PreviousDataRate: 10 Mbit/s 12.474 Mbit/s
ActualInterleaveDelay: 5 ms 16 ms
ActualImpulseNoiseProtection: 3 sym 10 sym
ActualNetDataRate: 9.999 Mbit/s 18 Mbit/s
ActualImpulseNoiseProtectionRein: 0 sym 0 sym
ActualImpulseNoiseProtectionNoErasure: 3 sym 10 sym
LATN: 18.4 dB 17.6 dB
SATN: 18.3 dB 17.6 dB
SNR: 12.8 dB 22.6 dB
ATTNDR: 22.527437 Mbit/s 79.591552 Mbit/s
ACTPS: -901 -901
ACTATP: 12.0 dBm 5.7 dBm

There is a lot of work to do. There are many stats available and while some are documented most are not. I'm using lantiq source and ECI /r OpenWRT enhanced stats by Kronos_2001 and HighBeta (from the ECI /r thread on here) and also searching the internet for reference. The TP-Link W9980 firmware doesn't have expr or printf but it does have sed:
Code: [Select]
#!/bin/sh

divide() {
local a
local b

let a=$1/$2
let b=$1%$2

echo "$a.$b"
}

remove_trailing_zeros() {
echo "$1" | sed 's/\(\.[0-9]*[1-9]\)0*$/\1/;s/\.0*$//'
}

# Mbit/s
scale_mbit() {
local a
local b

if [ "$1" -gt 1000000 ]; then
let a=$1/1000000
let b=$1%1000000
echo "$(remove_trailing_zeros "$a.$b") Mbit/s"
elif [ "$1" -gt 1000 ]; then
let a=$1/1000
let b=$1%1000
echo "$(remove_trailing_zeros "$a.$b") kbit/s"
else
echo "$1 bit/s"
fi
}

# kbit/s
scale_kbit() {
local a
local b

if [ "$1" -gt 1000 ]; then
let a=$1/1000
let b=$1%1000
echo "$(remove_trailing_zeros "$a.$b") kbit/s"
else
echo "$1 bit/s"
fi
}

stats_cmd() {
echo "$@" > /tmp/pipe/dsl_cpe0_cmd
cat /tmp/pipe/dsl_cpe0_ack
}

stats_val() {
echo "$1" | sed -n 's/.*'$2'=\([-\.[0-9][0-9]*\).*/\1/p'
}

stats_g997csg() {
local su sd
local vu vd

su=$(stats_cmd g997csg 0 0)
sd=$(stats_cmd g997csg 0 1)

vu=$(stats_val "$su" ActualDataRate)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ActualDataRate)
[ -z "$vd" ] && vd=0
echo "ActualDataRate: $(scale_mbit $vu) $(scale_mbit $vd)"

vu=$(stats_val "$su" PreviousDataRate)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" PreviousDataRate)
[ -z "$vd" ] && vd=0
echo "PreviousDataRate: $(scale_mbit $vu) $(scale_mbit $vd)"

vu=$(stats_val "$su" ActualInterleaveDelay)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ActualInterleaveDelay)
[ -z "$vd" ] && vd=0
echo "ActualInterleaveDelay: $(remove_trailing_zeros $(divide $vu 100)) ms $(remove_trailing_zeros $(divide $vd 100)) ms"

vu=$(stats_val "$su" ActualImpulseNoiseProtection)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ActualImpulseNoiseProtection)
[ -z "$vd" ] && vd=0
echo "ActualImpulseNoiseProtection: $(remove_trailing_zeros $(divide $vu 10)) sym $(remove_trailing_zeros $(divide $vd 10)) sym"

vu=$(stats_val "$su" ActualNetDataRate)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ActualNetDataRate)
[ -z "$vd" ] && vd=0
echo "ActualNetDataRate: $(scale_mbit $vu) $(scale_mbit $vd)"

vu=$(stats_val "$su" ActualImpulseNoiseProtectionRein)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ActualImpulseNoiseProtectionRein)
[ -z "$vd" ] && vd=0
echo "ActualImpulseNoiseProtectionRein: $(remove_trailing_zeros $(divide $vu 10)) sym $(remove_trailing_zeros $(divide $vd 10)) sym"

vu=$(stats_val "$su" ActualImpulseNoiseProtectionNoErasure)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ActualImpulseNoiseProtectionNoErasure)
[ -z "$vd" ] && vd=0
echo "ActualImpulseNoiseProtectionNoErasure: $(remove_trailing_zeros $(divide $vu 10)) sym $(remove_trailing_zeros $(divide $vd 10)) sym"
}

stats_g997lsg() {
local su sd
local vu vd

su=$(stats_cmd g997lsg 0 1)
sd=$(stats_cmd g997lsg 1 1)

vu=$(stats_val "$su" LATN)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" LATN)
[ -z "$vd" ] && vd=0
echo "LATN: $(divide $vu 10) dB $(divide $vd 10) dB"

vu=$(stats_val "$su" SATN)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" SATN)
[ -z "$vd" ] && vd=0
echo "SATN: $(divide $vu 10) dB $(divide $vd 10) dB"

vu=$(stats_val "$su" SNR)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" SNR)
[ -z "$vd" ] && vd=0
echo "SNR: $(divide $vu 10) dB $(divide $vd 10) dB"

vu=$(stats_val "$su" ATTNDR)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ATTNDR)
[ -z "$vd" ] && vd=0
echo "ATTNDR: $(scale_mbit $vu) $(scale_mbit $vd)"

vu=$(stats_val "$su" ACTPS)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ACTPS)
[ -z "$vd" ] && vd=0
echo "ACTPS: $vu $vd"

vu=$(stats_val "$su" ACTATP)
[ -z "$vu" ] && vu=0
vd=$(stats_val "$sd" ACTATP)
[ -z "$vd" ] && vd=0
echo "ACTATP: $(divide $vu 10) dBm $(divide $vd 10) dBm"
}

stats_g997csg
stats_g997lsg

To test the script:
Code: [Select]
/tmp # tftp -g -r test.sh 192.168.1.100 && sh test.sh
« Last Edit: September 24, 2014, 02:35:25 AM by vs1 »
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #19 on: October 09, 2014, 08:20:30 PM »

I thought I'd better make a new version to suppress the duplicated stats output. Here it is:
StatPOSTer-20141009.jar

It turns out that the program might work on any TP-Link modem/router that has a web interface that works the same way.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #20 on: October 09, 2014, 09:29:52 PM »

Well done ejs.   I cant test it though atm as Im using a different router..  but good work :)
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

impossible

  • Just arrived
  • *
  • Posts: 6
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #21 on: February 01, 2015, 08:59:45 AM »

Thanks for providing this ejs, I your jar whilst looking for an alternative to HG612_Modem_stats to use with my TPlink 9980.

If you're taking requests, would it be possible for your jar to;

Remember router credentials
run the jar with switches which would provide a file output of the retrieved info with the filename as datetime.txt

I hope to run it on a cron which would record my modem stats over the course of a month etc.


Imp
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #22 on: February 01, 2015, 01:36:14 PM »

I was not really intending to develop the Java program any further because it seemed very easy to crash the web interface, and the stats available are quite limited. I have been exploring other options but making no progress.

If you can flash the firmware via the serial port, one possibility would be to modify the firmware, and edit the /etc/init.d/rcS file to contain "telnetd -l /bin/login" instead of just "telnetd". I think the user/pass is root/1234 to login at that level. That would get shell access via telnet, with access to the Lantiq dsl_cpe_control program for full DSL stats.

Otherwise, rather than running a jar file via cron, you could just use small shell script:
Code: [Select]
#!/bin/bash
# script to get more detailed xDSL stats from TP-Link web interface

# standard HTTP basic authentication, but set in cookie
USERNAME="yourusername"
PASSWORD="yourpassword"
AUTH=$(echo -n "${USERNAME}:${PASSWORD}" | base64)

IP="192.168.1.1"

POSTDATA=$'[WAN_DSL_INTF_CFG#1,0,0,0,0,0#0,0,0,0,0,0]0,0\r
[WAN_DSL_INTF_STATS_TOTAL#1,0,0,0,0,0#0,0,0,0,0,0]1,0\r
'

curl \
    --data-binary "${POSTDATA}" \
    --referer "http://${IP}/" \
    --header "Cookie: Authorization=Basic ${AUTH}" \
    --header "Content-Type: text/plain; charset=UTF-8" \
    "http://${IP}/cgi?1&1"
Logged

impossible

  • Just arrived
  • *
  • Posts: 6
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #23 on: February 01, 2015, 09:38:52 PM »

Thanks for the response.

I do currently have serial access, I will give it a bash over the next few days.

Thanks for the suggestion.

Imp

Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #24 on: February 15, 2015, 08:49:31 PM »

I finally managed to get proper shell access, without opening the case. It was made possible by the person who figured out how to decrypt the saved config file. Standing on the shoulders of that, it was then merely a matter of working out where in the config to tack on a command to start telnet with the right parameters, hoping it gets copied onto a command line and executed, then adjusting the modified config file md5 checksum and padding, re-encrypting the config file, and uploading it. The root user/pass is admin/1234.

Quote
~ # cat /proc/version 
Linux version 2.6.32.32 (swd@localhost.localdomain) (gcc version 4.3.3 (GCC) ) #34 Fri Jun 13 08:36:06 CST 2014

~ # cat /proc/cpuinfo
system type             : VR9
processor               : 0
cpu model               : MIPS 34Kc V5.6
BogoMIPS                : 332.59
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 16
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 4, address/irw mask: [0x0ff8, 0x0ffc, 0x0ffb, 0x0ff8]
ASEs implemented        : mips16 dsp mt
shadow register sets    : 1
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

~ # free
             total         used         free       shared      buffers
Mem:         60888        38280        22608            0         3748
-/+ buffers:              34532        26356
Swap:            0            0            0

~ # dsl_cpe_control --console

DSL_CPE#>vig
nReturn=0 DSL_DriverVersionApi=4.11.4 DSL_ChipSetFWVersion=5.4.4.4.0.1 DSL_ChipSetHWVersion=Unknown DSL_ChipSetType=Lantiq-VRx DSL_DriverVersionMeiBsp=1.2.0 nHybrid=1

DSL_CPE#>lfcg 0
nReturn=0 nDirection=0 bTrellisEnable=1 bBitswapEnable=1 bReTxEnable=0 bVirtualNoiseSupport=0 b20BitSupport=-1

DSL_CPE#>lfcg 1
nReturn=0 nDirection=1 bTrellisEnable=1 bBitswapEnable=1 bReTxEnable=0 bVirtualNoiseSupport=0 b20BitSupport=0
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #25 on: February 28, 2015, 08:32:59 AM »

I've completed the initial coding to add config file decryption and re-encryption ability to the StatPOSTer program, so I should be able to release a new version within the next few days.

An inconvenience with the trick I'm using to gain shell access is that it gets removed from the config whenever you change any setting. And adding the trick back requires restoring the config file, which requires rebooting the device. And of course TP-Link could stop it working in newer firmware versions.

The TP-Link firmware keeps most processes running even though the relevant feature might be disabled, so there are a few things that could be killed off, depending on what you want to use:
Code: [Select]
killall cwmp
killall dyndns
killall noipdns
killall snmpd
killall ushare

Also the SNMP port appears to be open in the firewall:
Code: [Select]
~ # iptables -L -v -n
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           /* loop back */
  294 15488 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    4   168 ACCEPT     all  --  *      *       0.0.0.0/0            224.0.0.0/4         
    0     0 ACCEPT     udp  --  !br+   *       0.0.0.0/0            0.0.0.0/0           udp spt:67 dpt:68
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:161
    3   156 ACCEPT     all  --  br+    *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     icmp --  ppp1   *       0.0.0.0/0            0.0.0.0/0           icmp type 8

Which can be closed:
Code: [Select]
~ # iptables -D INPUT -p udp --dport 161 -j ACCEPT
Logged

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #26 on: March 01, 2015, 07:12:50 PM »

StatPOSTer-20150301.jar

Latest version fixes a bug with the "Auto" stats setting when encountering the HTTP 500 errors, and adds the config file decryption and encryption functions. The config file decryption automatically removes the MD5 checksum bytes and padding, and the encryption routine calculates the new MD5 checksum and adds the padding, so it might even save myself some time as I won't have to do that manually with a hex editor.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #27 on: March 01, 2015, 11:43:22 PM »

Excellent progress ejs.   Thank you for sharing your findings with us. :)

Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

ejs

  • Kitizen
  • ****
  • Posts: 2078
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #28 on: March 15, 2015, 09:38:04 PM »

Lots of commands, some of the more useful ones:
Code: [Select]
~ # /firmware/dsl_cpe_pipe.sh g997csg 0 0
nReturn=0 nChannel=0 nDirection=0 ActualDataRate=888000 PreviousDataRate=0 ActualInterleaveDelay=25 ActualImpulseNoiseProtection=2

~ # /firmware/dsl_cpe_pipe.sh g997csg 0 1
nReturn=0 nChannel=0 nDirection=1 ActualDataRate=3900000 PreviousDataRate=0 ActualInterleaveDelay=75 ActualImpulseNoiseProtection=12

~ # /firmware/dsl_cpe_pipe.sh g997lsg 0 1
nReturn=0 nDirection=0 nDeltDataType=1 LATN=373 SATN=373 SNR=50 ATTNDR=1100397 ACTPS=0 ACTATP=128

~ # /firmware/dsl_cpe_pipe.sh g997lsg 1 1
nReturn=0 nDirection=1 nDeltDataType=1 LATN=582 SATN=581 SNR=28 ATTNDR=3872000 ACTPS=0 ACTATP=185

~ # /firmware/dsl_cpe_pipe.sh pmcctg 0 0
nReturn=0 nChannel=0 nDirection=0 nElapsedTime=50370 bValid=1 nCodeViolations=455 nFEC=84050

~ # /firmware/dsl_cpe_pipe.sh pmcctg 0 1
nReturn=0 nChannel=0 nDirection=1 nElapsedTime=50370 bValid=1 nCodeViolations=0 nFEC=0

~ # /firmware/dsl_cpe_pipe.sh pmlsctg 0
nReturn=0 nDirection=0 nElapsedTime=50370 bValid=1 nES=323 nSES=0 nLOSS=0 nUAS=58 nLOFS=0

~ # /firmware/dsl_cpe_pipe.sh pmlsctg 1
nReturn=0 nDirection=1 nElapsedTime=50370 bValid=1 nES=0 nSES=0 nLOSS=0 nUAS=58 nLOFS=0

~ # /firmware/dsl_cpe_pipe.sh g997fpsg 0 0
nReturn=0 nChannel=0 nDirection=0 nNFEC=12 nRFEC=12 nLSYMB=240 nINTLVDEPTH=1 nINTLVBLOCK=12 nLPATH=0

~ # /firmware/dsl_cpe_pipe.sh g997fpsg 0 1
nReturn=0 nChannel=0 nDirection=1 nNFEC=10 nRFEC=10 nLSYMB=1058 nINTLVDEPTH=32 nINTLVBLOCK=10 nLPATH=0

Main issue is the CRC and FEC counters always at zero for what I think is the upstream. nDirection=0 obviously is referring to the upstream for some commands, but I looked in the dsl_cpe_control source code which says nDirection=0 means "near end" for pmcctg (PM_ChannelCountersTotalGet).

But the TP-Link layer of the firmware, stats taken at a different time, has given:
Code: [Select]
FECErrors=0
ATUCFECErrors=51965
HECErrors=0
ATUCHECErrors=0
CRCErrors=0
ATUCCRCErrors=312

I can't believe there could possibly have been no CRC or FEC errors on the upstream all day, so I suppose it's just a bug. I've been polling the stats at 10 second intervals, upstream SNR and attainable rate have been constant all day too.
Logged

Intikhab

  • Just arrived
  • *
  • Posts: 3
Re: Getting extra DSL stats from TP-Link TD-W8970
« Reply #29 on: March 31, 2015, 08:46:10 PM »

I have adsl router td-w8951nd and today I bought td-w8970. Will it work with 51nd model too? Also haven't yet setup td-8970 v1.2 . should I upgrade it's firmware straight to latest one before trying this kind of injection? I am very interested to use this software  and see how much more I can tweak with my routers and also use 8970 when I get my vdsl connection in future :) but I have no idea how to use the gui of the jar file, err could there be some tutorial or instructions? I am CS student so will learn everything you tell me to? I understand programming but new to this field altogether.
Logged
Pages: 1 [2] 3 4 ... 8