Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[BoredBoy]

I have one of these and was wanting to use CLI to poll the internet traffic volumes, but I can't find anything that will show it to me.

"atm debug gstats clear=disabled" shows me this continuously:

        # of received octets = 0.
        # of transmitted octets = 0.
        # of received cells = 0.
        # of transmitted cells = 0.
        # of unknown cells = 0.
        # of errors on the input = 0.
        # of errors on output = 0.

portstats is pretty much the same.

"adsl info" has no usage that I can see.

But the router's browser interface "Broadband Connection" page tells me

DSL Connection:

    Data Transferred (Sent/Received) [kB/GB]:   0,00 / 211,99

Internet:

     Data Transferred (Sent/Received) [MB/MB]:   46,64 / 370,83

Where the $%(*& can I get the same numbers out?

(I need to do this because I want to verify the ISP usage meter against what my router thinks I'm doing)

Thanks for any help
Andrew

BTW: Firmware revision is 7.4.2.7, Variant BM...

[AdrianH]

Welcome Andrew,

I am not sure about the stats with Thompson routers but have you looked at a separate app?  I run this http://www.softperfect.com/products/networx/  on my machines here and it  gives me a perfect breakdown of traffic.

[geep]

I use an ST546 which has similarish interface. What you could do is use the wget command and parse the resultant file. I'd use Perl for that. You just give wget the same URL that you use to browse the TG585.

On Linux you should already have wget, and probably Perl too.
For Windows: wget @ http://gnuwin32.sourceforge.net/packages/wget.htm and Perl @ http://www.activestate.com/activeperl

When I do:
wget http://192.168.1.254/cgi/b/is/?be=0&l0=1&l1=1
I find this line within the returned file index.html?be=0
<tr><td width='170'>Data Transferred (Sent/Received) [GB/GB]:</td><td colspan='3'>1.30 / 17.44</td></tr>

Maybe this is food for thought and/or experimentation.

Cheers,
Peter

[BoredBoy]

I am not sure about the stats with Thompson routers but have you looked at a separate app?  I run this http://www.softperfect.com/products/networx/  on my machines here and it  gives me a perfect breakdown of traffic.

Thanks for the link - I hadn't found it in my searching. It's hard to tell from the website (and without more time - I'm at work ) but it looks like this only looks at traffic on the PC, not from the router.  Since I have other devices using the internet (laptop, another desktop, HTPC, and smart phone covers most of them ) I need to get the actual usage from the router since it's the only place that the total is shown...

I'd use Perl for that.

Me too  

You just give wget the same URL that you use to browse the TG585

In fact, I already have an app in perl that used to work uses the CLI interface via Net::Telnet, but somebody asked about it recently, so I'm trying to resurrect it.  I can't actually remember, but I don't think I've run it since upgrading from my previous SpeedTouch (can't remember the model, unfortunately), so I assume it has never worked with the 585v7.

The CLI app was working although not very robustly.  The problem with the screen-scraping solution is that it isn't very high resolution (4 sig. digits?); once you get into GB, if you're just using MB it doesn't show up quick enough.  The app is configurable and currently works for two routers, but is fairly hard-coded for them.  Extension was Coming Later

The CLI interface used to give me results in Bytes, but now it seems to be always zero...

Any more ideas?

[geep]

Yes.
I just found a new, to me, CLI command: ip iflist

Code: [Select]

#!/usr/bin/perl 
use Net::Telnet (); 
# Set to your ST5x6 IP address or hostname (if any) 
$host = '192.168.1.254'; 
# Set to your ST5x6 administator user id (Administrator or admin or whatever) 
$user = 'Administrator'; 
# Set to your ST5x6 administator password (only you know what this is) 
$pass = ''; 
   
$t = new Net::Telnet ( Timeout => 10, 
                        Prompt => '/.*=>$/i', 
                        Errmode => 'die'); 
   
# Open host 
$t->open($host); 
   
# Login 
$t->login($user, $pass); 
   
# Command in modem
@result = $t->cmd(':ip iflist'); 
print "ip iflist gives:\n@result\n\n";

@result = $t->cmd(':ip iflist expand=enabled'); 
print "ip iflist expand=enabled gives:\n@result\n\n";

@result = $t->cmd(':atm debug gstats'); 
print "atm debug gstats gives:\n@result\n\n";
 
# Logout 
$t->close(); 

print "Screen scrape with curl gives:\n";
system("curl -s 'http://192.168.1.254/cgi/b/is/?be=0&l0=1&l1=1' | grep 'Data Transferred'");

exit;
on my ST546 gives:

Code: [Select]

ip iflist gives:
Interface                            Group  MTU   RX         TX         Admin  Oper
 0   loop. . . . . . . . . . . . . .  local  65535 111822354  55522892   UP     [UP]   
 1   Internet. . . . . . . . . . . .  wan    1500  1701879666 1437143464 UP     UP     
 2   LocalNetwork. . . . . . . . . .  lan    1500  1478591358 1791456171 UP     [UP]   


ip iflist expand=enabled gives:
Interface                            Group  MTU   RX         TX         Admin  Oper
 0   loop. . . . . . . . . . . . . .  local  65535 111822836  55523075   UP     [UP]   
     HW-address    : xx:xx:xx:xx:xx:xx
     BRHW-address  : ff:ff:ff:ff:ff:ff
     RX unicastpkts: 711265  brcastpkts : 26      mcastpkts : 14992 
     TX unicastpkts: 746066  brcastpkts : 866     mcastpkts : 1544    droppkts:0     
     Oper state    : UP      Admin State: UP     
     Flags         : LOOP MULTICAST INTERNAL
 1   Internet. . . . . . . . . . . .  wan    1500  1701879666 1437143464 UP     UP     
     RX unicastpkts: 16269779 brcastpkts : 0       mcastpkts : 0     
     TX unicastpkts: 11165788 brcastpkts : 0       mcastpkts : 0       droppkts:0     
     Oper state    : UP      Admin State: UP     
     Flags         : BOUND PTP MULTICAST LINKSENSING NAT
 2   LocalNetwork. . . . . . . . . .  lan    1500  1478591541 1791456653 UP     [UP]   
     HW-address    : xx:xx:xx:xx:xx:xx
     BRHW-address  : ff:ff:ff:ff:ff:ff
     RX unicastpkts: 11770061 brcastpkts : 7349    mcastpkts : 3929   
     TX unicastpkts: 16823603 brcastpkts : 26      mcastpkts : 14988   droppkts:0     
     Oper state    : UP      Admin State: UP     
     Flags         : PRIMARY ARP BROADCAST BOUND ARPTABLE MULTICAST NAT TRANSNAT STATIC


atm debug gstats gives:
        # of received octets = 1378.
        # of transmitted octets = 0.
        # of received cells = 26.
        # of transmitted cells = 0.
        # of unknown cells = 26.
        # of errors on the input = 0.
        # of errors on output = 0.


Screen scrape with curl gives:
<tr><td width='170'>Data Transferred (Sent/Received) [GB/GB]:</td><td colspan='3'>1.33 / 17.58</td></tr>



It seems that ip iflist shows data transmitted/received. But It doesn't seem to match with the data displayed by the web interface.

Also, this article is interesting: http://www.dslreports.com/forum/r21063176-Line-stats-monitoring-with-RRDTool-or-MRTG
I based my own monitoring on this script, plus several others. It growed like Topsy: http://pgg999.co.uk/stats/

Cheers,
Peter

[BoredBoy]

I just found "ip iflist" too, then came to report it here, and found you've beaten me to it   

It's weird that your numbers don't match - with mine, they're a lot closer.

I get:

1   Internet. . . . . . . . . . . .  wan    1500  690099291  122847610  UP     UP

which is  658.1299695968628 Rx and 117.1566104888916 Tx
Which (allowing for more traffic happening all the time) is pretty darn close to the numbers shown in

Data Transferred (Sent/Received) [MB/MB]:   117,17 / 658,15

Not exact, but pretty close.  Yours are way out, though - I don't know how curl works (I'm on Win7 here (no flames please )), but could there be some subtle caching going on or something techo like that? 

Or maybe there's a problem when volume gets too high?  I'm still only down in the MB's, so it'll be interesting to see what happens when I get higher...  (They wouldn't be storing the volume in 32 bits, surely - not when we can do TB of traffic...)

I'll try using it for a while anyway; I'll enhance the script to get the web version as well, and see If I can get some clues as to what's happening.

Thanks for the help, BTW...

[geep]

Hmm. All this has got me thinking about my own logging. I log usage figures from my ISP - screen scrape every 30 minutes.  The ISP figures and the router figures don't seem to align. As I'm on an unlimited use contract it's all a bit academic, but I do like my monitoring to be correct. Here's what I've now found...

So your calculations are:

Code: [Select]

RX 690,099,291 Bytes / (1024 * 1024) = 658.1299696 MB from CLI and 658.15 MB from web page
TX 122,847,610 Bytes / (1024 * 1024) = 117.1566105 MB from CLI and 117.17 MB from web page
Applying similar to my figures I get:

Code: [Select]

RX 1,701,879,666 Bytes / (1024 * 1024 * 1024) = 1.584998952 GB from CLI and 17.58 GB from web page.
Aha! I spot a 16GB error! Seems that there could have been an overflow.

TX 1,437,143,464 Bytes / (1024 * 1024 * 1024) = 1.338444151 GB from CLI and 1.33 GB from web page.
Essentially the same!
I wonder if there is really an overflow, or is the apparent 16 GB error just a coincidence? If there was an overflow, how does the web interface know what to do? It should never display > 4 GB.

32 bit unsigned max number is 4,294,967,295. If the RX counter overflowed 4 times, then the error should presumably be:

Code: [Select]

4,294,967,295 * 4 = 17,179,869,180 Bytes / (1024 * 1024 * 1024) = 16 GB
My Web interface currently (23 June 17:00) shows:

Code: [Select]

Type: PPPoA
Uptime: 5 days, 7:54:10
IP Address: 109.170.137.226
Data Transferred (Sent/Received) [GB/GB]: 1.41 / 19.35
Looking at the usage figures logged every 30 minutes (when PC is running) from my ISP. The closest entries to router reset time in my log of ISP data shows:

Code: [Select]

Date    Time RX TX
2011-06-23 17:00:12 28.192e9 1.980e9
2011-06-18 09:30:11 18.723e9 1.468e9
gives usage according to ISP of:
     5 days 7:30:01 9.469e9 0.512e9
So my router's RX/TX figures seem to bear no resemblance to the ISP's figures. Neither from the router's web interface, nor from the CLI figures.
Dunno what to think.

If RX/TX figures are limited to 32 bits in ST546's CLI, maybe your newer TG585 won't suffer from it.

Will be interested to hear how your monitoring goes, and if you can line up what your router says with what your ISP says.

Cheers,
Peter

[BoredBoy]

I think the 4GB overflow has become a bit more likely - a friend who was using my script until earlier this year when he moved and "upgraded" to a Thomson (which broke the script, but he didn't bother mentioning it ) has had a similar experience to yours.  I sent him the "ip iflist" and modified regexp required by the Thomson (I think he said his new modem is an ST5x5v6), and he gets these:

Code: [Select]

RX 1,310,535,716 Bytes / (1024 * 1024 * 1024) = 1.22 GB from CLI and 5.21 GB from web page.Exactly 4GB difference...  Now there's a surprise...

Code: [Select]

TX 851,830,849 Bytes / (1024 * 1024) = 812.37 MB from CLI and 811.75 MB from web page.Again, essentially matching.

I wonder if there is really an overflow, or is the apparent 16 GB error just a coincidence? If there was an overflow, how does the web interface know what to do? It should never display > 4 GB.

That is what's annoying the **** out of me, actually - if it knows, why can't it tell us?

I assume that they have an overflow counter that gets increment appropriately, and the web interface is adding it back in to compensate. Or even another 32-bit word they were too lazy to include in the CLI output? Can you see a "4" in any of the output anywhere?

32 bit unsigned max number is 4,294,967,295. If the RX counter overflowed 4 times, then the error should presumably be:

Code: [Select]

4,294,967,295 * 4 = 17,179,869,180 Bytes / (1024 * 1024 * 1024) = 16 GB

My thoughts exactly.

My Web interface currently (23 June 17:00) shows:

[interesting numbers removed for brevity]

So my router's RX/TX figures seem to bear no resemblance to the ISP's figures. Neither from the router's web interface, nor from the CLI figures.
Dunno what to think.

Has (a) your router been reset, or (b) your billing period restarted at the ISP?  Either of these will throw actual values off.

The script I had built for the other guy used "atm debug gstats clear=enabled" to get the data increments, which was then loaded into an Access database that he constructed to make pretty graphs etc.

He says his recollection is that it was pretty close to the ISP, and this would have covered a period when he was on an unlimited plan, so we're not talking small volumes here

So maybe the only way round this is to force the router to reset the counts whenever one of them approaches 4GB, or keep track of incrementals, rather than the actuals?  BTW - did you find a way to reset the "ip iflist" counts without resetting the router? 

I need to add the router uptime to my logging so I can see when there's been a reset...

If RX/TX figures are limited to 32 bits in ST546's CLI, maybe your newer TG585 won't suffer from it.

Will be interested to hear how your monitoring goes, and if you can line up what your router says with what your ISP says.

The ST5x5v6 seems to have the problem, so I'm not too hopeful for the v7, but it'd be nice. Unfortunately, by coincidence my router reset itself while I was playing with other commands trying to get more details (specifically "service settime" if it matters - looking for the uptime) so I'm back to zero on the counts again.  Since I get throttled to dial-up speeds at 20GB, I'm not too keen on blowing my cap in the interests of science   The last day of my billing period is the 4th, so if I remember I'll keep an eye on it round then (I can go over the 20GB in the last 24 hours without being penalised.  Not that I'd ever exploit a hole like that, of course...)

I am logging both the CLI and web numbers now (60 second intervals until I see how much disk it takes up, running on the HTPC which is usually on 24x7), so I'll have a good set of data to work with.  I was considering scraping the ISP website, but the most resolution I can get from there is daily anyway, and it's always available, so I probably won't bother.

I'm sure we'll be able to confirm or deny the rollover problem at least, even we can't find a nice solution.  I'll keep updating here if I find anything new...

Andrew

[geep]

A few follow-ups, but no solution to the overflow...

Has (a) your router been reset, or (b) your billing period restarted at the ISP?  Either of these will throw actual values off.

Took account of this when I did my comparison. My ISP (ADSL24) resets the upload/download counters on the 1st of the month, then updates every 30 mins. I do a screen-scrape every 30 mins.

atm debug gstats

Don't understand what it tells me:

Code: [Select]

gstats           : Display ATM global statistics.

{Administrator}[atm debug]=>gstats
        # of received octets = 1590.
        # of transmitted octets = 0.
        # of received cells = 30.
        # of transmitted cells = 0.
        # of unknown cells = 30.
        # of errors on the input = 0.
        # of errors on output = 0.
Can't find anywhere to reset "ip iflist" counts.

Don't properly understand the ST546 web interface - there are 2 flavours of "Data Transferred". The first one, presently 0, often shows lowish values - KB rather than GB.

DSL Connection shows:

Code: [Select]

Link Information
Uptime: 1 day, 17:42:39
Modulation: G.992.5 annex A
Bandwidth (Up/Down) [kbps/kbps]: 1,023 / 7,586
Data Transferred (Sent/Received) [KB/KB]: 0.00 / 0.00
Internet Services shows:

Code: [Select]

View more...
Type: PPPoA
Uptime: 8 days, 2:50:03
IP Address: xxx.xxx.xxx.xxx
Data Transferred (Sent/Received) [GB/GB]: 1.67 / 23.13
Have been logging lots of stuff every 30 secs for a couple of years now, as you may have seen on my website http://pgg999.co.uk/stats/. It doesn't take too much space. Some few hundred Mbytes, and that's with some old, duplicated, redundant unzipped raw data. The website has much less data, stored in .json format.

Been playing with RRDTool. I found it a bit difficult to understand at first. Still do. Its consolidation is interesting, but the results can be confusing or misleading unless I'm thinking VERY carefully. There's a snapshot of this work-in-progress here: http://pgg999.co.uk/rrdstats/. Eventually will updated every 5 mins or so while PC is running.

Must go out now - lunch then visiting open gardens in Harlow. Beautiful day here...

Cheers,
Peter

[BoredBoy]

atm debug gstats (adg):

This used to be where I got my usage from - the octets transmitted/received matched the numbers on the ISP website, and the router web interface pretty closely. 

However, since I built the script originally, I have (a)  changed ISP's, (b) changed to a new router, and (c) been migrated from DSL to ADSL2+.  Any one of these could have been what caused the change in logging.

Although the first ISP (IHUG) was leasing the copper from the second (Telecom/Xtra) so this on its own probably didn't change anything.  When I switched, they gave me the new router, and I decided to use it since I had a minor quibble with the old one (something funny about connecting to the WLAN).

"DSL Connection" vs "Internet":

Extrapolating from the numbers, I have kind of decided that "DSL Connection:" is the low-level protocol counts, and "Internet" shows the effective traffic volumes.  My DSL Connection numbers behave very strangely - when I started looking at this again, they showed 0MB/211GB at the same time that the Internet numbers were 0MB/0MB (just after a reset).  I'd also swear that after a while they went  back to 0/211GB 

My DSL connection "always" shows a slightly longer duration than the "Internet".  Does your DSL duration get reset when the ISP resets the connection, but without affecting the Internet duration?  It looks a bit strange comapred to what I've seen...

I think there's a whole lot more I would need to understand about all this before I can interpret the CLI data properly - the Thomson manual isn't very much help, as you're probably aware...

I had a look at your website - it's way above what I understand at present   More research is required...

We're wintering at the moment - rare sunny patches, but cold still...

Cheers,
Andrew...

[geep]

My DSL connection "always" shows a slightly longer duration than the "Internet"

Yes. By just a few seconds.

Does your DSL duration get reset when the ISP resets the connection, but without affecting the Internet duration?

No, the other way round.  I just tried it. My ISP's toolbox has the facility to force a reconnect. (I hope I'm using the correct uptime in my monitoring!).

Code: [Select]

Kick session.
This page allows you to kick your current PPP session and force your router to re-connect.
Your router will not lose SYNC, but instead drop the session with our server then re-connect. This will take effect immediately.

DSL Connection   
Link Information
Uptime:   0 days, 15:07:30
Modulation:   G.992.5 annex A
Bandwidth (Up/Down) [kbps/kbps]:   1,023 / 7,689
Data Transferred (Sent/Received) [KB/KB]:   0.00 / 0.00

Internet   
View more...
Type:   PPPoA
Uptime:   0 days, 0:00:38
IP Address:   109.170.137.226
Data Transferred (Sent/Received) [GB/GB]:   1.73 / 24.64

Monitored an overflow event yesterday - see attached image below.

Yesterday's gardens were excellent - ending with a cream tea.
The 2 day heatwave (=summer!) continues, but forecast end with a bang this evening and a couple of inches of rain.

Cheers,
Peter

[attachment deleted by admin]

[BoredBoy]

Finally got my router to 4GB:

Timestamp	RxBytes	TxBytes	RxWeb	TxWeb	UptimeDays	UpTime
28/06/2011 23:38	4,292,973,916	202,168,955	3.99 GB	192.80 MB	1 day	3:30:34
28/06/2011 23:39	35,617,228	203,556,978	4.03 GB	194.13 MB	1 day	3:31:34

So I think we've answered the question of whether the bug unrequested feature has been fixed in the TG585v7 with 7.4.2.7 firmware...

[geep]

In the absence of a flag to indicate that an overflow has happened, I decided to mingle together the value shown by the web interface (via a screenscrape) and the CLI value. I get bits 1 to 32 from the CLI, and bits 33 upwards from the web.

I wrote a Perl script to do this, which seems to work OK on my 64bit Linux. As I haven't yet downloaded 4GB (after a router powerdown) I can't completely test it yet. Should be there in a couple of days.

It also runs on my 32bit (Virtual) WinXP, using ActivePerl. but of course doesn't buy me anything as I'm limited to 32 bits on 32bit XP!!
(Added - Yes! See below - Perl can indeed handle "big numbers" on 32bit OS, > 2**32 - 1.

But - Perl does have a bigint function which I'm now looking at, which should allow a 32bit machine to run the script "properly". I'll post a revised version when I've finished it.

Cheers,
Peter

30 June Code updated - should now run OK on any 32bit machine. Runs OK on 32bit WinXP with ActivePerl. Trivial change - just added use bignum; - Transparent BigNumber support for Perl. Shame that there's no mention of this magic function in my "Programming Perl 3rd Edition" nor "Learning Perl 4th Edition"! Now just waiting for my downloaded bytes to be > 4.3GB to see if it really works.

There just seems to be one anomaly - on 32bit WinXP the binary value for xexact isn't correct when > 32 bits - it's always all 1s. But all is OK on Linux. As this value is just generated for debug purposes I'll ignore the problem.

2 July Code and examples updated All is now OK, thanks to assistance from the Perl Monks http://www.perlmonks.org/?node_id=912373

Code: [Select]

    xweb: 8600000000
     xcli: 100
 mask1_32:   0b11111111111111111111111111111111 4294967295
  mask33_: -0b100000000000000000000000000000000 -4294967296
     xweb: 0b1000000000100110011001011000000000 8600000000
  xwebmsb: 0b1000000000000000000000000000000000 8589934592
     xcli: 0b1100100 100
   xexact: 0b1000000000000000000000000001100100 8589934692

    value: 385.88  units: MB
    Bytes: 404624506.88 Bytes
Int Bytes: 404624506 Bytes

    value: 2.31  units: GB
    Bytes: 2480343613.44 Bytes
Int Bytes: 2480343613 Bytes

     xweb: 404624506
     xcli: 404634057
 mask1_32:   0b11111111111111111111111111111111 4294967295
  mask33_: -0b100000000000000000000000000000000 -4294967296
     xweb: 0b11000000111100001010001111010 404624506
  xwebmsb: 0b0 0
     xcli: 0b11000000111100011100111001001 404634057
   xexact: 0b11000000111100011100111001001 404634057

     xweb: 2480343613
     xcli: 2489739176
 mask1_32:   0b11111111111111111111111111111111 4294967295
  mask33_: -0b100000000000000000000000000000000 -4294967296
     xweb: 0b10010011110101110000101000111101 2480343613
  xwebmsb: 0b0 0
     xcli: 0b10010100011001100110011110101000 2489739176
   xexact: 0b10010100011001100110011110101000 2489739176

web tx: 385.88 MB rx: 2.31 GB
web bytes tx: 404624506 rx: 2480343613
cli tx: 404634057 rx: 2489739176
cli corrected tx: 404634057 rx: 2489739176


Code: [Select]

#!/usr/bin/perl
use Net::Telnet ();  
use LWP::Simple;
use bignum;
use Math::BigInt;
use strict;
use warnings;

my ($x, $xweb, $xcli);

#This bit just for testing....
#$xweb = 42949672979;
$xweb = 4300000000 * 2;
$xcli = 4294967295;
$xcli = 100;
$x = &exactify($xweb, $xcli);
#exit;
#...end of testing bit

# Set to your ST5x6 IP address or hostname (if any)  
my $host = '192.168.1.254';  
# Set to your ST5x6 administrator user id (Administrator or admin or whatever)  
my $user = 'Administrator';  
# Set to your ST5x6 administrator password (only you know what this is)  
my $pass = '';  
  
my $t = new Net::Telnet ( Timeout => 10,  
                        Prompt => '/.*=>$/i',  
                        Errmode => 'die');  
  
# Open host  
$t->open($host);  
  
# Login  
$t->login($user, $pass);  
 
#Get rx and tx from CLI - 32bit counter - max integer value is 4,294,967,295
# Command in modem
my @result = $t->cmd(':ip iflist');  
#print "ip iflist gives:\n@result\n"; #debug

#Parse for typical record thus:
#1   Internet. . . . . . . . . . . .  wan    1500  373145291  64661072   UP     UP  
my ($mtu, $rxcli, $txcli);
while(@result)
{
  if ($result[0] =~ /wan\s+(\d+)\s+(\d+)\s+(\d+) /x)
    {
      $mtu = $1; $rxcli = $2; $txcli = $3;
      #print "mtu: $mtu rxcli: $rxcli txcli: $txcli\n"; #debug
    }
  shift(@result);
}

 # Logout  
$t->close();  

#Screen-scrape rx and tx from web page (seems to be only to 5 significant digits) and convert to Bytes

my $content = get("http://192.168.1.254/cgi/b/is/?be=0&l0=1&l1=1");
my @contentlines = split /\n/, $content;

#Parse for typical record thus:
#<tr><td width='170'>Data Transferred (Sent/Received) [MB/MB]:</td><td colspan='3'>62.64 / 398.98</td></tr>
my ($txunit, $rxunit, $txscreen, $rxscreen);
while(@contentlines)
{
  #print "@content[0]\n"; #debug
    if ($contentlines[0] =~ /Data.Transferred..Sent.Received..\[(\w+)\/(\w+)\]:<\/td><td.colspan=.3.>(\d+.\d+)\s\/\s(\d+.\d+) /x)
    {
      $txunit = $1; $rxunit = $2; $txscreen = $3; $rxscreen = $4;
      #print "tx: $txscreen $txunit rx: $rxscreen $rxunit\n"; #debug
    }
  shift(@contentlines);
}

my $txweb = &convert_to_bytes($txscreen,$txunit);
my $rxweb = &convert_to_bytes($rxscreen,$rxunit);

#Calculate the exact values for rx and tx.
#Get the value, in bytes, from the web page.
#Mask out the bottom 32 bits to get the most significant bits - those above 32bits.
#Add the most significant bits (33 upwards) to the CLI value (the least significant 32 bits) (use OR to do this).

my ($tx, $rx);
$tx = &exactify($txweb, $txcli);
$rx = &exactify($rxweb, $rxcli);

print "web\t\ttx: $txscreen $txunit\trx: $rxscreen\ $rxunit\n";
print "web bytes\ttx: $txweb\trx: $rxweb\n";
print "cli\t\ttx: $txcli\trx: $rxcli\n";
print "cli corrected\ttx: $tx\trx: $rx\n";

exit;

sub exactify {
my ($xweb,$xcli,$xexact,$xwebmsb,$mask1_32,$mask1_32bytes,$mask33_,$mask33_bytes,$xwebbytes,$xclibytes,$xexactbytes,$xwebmsbbytes) = @_;

print "     xweb: $xweb\n     xcli: $xcli\n"; #debug

$mask1_32 = 2**32 - 1;
print " mask1_32:   ", $mask1_32->as_bin()," ",$mask1_32->as_int(),"\n"; #debug

$mask33_ = ~$mask1_32;
print "  mask33_: ", $mask33_->as_bin()," ",$mask33_->as_int(),"\n"; #debug

print "     xweb: ",$xweb->as_bin()," ",$xweb,"\n"; #debug

$xwebmsb = ($xweb & $mask33_);
print "  xwebmsb: ", $xwebmsb->as_bin()," ", $xwebmsb->as_int(),"\n"; #debug

# $xcli * 1 : need to make sure that it really is a bignum - to avoid:
# "Can't call method "as_bin" without a package or object reference"
$xcli= $xcli * 1;
print "     xcli: ",$xcli->as_bin()," ",$xcli,"\n"; #debug

#$xexact = ($xweb & $mask33_) | $xcli;
$xexact = $xwebmsb | $xcli;
print "   xexact: ",$xexact->as_bin()," ",$xexact,"\n\n"; #debug

return($xexact);
}

sub convert_to_bytes {
#Convert to Bytes. e.g. convert 12.3 MB to Bytes
my ($value,$units,$mult) = @_;
#print "    value: $value  units: $units\n"; #debug

unless ($units =~ m/(TB|GB|MB|KB|B)/) { print "PANIC - Unknown multiplier: $value $units\n"; exit; }
$mult = 1024**4 if $units eq "TB";
$mult = 1024**3 if $units eq "GB";
$mult = 1024**2 if $units eq "MB";
$mult = 1024 if $units eq "KB";
$mult = 1 if $units eq "B";

print "    value: $value  units: $units\n"; #debug
$value = $value * $mult;
print "    Bytes: $value Bytes\n"; #debug

$value = int($value);
print "Int Bytes: $value Bytes\n\n"; #debug

return($value);
}

[BoredBoy]

In the absence of a flag to indicate that an overflow has happened, I decided to mingle together the value shown by the web interface (via a screenscrape) and the CLI value. I get bits 1 to 32 from the CLI, and bits 33 upwards from the web.

Smart 

Since I'm just logging to a text file, I've switched to storing the strings as parsed out of the text. I'll eventualy get round to loading it into Access  or Oracle (it's what I do at work, so it'll be quicker), and look at fiddling it then... 

I really need to find a quick reset for the counts (a la "atm debug gstats clear=enabled" i.e. read-and-reset) so I can just store increments, which are kind of more useful anyway, I think.

I've been investigating other ways to extract the numbers - I got all excited when I read some SNMP documentation that mentioned 64-bit counters (ifHInOctets, ifHOutOctets or something similar), but they were made mandatory for SNMP V2c, and the 585v7 seems to only be v1.  I'm using snmpwalk to play with it - I can get the 32-bit counters (ifInOctets, ifOutOctets) no problem.

I also had a look at the cmwp(?) stuff - I think it's how the ISP's are supposed to be able to control/configure/genrally screw with our equipment (TR-069 is the standard), but decided fairly quickly that the ISP (NZ Telecom) have locked me out of enabling/changing the access so I can log on.  Not sure if they actually use it for anything, but all the usernames etc seem to have been defined already. Also I wasn't sure I wanted to learn yet another protocol for this

Maybe we should write our own router firmware...

[geep]

I seem to be seeing significant differences comparing the ST46 log with my ISP log. The two logs were zeroed at different  dates, but just looking at the data for the recent past I see a significant difference.

ISP:

Code: [Select]

Usage report
Current month
Month    Downloaded Uploaded Total        No. of sessions
Jul 2011 7.150 GB   0.540 GB 7.690 GB
ST546

Code: [Select]

DSL Connection
Uptime: 7 days, 5:39:15
Bandwidth (Up/Down) [kbps/kbps]: 1,023 / 7,376
Data Transferred (Sent/Received) [KB/KB]: 0.00 / 0.00
Internet Connection OK
Internet

Type: PPPoA
Uptime: 7 days, 5:38:52
IP Address: xx.xx.xx.xx
Data Transferred (Sent/Received) [MB/GB]: 764.79 / 8.41
Cheers,
Peter

[attachment deleted by admin]