Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 [2]

Author Topic: A script to display saved DSLStats graphs  (Read 3157 times)

tiffy

  • Reg Member
  • ***
  • Posts: 548
Re: A script to display saved DSLStats graphs
« Reply #15 on: April 03, 2018, 10:46:57 AM »

@broadstairs:

Many thanks for the advice, much appreciated.
As I recently fitted a 500 Gb. SSD on my desktop PC as C drive and retained the original 1Tb. HD I have plenty of avaiable space and tend to prioritise SSD installations where possible anyway.

Will certainly give XAMPP installation a try retaining UAC, as I said earlier, I always install new programs via "Revo Uninstaller" using tracked install option which to date has always been very efficient and trouble free with any future complete removal if so required.

Congratulations on your work to date with DSLStats php script, I follow the thread with great interest.
Logged
Plusnet 40/10 - DSLStats/RPi - ZyXEL VMG1312-B10A - Huawei Cab - MDWS (sadly gone 12/03/18)

broadstairs

  • Kitizen
  • ****
  • Posts: 3181
Re: A script to display saved DSLStats graphs
« Reply #16 on: April 11, 2018, 11:09:40 AM »

I have been testing a script which uses javascript to do the actual graphing on the web browser rather than doing it on the server so this I hope will still be useful on lower powered machines acting as a server.

This is to graph the stats.log file created by DSLStats. So far I have it working graphing SNRM, ES and FEC values from any single day in the stats.log file and in addition you can select a range of hours to display. There is a huge amount of data in this stats file and I'm not really sure how much of it is really used. I started with these values as they seem to be the ones which would show problems (yes I know CRCs and SESs are needed and will be added). I am thinking that attainable sync and maybe actual sync would be useful but not sure how much further to go with this, especially if there may be a MDWS replacement in the relatively near future.

What do others feel are essential graphs for debugging purposes, and which ones would be nice to have bearing in mind that the stats.log is effectively a CSV file with blank separators and therefore can be loaded in any spreadsheet.

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220

tiffy

  • Reg Member
  • ***
  • Posts: 548
Re: A script to display saved DSLStats graphs
« Reply #17 on: April 11, 2018, 09:38:55 PM »

@broadstairs:

From my somewhat simplistic point of view, I only really monitor SNRM and ES on a very regular basis from my 3 RPi's on the lines I look after, I do snapshot SNRM, CRC's, ES's and Conn Speed's on a 2 day graph page basis, also Avg Errors text page at the same time scale.
If any cause for concern is noted or a router (DLM) re-synch should occur (notified by e-mail) then all the additional parameters can be examined from DSLStats historical data.

Obviously not as good as MDWS monitoring when this was active where direct, live comparisons to all other users lines could be made and any line issues could be viewed / diagnosed by the more knowledgable forum members.

Once again, many thanks for sharing your php/ftp script, I run this on my Win 8.1 desktop PC (php with Apache) and find it really useful, once the archieved data files are loaded to the appropriate areas viewing becomes very easy.
Logged
Plusnet 40/10 - DSLStats/RPi - ZyXEL VMG1312-B10A - Huawei Cab - MDWS (sadly gone 12/03/18)

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32023
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: A script to display saved DSLStats graphs
« Reply #18 on: April 11, 2018, 11:49:25 PM »

I have been testing a script which uses javascript to do the actual graphing on the web browser rather than doing it on the server so this I hope will still be useful on lower powered machines acting as a server.

Well done :)

Quote
What do others feel are essential graphs for debugging purposes, and which ones would be nice to have bearing in mind that the stats.log is effectively a CSV file with blank separators and therefore can be loaded in any spreadsheet.

Personally, aside from basic stats,  the things I look at most are SNRM, CRC & ErrSecs

[Moderator edited to replace an inadvertent [/url] tag with a [/quote] tag.]

Note by kitz. I think we both edited at same time.  I spotted it and changed it as soon as I hit send.  You must have been quick on the mark to spot it before I changed it, as my edit was still within the courtesy 2min timeframe which didn't leave an 'edited by' tag :)
« Last Edit: April 12, 2018, 12:13:59 AM by kitz »
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 26405
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: A script to display saved DSLStats graphs
« Reply #19 on: April 11, 2018, 11:58:27 PM »

When using MDWS to view others' circuit behaviour, I would routinely select SNRM, FEC, CRC, QLN & Hlog. The latter two were selected to give me an idea of the quality of circuit's the physical infrastructure.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32023
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: A script to display saved DSLStats graphs
« Reply #20 on: April 12, 2018, 12:08:29 AM »

especially if there may be a MDWS replacement in the relatively near future.

I'm not sure if anyone is actually doing anything about this. It's partially why I wont spend any more time developing my scripts...  and partially because no-one else seems to have shown much interest in viewing stats from a remote website rather than local machine and what I have done so far.   So as mine does all that I personally need, then I drew a line under it after this point here.

I was tackling it from a different angle in that it doesn't require installation of WAMP/LAMP and instead runs from a website and tries to keep the file sizes small.   It automatically FTP's live data to the site every min.

Quote
There is a huge amount of data in this stats file and I'm not really sure how much of it is really used.

Over time the files will get quite large.  This is ok if you are running a webserver on the local machine..  but if you are running it remotely, then FTP of data will become problematic.     Ideally you'd need to backward engineer so that it only sent up the last row..  then add that data to a full file retained on the remote server.   

I would expect that for MDWS, DSLstats only uploaded the last row and then Tony added that row to his records.  Otherwise bandwidth for both the EU and MDWS would become rather silly.     
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 26405
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: A script to display saved DSLStats graphs
« Reply #21 on: April 12, 2018, 12:47:49 AM »

[Moderator edited to replace an inadvertent [/url] tag with a [/quote] tag.]

Note by kitz. I think we both edited at same time.  I spotted it and changed it as soon as I hit send.  You must have been quick on the mark to spot it before I changed it, as my edit was still within the courtesy 2min timeframe which didn't leave an 'edited by' tag :)

As you know, kittehs will eat, sleep and, just occasionally, do something very quickly!  ;)
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32023
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: A script to display saved DSLStats graphs
« Reply #22 on: April 12, 2018, 01:11:15 AM »

 :D :D

Just noticed a typo - quick on = quick off :D   Not editing again :D

When using MDWS to view others' circuit behaviour, I would routinely select SNRM, FEC, CRC, QLN & Hlog. The latter two were selected to give me an idea of the quality of circuit's the physical infrastructure.

http://www.kitz.co.uk/linestats/line.htm

Would you ever be interested in historic data for QLN & Hlog or would the most recent suffice?
Atm I've only got historic data for SNRM & Err Secs purely because that's what I would look at first when it came to any historic data.   Connection speed as already mentioned by Stuart is probably another one that could be added for history. 
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 26405
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: A script to display saved DSLStats graphs
« Reply #23 on: April 12, 2018, 01:23:00 AM »

:D :D

Just noticed a typo - quick on = quick off :D   Not editing again :D

Ah, yes. I've seen it.  :D

Quote
Would you ever be interested in historic data for QLN & Hlog or would the most recent suffice?

Normally it would be the most recent . . . very rarely would I go back to the historic plots.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39436
  • Penguins CAN fly
    • DSLstats
Re: A script to display saved DSLStats graphs
« Reply #24 on: April 12, 2018, 07:11:02 AM »

Quote
I would expect that for MDWS, DSLstats only uploaded the last row and then Tony added that row to his records.  Otherwise bandwidth for both the EU and MDWS would become rather silly. 

That's correct. Once a minute the latest stats (about 500 bytes for VDSL2) were uploaded, and once an hour the four per-tone zip files were uploaded. MDWS stored the history in its own format.

At present, DSLstats is storing/uploading everything because there's no other service retaining the history. If a successor to MDWS does emerge, then I would expect to upload just the latest set of data.
Logged
  Eric

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 5720
Re: A script to display saved DSLStats graphs
« Reply #25 on: April 12, 2018, 08:56:23 AM »

I'm not sure if anyone is actually doing anything about this. It's partially why I wont spend any more time developing my scripts...  and partially because no-one else seems to have shown much interest in viewing stats from a remote website rather than local machine and what I have done so far.   So as mine does all that I personally need, then I drew a line under it after this point here.

I was tackling it from a different angle in that it doesn't require installation of WAMP/LAMP and instead runs from a website and tries to keep the file sizes small.   It automatically FTP's live data to the site every min.

Over time the files will get quite large.  This is ok if you are running a webserver on the local machine..  but if you are running it remotely, then FTP of data will become problematic.     Ideally you'd need to backward engineer so that it only sent up the last row..  then add that data to a full file retained on the remote server.   

I would expect that for MDWS, DSLstats only uploaded the last row and then Tony added that row to his records.  Otherwise bandwidth for both the EU and MDWS would become rather silly.     

Me and Martin have been busy, for sure the biggest hurdle is code.  So if scripts are developed by people it will speed things along nicely.  If Tony had shared his code a MDWS replacement would probably be already running, legal issues, hosting costs etc, all minor, the issue is working code to process the data and display it.
Logged
Sky Fiber Pro - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

broadstairs

  • Kitizen
  • ****
  • Posts: 3181
Re: A script to display saved DSLStats graphs
« Reply #26 on: April 12, 2018, 09:27:00 AM »

Yes I agree building the graphing code and understanding what is contained in the stats.log are the biggest hurdles not only for an MDWS replacement but anyone building scripts to graph the values locally and ensuring the display is meaningful. Displaying the archived graphs locally from DSLStats is easy though.

I have already discovered for example that there are only two values for SES one US and one DS and they represent the total SES number but since when is my issue. It looks like Tony would actually manipulate the values before storing them all in a database as I dont remember seeing SES for example as a running total but a minute by minute value.

Stuart

Edit: An update from Eric on the SES thing, he has suggested he adds two currently unused fields for the SES DS and US delta (per minute values) as DSLStats already calculates those values for its graphs which I think is an excellent idea.
« Last Edit: April 12, 2018, 04:15:35 PM by broadstairs »
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220

broadstairs

  • Kitizen
  • ****
  • Posts: 3181
Re: A script to display saved DSLStats graphs
« Reply #27 on: April 12, 2018, 10:34:33 AM »

I've uploaded a demo version of my Stats.php script which displays some of the values from the Stats.log file created by DSLStats. To try it out go to https://www.stella-maris.org.uk/Stats.php. This is the web server version not the one which uses an FTP server for access to the files. This uses javascript to perform the actual graphing on the web browser not the server so please make sure you have javascript enabled. Also note that the FEC graphs are in logarithmic format and currently the SES value is a total and not by minute value.

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220
Pages: 1 [2]
 

anything