Kitz Forum
Broadband Related => Router Monitoring Software => Topic started by: broadstairs on March 31, 2018, 12:28:05 PM
-
This is very much a work in progress at present. I run a full blown Apache webserver locally and have started on a php script to display the archived graphs from DSLStats. Now this will not work using the DSLStats built in webserver but it should (I hope) work using a server installed using xampp on Windows as this provides both Apache and PHP. Attached is a screen shot showing the initial test. It allows selecting the date directory and then the graph type then will display all of the graphs for that day in descending date/time order. I hope to have a version cleaned up in a few days which I can make available to anyone who wishes to use it.
Stuart
-
I have now tested this version on my Windows xampp installation and it works, obviously it works on my Linux server as well. Currently the script will only display graphs saved in daily folders in name order by DSLStats and this directory tree must be accessible to xampp. The easiest way to do this is to get DSLStats to use a folder in the xampp/htdocs directory to save the snapshots in. So far I've not tested this accessing the snapshots on anything other than a disk on Windows (or Linux). Later I'll see if I can add using network attached disks but that will have to wait for now.
If anyone would like to try this either using xampp or their own webserver with PHP support or a linux box then please either reply here or PM me and I will let you have the code.
Stuart
-
Well done you :)
I've also been playing and managed to get something working which is uploading files to a remote webserver, with the idea that it would give me more power with PHP to do something for history.
It's still a bit messy right now, but the idea is that it keeps storage small for those who may want to run it from cheap hosting.
atm it's here (http://www.kitz.co.uk/linestats/snrm.htm) whilst I'm testing, but eventually I'll be moving it to another home that doesn't belong to kitz.co.uk
-
ATM I am monitoring with RPi and saving daily snapshots in categories on my local NAS, on the assumption that it would then be easier to scan through the history of any given parameter.
Sadly I have no knowledge of PHP or running a webserver, but what you are both doing sounds as though it has great potential for uneducated people like me, when you have it working.
I shall watch this thread with interest! :fingers:
-
I now have managed a method of retrieving the required data from an FTP server running on the network and displaying the graphs. The only thing is that so far the user/password combination is in clear in the url used which is not very secure so it would be fine on a local lan but not sure I'd want to use it in the wide world internet. OK I guess if your server does support anonymous but mine on my NAS does not. I'm guessing that most of the use of this would be on a private lan to view data so not too much of a security issue.
Stuart
-
I've managed to get something working (http://www.kitz.co.uk/linestats/index.htm). It still needs a bit of additional tidying up and it's designed for working on a remote webserver. I'm conscious that if anyone is buying hosting, then storage will be more problematic than from a local drive, so atm it's only displaying history for SNRM and Err/Secs which TBH are about the only ones I do look at (and that's only because of the weird spiking I get with Errors), but I could easily add others.
Live graphs appear to be working fine. I'm automatically uploading to the server and overwriting the current file so it's hardly using any storage. :)
I think I may have to rethink my dslstats graphing habits though when it comes to the display settings.
I currently have my display set to 8hrs for easy viewing, but I have DSLstats snap shotting every 2 hrs. I learnt on several occasions long ago that an unexpected shut down could cause the loss of up to 8hrs data. :/
As such that means that I do have quite a lot of graphs per day (12) for each statistic which could soon add up. Unless I've missed some setting in DSLstats I can't think of any other way other than 24 hr graphing. May I ask how often you get DSLstats to capture graphs as its a bit hard to see from bitswap.
-
I now have a version which can be configured to use either a directory on the web server or an FTP server to obtain and display the saved DSLStats graphs. It is still a bit of a work in progress as it needs to have so error checking added but it does work providing you get the user configuration variables correct ;) Obviously DSLStats does not FTP these graphs to a server but if you manually archive these files say daily then the disk space required locally is much less as there is no need to keep more that a couple or so days of graphs on say an RPi. I have a scheduled job running every 24 hours to place all my graphs on my 2TB NAS FTP server. In both modes it works fine on my W7 mini system using xampp as well as my full blown Linux Apache web server.
At present it only handles the png graphs but could be expanded to display the text files as well if needed.
A future plan is to have another script to try to handle the Stats file being FTP'd by DSLStats and draw some graphs however using server side code like PHP I suspect this may well be beyond the processing power for RPi machines or smaller Windows machines like my Brix but it might be interesting to try. Of course javascript may be a better way to go to draw graphs as the processing is done on the browser but currently that would be a big learning curve for me however if we have a javascript guru here it might make an interesting project ;)
Stuart
-
Kitz I display 2 hours on screen and currently snapshot every 2 hours giving me 12 files for each graph save and save most of them. I offload once per 24 hours to my 2TB NAS which is where I access the older graphs by FTP. I only keep 5 days of graphs on my local disk but as it is a 500Gb HDD I'm not short on space! I would not consider using my hosting service for this for a number of reasons, not least that I use it for my website and only have 5Gb space and that costs about £40/year.
Stuart
-
I am interested, thanks.
-
I am interested, thanks.
You have a PM
Stuart
-
With reference to the javascript I have found a downloadable (free) javascript which does graphing so all I need is to wrap PHP round the stuff to get the data and build the page for graphing the stats and this offloads the actual create/display of the graph to the browser which is good for low power processors. However it will take some time to get something working.
Stuart
-
I've made some improvements in my script and now have a demo version on my website. To try it out go to https://www.stella-maris.org.uk/DSLgraphs.php (https://www.stella-maris.org.uk/DSLgraphs.php). This is the web server version not the one which uses an FTP server for access to the files. Note that my traffic files actually do contain the word 'blank' ;)
Stuart
-
@broadstairs:
Have been following your thread with interest, thought I would install XAMPP suite on my desktop PC running Win 8.1 64b. just to have a little play around but was somewhat put off with the pre-install warnings ref. not installing to Program Files (x86) and/or deactivating UAC.
I note you mention running XAMPP on a Win 7 environment so perhaps you could advise:
Have you found it necessary to deactivate UAC ?
Is it deemed safe to do so ?
Is it necessary to avoid installing XAMPP to Program Files (x86) ?
Have currently aborted the install, aways use "Revo Uninstaller" for all new apps. so easy to back out.
-
I've made some improvements in my script and now have a demo version on my website. To try it out go to https://www.stella-maris.org.uk/DSLgraphs.php (https://www.stella-maris.org.uk/DSLgraphs.php). This is the web server version not the one which uses an FTP server for access to the files. Note that my traffic files actually do contain the word 'blank' ;)
Stuart
Very impressive work. Thanks Tony
-
Tiffy I always install programs like this on the D drive never on the C drive and to be honest UAC has never been a problem and never had to turn it off. Windows gets funny about some stuff. I have never used anything higher than W7 (and never will ;) ) so not sure about W8 or W10 but my guess is it will work OK as long as you dont install it on C and UAC should be fine left on.
Stuart
-
@broadstairs:
Many thanks for the advice, much appreciated.
As I recently fitted a 500 Gb. SSD on my desktop PC as C drive and retained the original 1Tb. HD I have plenty of avaiable space and tend to prioritise SSD installations where possible anyway.
Will certainly give XAMPP installation a try retaining UAC, as I said earlier, I always install new programs via "Revo Uninstaller" using tracked install option which to date has always been very efficient and trouble free with any future complete removal if so required.
Congratulations on your work to date with DSLStats php script, I follow the thread with great interest.
-
I have been testing a script which uses javascript to do the actual graphing on the web browser rather than doing it on the server so this I hope will still be useful on lower powered machines acting as a server.
This is to graph the stats.log file created by DSLStats. So far I have it working graphing SNRM, ES and FEC values from any single day in the stats.log file and in addition you can select a range of hours to display. There is a huge amount of data in this stats file and I'm not really sure how much of it is really used. I started with these values as they seem to be the ones which would show problems (yes I know CRCs and SESs are needed and will be added). I am thinking that attainable sync and maybe actual sync would be useful but not sure how much further to go with this, especially if there may be a MDWS replacement in the relatively near future.
What do others feel are essential graphs for debugging purposes, and which ones would be nice to have bearing in mind that the stats.log is effectively a CSV file with blank separators and therefore can be loaded in any spreadsheet.
Stuart
-
@broadstairs:
From my somewhat simplistic point of view, I only really monitor SNRM and ES on a very regular basis from my 3 RPi's on the lines I look after, I do snapshot SNRM, CRC's, ES's and Conn Speed's on a 2 day graph page basis, also Avg Errors text page at the same time scale.
If any cause for concern is noted or a router (DLM) re-synch should occur (notified by e-mail) then all the additional parameters can be examined from DSLStats historical data.
Obviously not as good as MDWS monitoring when this was active where direct, live comparisons to all other users lines could be made and any line issues could be viewed / diagnosed by the more knowledgable forum members.
Once again, many thanks for sharing your php/ftp script, I run this on my Win 8.1 desktop PC (php with Apache) and find it really useful, once the archieved data files are loaded to the appropriate areas viewing becomes very easy.
-
I have been testing a script which uses javascript to do the actual graphing on the web browser rather than doing it on the server so this I hope will still be useful on lower powered machines acting as a server.
Well done :)
What do others feel are essential graphs for debugging purposes, and which ones would be nice to have bearing in mind that the stats.log is effectively a CSV file with blank separators and therefore can be loaded in any spreadsheet.
Personally, aside from basic stats, the things I look at most are SNRM, CRC & ErrSecs
[Moderator edited to replace an inadvertent [/url] tag with a [/quote] tag.]
Note by kitz. I think we both edited at same time. I spotted it and changed it as soon as I hit send. You must have been quick on the mark to spot it before I changed it, as my edit was still within the courtesy 2min timeframe which didn't leave an 'edited by' tag :)
-
When using MDWS to view others' circuit behaviour, I would routinely select SNRM, FEC, CRC, QLN & Hlog. The latter two were selected to give me an idea of the quality of circuit's the physical infrastructure.
-
especially if there may be a MDWS replacement in the relatively near future.
I'm not sure if anyone is actually doing anything about this. It's partially why I wont spend any more time developing my scripts... and partially because no-one else seems to have shown much interest in viewing stats from a remote website rather than local machine and what I have done so far. So as mine does all that I personally need, then I drew a line under it after this point here (http://forum.kitz.co.uk/index.php/topic,21305.msg369052.html#msg369052).
I was tackling it from a different angle in that it doesn't require installation of WAMP/LAMP and instead runs from a website and tries to keep the file sizes small. It automatically FTP's live data to the site every min.
There is a huge amount of data in this stats file and I'm not really sure how much of it is really used.
Over time the files will get quite large. This is ok if you are running a webserver on the local machine.. but if you are running it remotely, then FTP of data will become problematic. Ideally you'd need to backward engineer so that it only sent up the last row.. then add that data to a full file retained on the remote server.
I would expect that for MDWS, DSLstats only uploaded the last row and then Tony added that row to his records. Otherwise bandwidth for both the EU and MDWS would become rather silly.
-
[Moderator edited to replace an inadvertent [/url] tag with a [/quote] tag.]
Note by kitz. I think we both edited at same time. I spotted it and changed it as soon as I hit send. You must have been quick on the mark to spot it before I changed it, as my edit was still within the courtesy 2min timeframe which didn't leave an 'edited by' tag :)
As you know, kittehs will eat, sleep and, just occasionally, do something very quickly! ;)
-
:D :D
Just noticed a typo - quick on = quick off :D Not editing again :D
When using MDWS to view others' circuit behaviour, I would routinely select SNRM, FEC, CRC, QLN & Hlog. The latter two were selected to give me an idea of the quality of circuit's the physical infrastructure.
http://www.kitz.co.uk/linestats/line.htm
Would you ever be interested in historic data for QLN & Hlog or would the most recent suffice?
Atm I've only got historic data for SNRM & Err Secs purely because that's what I would look at first when it came to any historic data. Connection speed as already mentioned by Stuart is probably another one that could be added for history.
-
:D :D
Just noticed a typo - quick on = quick off :D Not editing again :D
Ah, yes. I've seen it. :D
Would you ever be interested in historic data for QLN & Hlog or would the most recent suffice?
Normally it would be the most recent . . . very rarely would I go back to the historic plots.
-
I would expect that for MDWS, DSLstats only uploaded the last row and then Tony added that row to his records. Otherwise bandwidth for both the EU and MDWS would become rather silly.
That's correct. Once a minute the latest stats (about 500 bytes for VDSL2) were uploaded, and once an hour the four per-tone zip files were uploaded. MDWS stored the history in its own format.
At present, DSLstats is storing/uploading everything because there's no other service retaining the history. If a successor to MDWS does emerge, then I would expect to upload just the latest set of data.
-
I'm not sure if anyone is actually doing anything about this. It's partially why I wont spend any more time developing my scripts... and partially because no-one else seems to have shown much interest in viewing stats from a remote website rather than local machine and what I have done so far. So as mine does all that I personally need, then I drew a line under it after this point here (http://forum.kitz.co.uk/index.php/topic,21305.msg369052.html#msg369052).
I was tackling it from a different angle in that it doesn't require installation of WAMP/LAMP and instead runs from a website and tries to keep the file sizes small. It automatically FTP's live data to the site every min.
Over time the files will get quite large. This is ok if you are running a webserver on the local machine.. but if you are running it remotely, then FTP of data will become problematic. Ideally you'd need to backward engineer so that it only sent up the last row.. then add that data to a full file retained on the remote server.
I would expect that for MDWS, DSLstats only uploaded the last row and then Tony added that row to his records. Otherwise bandwidth for both the EU and MDWS would become rather silly.
Me and Martin have been busy, for sure the biggest hurdle is code. So if scripts are developed by people it will speed things along nicely. If Tony had shared his code a MDWS replacement would probably be already running, legal issues, hosting costs etc, all minor, the issue is working code to process the data and display it.
-
Yes I agree building the graphing code and understanding what is contained in the stats.log are the biggest hurdles not only for an MDWS replacement but anyone building scripts to graph the values locally and ensuring the display is meaningful. Displaying the archived graphs locally from DSLStats is easy though.
I have already discovered for example that there are only two values for SES one US and one DS and they represent the total SES number but since when is my issue. It looks like Tony would actually manipulate the values before storing them all in a database as I dont remember seeing SES for example as a running total but a minute by minute value.
Stuart
Edit: An update from Eric on the SES thing, he has suggested he adds two currently unused fields for the SES DS and US delta (per minute values) as DSLStats already calculates those values for its graphs which I think is an excellent idea.
-
I've uploaded a demo version of my Stats.php script which displays some of the values from the Stats.log file created by DSLStats. To try it out go to https://www.stella-maris.org.uk/Stats.php (https://www.stella-maris.org.uk/Stats.php). This is the web server version not the one which uses an FTP server for access to the files. This uses javascript to perform the actual graphing on the web browser not the server so please make sure you have javascript enabled. Also note that the FEC graphs are in logarithmic format and currently the SES value is a total and not by minute value.
Stuart