Kitz Forum
Broadband Related => Router Monitoring Software => Topic started by: broadstairs on May 06, 2018, 10:44:25 AM
-
I decided to start my own thread for my scripts which display DSLStats data, two reasons - first it is cluttering up Kitz' thread and taking focus from her efforts, and second it will make it easier for me to keep on track with bugs, comments and suggestions for my scripts.
I have created these scripts to work both locally on a PHP capable webserver (the built in DSLStats webserver does not support PHP) or using an FTP server. Again my intention with the FTP support is primarily for people who use a NAS box locally which supports FTP and allows storing of data away from the likes of an RPi where storage is at a premium. It may well work with a hosted FTP site but as yet I have not been able to test SFTP which should really be used on a hosted service as it is far more secure than normal FTP which is fine on a local lan.
Because of a problem I had recently I have now changed the scripts to work with files stored in daily directories but removed the restriction of having to save them in name order, so now both name and date order work fine. I think the daily folder restriction is worth keeping as it reduces the amount of graphs being displayed.
I am still working on these scripts and new versions should be available in the next few days. So currently I am not making them downloadable but will send them to those who have requested them via PM with an email address. Once I am reasonably happy with them I will provide a downloadable zip file.
My existing versions can be seen at https://www.stella-maris.org.uk/DSLgraphs.php (https://www.stella-maris.org.uk/DSLgraphs.php)
Stuart
-
Just a quick update on SFTP. I have been testing this now and to support it will need quite a rewrite of the code so that part will now be delayed. So I hope by the end of today or perhaps first thing tomorrow to be in a position to send the scripts to those who have requested them.
Stuart
-
I have just emailed all those (I think) who have asked for a copy of my scripts a zip file. This code does NOT support SFTP only vanilla FTP for now, the SFTP support will have to come later.
Stuart
-
Stuart
Thanks. I have mine and will be testing later. I will need to manually upload to my server as I only have SFTP access. But I will be able to give feedback.
Tony
-
I thought I had a way to develop SFTP using my hosting site however because I'm on the basic package I do not have shell access, it looks like the next package up is the one I would need which is £80 per year and I currently pay £40. I have gone back to them to ask why their web site shows SFTP as available on my package but obviously does not work.
So for now with no access to an SFTP capable server my testing on this is at a standstill.
Stuart
-
Hi broadstairs
Kudos to you for your development
Sorry, why do you need shell access for sftp
Our platforms allow ftp and sftp, which both work from an ftp client
Please don’t feel you have to answer as I was just wondering or having a blonde moment
Also, there are various way to implement sftp server side, so what may work on some platforms may not work on others
Many thanks
John
-
Hi broadstairs
Kudos to you for your development
Sorry, why do you need shell access for sftp
Our platforms allow ftp and sftp, which both work from an ftp client
Please don’t feel you have to answer as I was just wondering or having a blonde moment
Also, there are various way to implement sftp server side, so what may work on some platforms may not work on others
Many thanks
John
John I have now had a further update from my hosting company. It seems the person who replied first quote "got their knickers in a twist". Apparently on their basic package SFTP should work but only with my main account username and password which I had not tried. So now I will test this and see if it works, they said they will help if it still fails. Indeed shell access is not required for SFTP.
Stuart
-
If they cannot provide it, message marjohn56 or myself and we will provide you an sftp account. I probably should extend this invitation to Eric as well to help him add it to dslstats.
Normally a valid sshd account is indeed needed for sftp, but I managed to get it working on virtual ftp accounts with user/pass auth using the sftp plugin on proftpd.
-
I am still having problems with my sftp testing on my hosting company servers. I have made some progress and now can connect using command line and filezilla just fine but using a PHP script simply fails.
Now I suspect the issue is with the key used by the server. When I initially used both command line and filezilla I was prompted to accept the key as it does not correctly represent the server being connect to, once accepted it works. However using the SSH2 additions to PHP I am not prompted and although it works using ssh2_connect and ssh2_auth_password as soon as I try the ssh2_sftp I get kicked off by the server. So currently I am at a loss to know how to progress this.
I do wonder if Eric's code in DSLStats may have an issue when folks start using their hosting companies to sftp because a lot of them use generic keys which do not always correctly identify the server. Anyone any ideas please?
Stuart
-
You may need to configure an auto acceptance of the certificate.
The scripts I made, that skyeci, marjohn56 both use, has a line that auto accepts the certificate so the login process can be completed. I am pretty sure PHP has something specific that needs to be set to auto accept untrusted certificates.
-
You may need to configure an auto acceptance of the certificate.
The scripts I made, that skyeci, marjohn56 both use, has a line that auto accepts the certificate so the login process can be completed. I am pretty sure PHP has something specific that needs to be set to auto accept untrusted certificates.
There is an option in ssh_config which allows this but so far I've not found anything in PHP which allows me to bypass the certificate verification. You are also supposed to be able to have a config file in the user space in Linux which should override defaults but no way have I been able to get this working. Also a shell script can be run to issue an ssh command but this is problematic as not all hosting package allow shell scripts.
I am starting to feel that providing SFTP in my scripts is more trouble than it is worth. For folks who have their DSLStats data on a hosted site you do not need SFTP in the scripts if the script is run from the same host. Again if the data is all on a local network then again SFTP is not really needed, nor is it if the data is on a local disk accessible by a local server.
If anyone can come up with something in PHP which will overcome my issue with certificates then I will have another go at this but right now I cannot continue to develop SFTP support.
Stuart
-
Hi broadstairs
Kudos to you
Sorry this may help or you could wait a week or so while roseway creates new sftp uploader in curl
Many thanks and sorry if you had already read this
https://stackoverflow.com/questions/28256655/connect-to-sftp-using-php-and-private-key
John
-
Hi broadstairs
Kudos to you
Sorry this may help or you could wait a week or so while roseway creates new sftp uploader in curl
Many thanks and sorry if you had already read this
https://stackoverflow.com/questions/28256655/connect-to-sftp-using-php-and-private-key
John
No I had not seen that John. I am not using phpseclib but trying to do this in vanilla PHP code and functions on the basis that there are folks here who want a plug and play solution and not a plug - mess around installing stuff - and pray solution. I just wanted it to work without having to run shell scripts, mess with PHP settings or installing stuff like phpseclib, yes I know there are people here who can do all that stuff with their eyes shut but there are also people who either cannot or simply do not want to do all this.
If only hosting companies had certificates which represented the sites correctly none of this wold be needed :(
Stuart
-
Your script works great Stuart with WinSCP and a couple of scripts ;)
https://www.team-rebellion.net/Ned/Statstest.php
-
Your script works great Stuart with WinSCP and a couple of scripts ;)
https://www.team-rebellion.net/Ned/Statstest.php
Thanks... that neatly shows why SFTP is not needed because it is accessing the data on the same server the script runs on. Yes we need Eric to get his DSLStats/cURL solution working in order to upload the data to those servers requiring SFTP.
Stuart
-
Hi Broadstairs
quote - If only hosting companies had certificates which represented the sites correctly none of this wold be needed
Please see this link why the keys have to be accepted/rejected
https://en.wikipedia.org/wiki/Trust_on_first_use
Many thanks
John
-
There is an option in ssh_config which allows this but so far I've not found anything in PHP which allows me to bypass the certificate verification. You are also supposed to be able to have a config file in the user space in Linux which should override defaults but no way have I been able to get this working. Also a shell script can be run to issue an ssh command but this is problematic as not all hosting package allow shell scripts.
I am starting to feel that providing SFTP in my scripts is more trouble than it is worth. For folks who have their DSLStats data on a hosted site you do not need SFTP in the scripts if the script is run from the same host. Again if the data is all on a local network then again SFTP is not really needed, nor is it if the data is on a local disk accessible by a local server.
If anyone can come up with something in PHP which will overcome my issue with certificates then I will have another go at this but right now I cannot continue to develop SFTP support.
Stuart
yeah I am not sure of the purpose of your ftp/sftp stuff, so I wouldnt stress over it too much.
It seems to make sense to have the files hosted on the same machine the script is run on. I dont use the function, the files are uploaded to my server prior to the script been used.
This is what I have in roundcube to make it ignore trust issues on imap/smtp connections, since there is no trusted localhost cert.
$config['smtp_conn_options'] = array(
'ssl' => array(
'verify_peer' => false,
'verify_peer_name' => false,
'verify_depth' => 3,
),
);
These flags set in php itself, that syntax is just how its applied in the roundcube config.
So you basically need to disable peer verification.
Another option is just to add your own CA as a trusted provider in your local SSL cert repo.
-
Hi Broadstairs
quote - If only hosting companies had certificates which represented the sites correctly none of this wold be needed
It isnt common because its an administrative nightmare, having a trusted cert for every single endpoint into the server including localhost. It also would be expensive prior to lets encrypt.
-
While I agree about the administrative problems the issue with most certs from hosting companies is that you get a warning that it does not represent the site you are accessing which worries a lot of people. I do understand that certs should be verified on first use. In part that is an issue for me in that I am still having problems with PHP even though using Filezilla I have accepted the certs and they are in my known hosts file!
Stuart
-
Filezilla is different software to PHP.
I believe you need to disable peer verification then you good, or add the cert as trusted in the local repo (not the same as been in known hosts).
https://secure.php.net/manual/en/context.ssl.php
-
I have decided now to stop all work on providing SFTP capability in my scripts. For two reasons, first and foremost it is unnecessary if using a hosting site for the data because the scripts both run in web server mode perfectly well on the same hosting site, and secondly I have other things I want to do rather than spend time on this. I will leave the vanilla FTP capability because it is useful for those who host their data on a private lan and run their own server on the lan as well, as I have already mentioned I run using the FTP mode to access my historical data on my NAS which supports FTP but does not have a web server on it.
I will enhance the historical script to add more graphs from the stats.log file if anyone wants to request them. Eric does provide a file on the DSLStats download site which explains what is available in the file so you can work out what is possible.
Stuart
-
Yep, ideally we could get graphs for everything MDWS had. So I request that functionality if possible. IF you want specific requests I will check Eric's list later. For sure a visual graph for QLN and HLOG would be nice as the historical file is just a text file with numbers.
-
Well in my local test script I've been able to graph one of the hlog files now I need to enhance it to extract the data from the zip file and allow the user to select the relevant zip file. How often does Hlog and QLN change as there are 22 zip files on my system for both Hlog and QLN. In fact there are 22 zip files for each category.
Stuart
-
I think they just change on a new sync event.
-
I think they just change on a new sync event.
That's what I thought, the issue is which one do I display or do I display graphs for all available files which is likely to be quite a workload with 22 zips to extract and 22 graphs to display?
Stuart
-
Many thanks for the script Stuart. Working perfectly on my two Pi systems.
https://pi4.snaddy.me.uk/dslstats/Stats.php (https://pi4.snaddy.me.uk/dslstats/Stats.php)
https://pi3.snaddy.me.uk/dslstats/Stats.php (https://pi3.snaddy.me.uk/dslstats/Stats.php)
-
@vic0239 glad to see you have it working.
As for the work in progress I now have it graphing (in test) the data from the zip files for Bits, Hlog, QLN and SNR. However this raises issues, there are a number of these zip files stored for each day my system has 23 numbered 00 to 22 for each category so I'm not sure people will need all graphed at once but I'm guessing people will want to select which one. The other issue is Eric's graphs are colour coded for u/s d/s and other, right now I dont know if I can do this with the javascript program and second if I can how it might be done or what boundaries Eric uses and are they standard?
So a number of questions there which I'd appreciate some opinions on.
Stuart
-
Stuart, the US and DS bands are derived from the pbParams data. In the UK I understand that there are just two band plans in place, one for Huawei DSLAMs and one for ECI, so you could look in the Stats summary for the DSLAM chipset (BDCM = Huawei DSLAM or IFTN = ECI DSLAM). Each will have its own known band plan.
-
I know have my test code being able to graph the Bits, Hlog, QLN and SNR from the zip files OK, and allowing the user to select which hourly file they want to view from all the zips that day. I have not yet investigated change the colouring per band plans but the actual graphs look OK when compared to the DSLStats ones apart from colours. I think sorting the colours will take a bit of effort as well as determining the correct plan details fo Huawei and ECI.
Stuart
-
I have put the latest versions of my script up so people can see what they now do. There has been some re-work and the radio buttons are now gone in favour of dropdown menus and now both script show the latest code and graphs. I now have to organise the Bits, QLN SNR and Hlog zip graphs to do the colour coding according to band plans but this might take a while because the javscript code does not support what I want so the re-work is more than I intended but less that using a different javascript program.
The latest scripts can be seen at https://www.stella-maris.org.uk/DSLgraphs.php (https://www.stella-maris.org.uk/DSLgraphs.php).
Stuart
-
I now have the band plan colouring and naming done for the graphs of the zip files. As yet I have only done this for my bands on my VDSL connection to an ECI cabinet. If someone can let me know what the bands are for VDSL on a Huawei cabinet that would be helpful. Also of help would be a sample of a zip file for the same. One other question perhaps Eric can answer, is there any way of telling from the stats.log file whether the cabinet is ECI or Huawei?
As to ADSL is there an equivalent band plan or is this not worth doing? I've been on VDSL for so long now I have no data from my ADSL days.
Stuart
-
Here's the band plan for a Huawei DSLAM:
Discovery Phase (Initial) Band Plan
US: (7,32) (871,1205) (1972,2782)
DS: (33,859) (1216,1961) (2793,3959)
The DSLAM type can't (as far as I know) be determined from the full stats (the output from the command "<command prefix> info --stats") but it's in the first line of the DSLstats Stats summary.
-
Here's the band plan for a Huawei DSLAM:
The DSLAM type can't (as far as I know) be determined from the full stats (the output from the command "<command prefix> info --stats") but it's in the first line of the DSLstats Stats summary.
Thanks Eric.
Stuart
-
Eric one further question. When I look at the band plan for my line the final one is different to the initial one although only on the DS1 band so which one should I display?
-
The problem with the final band plan is that it's potentially different for every connection. You can drive yourself mad trying to deal with all the combinations which appear in practice. That's what I used to try to do with DSLstats. The unallocated tones are the ones I displayed in blue. What happened in practice was that some of the unallocated tones appeared to actually be in use. Now for the sake of my sanity I use the discovery phase band plan, which is what HG612-Stats always did. Using this you can be confident that you won't mix up the upstream and downstream tones.
-
The problem with the final band plan is that it's potentially different for every connection. You can drive yourself mad trying to deal with all the combinations which appear in practice. That's what I used to try to do with DSLstats. The unallocated tones are the ones I displayed in blue. What happened in practice was that some of the unallocated tones appeared to actually be in use. Now for the sake of my sanity I use the discovery phase band plan, which is what HG612-Stats always did. Using this you can be confident that you won't mix up the upstream and downstream tones.
Thanks Eric I will follow your lead.
Stuart
-
I have now got the zip files graphing along with the stats.log. I have also made some structural changes and now the user variables part is now one separate file for both scripts which will hopefully clear up what directories and what. For the zip files there is another option to select either Huawei or ECI to get the correct band plan, the default on start is Huawei but it will remember the selection for the duration of the script running.
I have a little more testing to do and will in a day or so send out new copies to all those who have requested them. I have made it clear that the FTP capability is not secure and should only be used on a local lan and I do not intend to add SFTP capability as I believe for the majority of users it will not be necessary.
Stuart
-
Thanks to Eric since DSLStats V6.5.8 all the files needed for display or graphing can now be FTP's to a server directly from DSLStats, this can be a local FTP server (say on a NAS) or an external server for subsequent display on the internet.
I have decided now to make these scripts available for download from my website at https://www.stella-maris.org.uk/code/Scripts.zip (https://www.stella-maris.org.uk/code/Scripts.zip). If you have any questions or problems either PM me or update this thread.
Stuart
-
I have just tested adding the LEFTRS delta DS, LEFTRS delta US, rtx_tx delta DS and rtx_tx delta US to my stats.php script as it was suggested on another thread it might be useful for G.INP and the data is available on the stats.log file. It is fairly easy to add stuff which is already in the stats.log file data, however I cannot test the validity as I dont have G.INP on my ECI cab :-X
If anyone needs to test this let me know and I'll update my script zip file with this.
Stuart
-
Hi Stuart
I can test it my other line which is on huawei cab. if you are happy to send it I would like to try it as running your scripts already ;)
Thanks
Ned
-
Ned I've updated the download file with the updated Stats.php so just pull the zip again and try out the Stats.php file. The only change was those G.INP things.
Stuart
-
Thanks Stuart
Looks like its pulling data ok. This line has g.inp on huawei ..
https://www.team-rebellion.net/bmax/Stats.php
cheers
-
Thanks for putting that up. At least it seems to be graphing OK. I assume it is correct. I can add most things which are in the stats.php file which are capable of being graphed quite easily now, I just need to know what is really interesting as I dont want to add everything just in case.
Stuart
-
Hi
Thought I'd like to view historical results from DSLstats, so, set up a new raspberry pi html server with apache and php and installed a combination of kitz script with your script in the /var/www/html directory. My raspi running DSLstats sftp's to this new raspi html server.
Found no graphs drawn for bits, qln, hlog and snr. Looking through your code, realised that all these come from zipped files, and the unzipping wasn't working. Your commented out "echo" lines were very helpful.
A fair bit of googling later lead me to install php-zip (sudo apt-get install php-zip) on this new html server.
Your excellent program now displays all the graphs.
Thanks for your work.
Just in case someone else was wondering.
Terry
-
Not sure if this should be a new topic, moderators, please move if deemed necessary.
Having recently signed up for web hosting with John (d2d4j on this forum) I am in the process of posting my router stat's on line, harvested via DSLStats running on a RPi 3B and displayed on Broadstairs utility.
Currently uploading only SNRM, Conn Speed, CRC Errors, ES's & Avg Error Rates as snapshots on a once every 2 day basis.
All snapshots display when web accessed via their dated folder with the exception of SNRM which exibits "no valid file found".
After further investigation very much with assistance from John (d2d4j) who has been fantastically supportive, identified that the SNRM.png file name was changing from "SNRMargin-(+ date & time stamp).png" to "SNRM-(+ date & time stamp).png" at some stage during the upload process as the file was correctly named in the RPi .dslstats/snapshots folder.
If the SNRM file on the web server is re-named or manually uploaded using the web server file manager (and over written) with the correct file name, display via Broadstairs utility is normal.
Can't currently explain why this is happening, John is presently trying to replicate the issue from his RPi / web server using a similar 2 day time scale.
As quite a few forum members are now uploading and publicising their router stat's via DSLStats to an on line server, was wondering if anyone has experienced a similar issue ?
-
Looking at this on my system here I can see that the files are FTP'd by DSLStats with a new name, however I had not noticed this because my webserver is on the same PC as DSLStats so my data files are simply created by DSLStats locally. I use DSLStats to archive my files to a NAS and that uses FTP where I can see the files have their names changed.
So bottom line is that this is a DSLStats issue with roseway needs to look at.
Stuart
-
Yes, this has been reported and I agree that it's a bug, but for some time now I've been very limited by a sight problem when viewing screens for more than a few minutes. I'm trying to do a little programming at a time, and I'll give this issue my best shot as soon as I can.
-
Looking at this on my system here I can see that the files are FTP'd by DSLStats with a new name, however I had not noticed this because my webserver is on the same PC as DSLStats so my data files are simply created by DSLStats locally. I use DSLStats to archive my files to a NAS and that uses FTP where I can see the files have their names changed.
So bottom line is that this is a DSLStats issue with roseway needs to look at.
Stuart
Stuart, many thanks for confirming this and indeed for the provision and use of your utility program.
I know that Eric (roseway) is somewhat indisposed at the moment so reluctant to submit any queries or help requests at present.
From another angle, is there any way I can easily modify your program code to look for the modified SNRM file name, making the assumption that it must be expecting the original SNRMargin- (+ date & time stamp).png format and generating the "file not found" error in it's absence ?
I appreciate that each SNRM.png file uploaded will be unique by virtue of it's date & time stamp, it's the name prefix change (SNRMargin to SNRM) that's causing the issue.
I am using the latest program you have made public on your web site.
-
Unfortunately I am also not in a position to promise anything right now. I will try to take a look in the next few days. Basically if you are familiar with PHP you could try to find the part which defines the prefix used to list the directory and alter it.
Stuart
-
Yes, this has been reported and I agree that it's a bug, but for some time now I've been very limited by a sight problem when viewing screens for more than a few minutes. I'm trying to do a little programming at a time, and I'll give this issue my best shot as soon as I can.
Many thanks Eric, was posting a reply at the same time.
Did not realise that this was a known issue, apologies for the repetition.
I fully appreciate your limitations at present, hope things improve in the very near future, I can certainly work around this now having established (very much with John's (d2d4j) help) exactly what the issue is.
-
Unfortunately I am also not in a position to promise anything right now. I will try to take a look in the next few days. Basically if you are familiar with PHP you could try to find the part which defines the prefix used to list the directory and alter it.
Stuart
Completely understand Stuart, I'am certainly not a php coding expert but I will have a go, as long as I keep an original file can always go back !
Just a quick hint, where is the relevant coding likely to exist, DSLgraphs.php or Stats.php file ?
-
Hi
@broadstairs and roseway - many thanks
@tiffy - we have changed the php to reflect the new snrm name for you, which should work going forward. We changed the snrmmargin file to snrm and tested, which displays lovely
The change needs to happen in DSLgraphs.php and simply change SRNMargin to SRNM. All other png appear to work from our test outside of tiffy
Many thanks
John
-
Just one further comment. I believe it needs to be considered a bit more as anyone running on a local web server is getting SRRMargin as a prefix and anyone using FTP will get SNRM as a prefix so whichever way Eric goes will mess up one or the other. I'll try to take a look to see if I can cope with either, but it will take a few days.
Stuart
-
@d2d4j
John, thank you for the edit of my DSLgraphs.php file on the server, change noted.
Strangley enough, both SNRM & SNRMargin suffix .png files now open, I had earlier changed my file names suffix's to SNRMargin and I noted that you had changed my latest date file back to SNRM suffix in order to test, both permutations now open from Broadstairs utility !
@broadstairs
Stuart, yes, I'am sure your point is valid and will need to be sorted in the longer term between yourself and Eric, however, the file edit as identified by John will give you both a bit of breathing space.
Many thanks to both for your help and support.
-
I have just taken a quick look at my code and the fix you've done only works if you have only SNRM files and none of the SNRMPerBand files as well. So I need to take some time to work out if I can handle both file names easily.
Certainly in my view all the files should be identical whether you use a local web server or one where files are FTP'd to a remote server, however whatever Eric decides will cause some aggravation to users as they will end up with a mix of file names and in that case a PHP script to rename the offending files to the correct name would be easiest to fix up the issue.
Stuart
-
Hi broadstairs
Many thanks
I have sent you a pm so you can see on our platforms as it does appear to work. Note the domain I sent is a test only
Many thanks
John
-
In order to cope with both versions I have changed my code to provide two selection options in the drop down. You can select SNRM- or SNRMargin depending on which files you need to look for and this also eliminates the problem of showing other files with a prefix of SNRM which some including me have in the web server directories.
It's a simple change to line 5 in the DSLGraphs.php script
$filename = array("" ,"Bitloading" ,"Bitswap" ,"CRC" ,"ES" ,"FEC" ,"HLog" ,"QLN" ,"SES" ,"SNRM-" ,"SNRMargin" ,"SNRMperbandDownstream" ,"SNRMperbandUpstream" ,"SNRpertone" ,"Speed" ,"AverageErrors" ,"ConnectionStats" ,"Stats" ,"Traffic" );
This change is only required when using the FTP function of DSLStats to upload the graph files to a web server or in my case a NAS box which the server accesses via FTP itself. Once Eric decides how to fix this then we can look again as to what to do to clean it up.
One last thing the regression which caused this may have happened around the end of May and the start of June this year because that's when my file names changed from SNRMargin to SNRM
in my directories where they are FTP'd to by DSLStats. Just looked at my downloads directory and I can see two versions of DSLStats around that time 6.5.6 and 6.5.8 and I suspect the latter may be the culprit.
Stuart
-
@broadstairs
Many thanks for the file revision, will update my server files accordingly.
As reported earlier, since the edit that John made to my server php file last night, changed "SNRMargin" to "SNRM" both uploaded SNRM suffix variations are opening without any issues, have tried both permutations.