Kitz Forum

Broadband Related => Router Monitoring Software => Topic started by: roseway on April 09, 2018, 10:58:35 AM

Title: DSLstats data store/upload format
Post by: roseway on April 09, 2018, 10:58:35 AM
So far I've retained the stats format used for MDWS uploads because this is compatible with the log file of HG612_stats. The format is a bit illogical in places for historical reasons, but I see no overpowering reason to change it at this stage, unless there is strong disagreement.

For those who've downloaded the spreadsheet describing the format, I've noticed one small error in the 11 March 2018 version of the spreadsheet: Cell B129 should read "Interleave depth B1 US". I'll be uploading a revised version dated 9 April 2018 shortly.

[Later] Revised spreadsheet now available from: http://dslstats.me.uk/files/UploadStats.ods
Title: Re: DSLstats data store/upload format
Post by: burakkucat on April 09, 2018, 04:26:09 PM
Thank you. Updated spread-sheet downloaded and replaced the prior version.
Title: Re: DSLstats data store/upload format
Post by: broadstairs on April 12, 2018, 07:59:15 AM
Eric I'm just looking to add CRCs to my script but the spreadsheet does not seem to have a CRC US value at all only a DS one.

Stuart
Title: Re: DSLstats data store/upload format
Post by: roseway on April 12, 2018, 10:06:45 AM
Stuart, CRC is the same as OHFErr, so CRC delta US is available from field 128 (OHFErr delta US). This is one of the historical hangovers from the way the format for the stats file was changed over time.
Title: Re: DSLstats data store/upload format
Post by: broadstairs on April 12, 2018, 10:15:00 AM
Thanks Eric, I knew there was an explanation.

Stuart
Title: Re: DSLstats data store/upload format
Post by: broadstairs on April 12, 2018, 10:35:22 AM
Eric another question, is there a delta value for SES somewhere called something else?

Stuart
Title: Re: DSLstats data store/upload format
Post by: roseway on April 12, 2018, 10:43:53 AM
DSLstats does calculate delta SES values for display in its own graphs, but they are not currently included in the Stats.log file. It would be simple to add them (I suggest fields 48 and 49) if that's helpful.
Title: Re: DSLstats data store/upload format
Post by: broadstairs on April 12, 2018, 10:53:48 AM
If that's easy to add Eric then yes please. Currently it only displays the total values from the other fields but there is no (easy) way to know when that count started but showing a per minute vale I think is better.

Stuart
Title: Re: DSLstats data store/upload format
Post by: tiffy on May 11, 2018, 09:39:50 AM
There are a number of threads on the go at present ref. DSLStats display in various web interface variations, don't wish to start another so hope this is the best location choice to post my question.

I'am trying to make use of the "WS Upload" feature of DSLStats to upload stats as selected in the "Webserver" tab, running DSLStats on a RPi 3B+ successfully storing the currently selected stats in "/home/pi/.dslstats/Webserver" directory, RPi is currently Wi-Fi connected to ZyXEL VMG1312-B10A modem/router.

Have creatred a FTP server on my Win 8.1 desktop PC using xampp & FileZilla server, have tested the created account access using FileZilla client.

My issue is, can't get the webserver files to upload from the RPi to the PC FTP server, the "test" facility is failing.
Believe the problem is with the format of the "Upload URL:" I am using, have tried using the PC IP address and the default xampp server IP address (127.0.0.1) with various permutations but no joy.
The "Upload Directory:" is the xampp default <drive letter:>\xampp\htdocs, username & password as per FTP server account, FTP port 21 and Auto TLS selected.

My AV/Firewall software on PC is Bitdefender Internet Security 2018, don't think this is part of the issue but could be wrong ?
Any suggestions appreciated.

Title: Re: DSLstats data store/upload format
Post by: roseway on May 11, 2018, 10:33:44 AM
The latest version of DSLstats is quite fussy about the format of the upload URL and directory. I suspect in this case that you're putting the full path in the upload directory, whereas you should only put the extra part beyond the point where you go when first logging in.

In any event, I've completely rewritten the FTP upload code using an external program (Curl) to perform the upload, because this also supports SFTP (which I couldn't get working any other way). So with apologies for your wasted time, I can only ask that you wait another week or so until I'm ready to release the next version.
Title: Re: DSLstats data store/upload format
Post by: d2d4j on May 11, 2018, 11:20:20 AM
Hi Roseway

Kudos to you

I am most likely entirely wrong, so apologies in advance, but I think the issue maybe connected with the server trust of rsa2 keys.  See picture as taken from using scp test

Also, it maybe that sftp usually runs on a different port, which on our platforms is 24

Many thanks

John
Title: Re: DSLstats data store/upload format
Post by: tiffy on May 11, 2018, 11:23:59 AM
Eric,

Many thanks for your response, much appreciated.
Your comments on the pending revision noted, no need for apologies, my time is currently cheap !

As per advice, now using "/Uploads" as Upload directory having created same in xampp/htdocs.

Still very confused on the format of the Upload URL required:
Have tried "http://192.168.0.2" (my PC IP add.)
Have tried "http://127.0.0.1" (web server local host IP)
Have tried "192.168.0.2" (this locks up DSLStats for approx 2 min's)
Have tried "127.0.0.1" (registers failure)
Reading you configuration help, "Data Store / Data Upload" section I note you prefix the Upload URL with "ftp.", have also tried this with above permutations.

I accept and very much appreciate you are currently working on an update so please don't feel obliged to respond further.
Title: Re: DSLstats data store/upload format
Post by: roseway on May 11, 2018, 11:45:38 AM
@d2d4j: Many thanks for the suggestion, but my problem was more basic than that - I couldn't find an SFTP library which I could even build into DSLstats without compiler errors. There are commercial libraries, but they're quite expensive. Curl works very well, and versions with SSH support are readily available now.
Title: Re: DSLstats data store/upload format
Post by: roseway on May 11, 2018, 11:52:22 AM
@tiffy: The URL should be just the domain name without any prefixes. Certainly not http:// because that would be specifying the http protocol. Internally DSLstats adds ftp:// to the domain name when connecting. However your FTP server on the LAN is something I hadn't envisaged, so I'd need to think more about that.
Title: Re: DSLstats data store/upload format
Post by: vic0239 on May 11, 2018, 11:59:29 AM
At the moment I am uploading to my NAS on the local LAN and the upload URL is simply the DNS name of the NAS which resolves to 192.168.1.xx. I've seen a lock up, but it was down to incorrect specification of the subsequent FTP parameters.
Title: Re: DSLstats data store/upload format
Post by: broadstairs on May 11, 2018, 01:02:49 PM
Re the SFTP port my hosting company uses 722, although right now I am unable to get it working with a basic PHP script  :'(

Stuart
Title: Re: DSLstats data store/upload format
Post by: tiffy on May 11, 2018, 06:31:14 PM
@ roseway & vic0239:

Thanks to both for the further information, much appreciated.
Investigative activites had to be quickly abandoned this morning as my son managed to locate a 22 mm plastic water pipe with a wood screw at his upstairs landing when adding additional "anti-squeek" securing to his flooring, quite a mess, all hands to the pumps !

Will await further developments.
Title: Re: DSLstats data store/upload format
Post by: tiffy on May 11, 2018, 11:11:23 PM
At the moment I am uploading to my NAS on the local LAN and the upload URL is simply the DNS name of the NAS which resolves to 192.168.1.xx. I've seen a lock up, but it was down to incorrect specification of the subsequent FTP parameters.

Yes, I believe you are correct ref. Upload URL, the only option which appears to do anything for me, all be it producing a program "lock up" is using my server PC IP address, however have tried virtually every conceivable permutation of Upload directory and still get the same effect.
I know the Username & Password are correct having created the account and tested access using FileZilla client.

Hopefully Eric will produce a solution in the longer term with his next program revision.