Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: DSLstats data store/upload format  (Read 1879 times)

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39471
  • Penguins CAN fly
    • DSLstats
DSLstats data store/upload format
« on: April 09, 2018, 10:58:35 AM »

So far I've retained the stats format used for MDWS uploads because this is compatible with the log file of HG612_stats. The format is a bit illogical in places for historical reasons, but I see no overpowering reason to change it at this stage, unless there is strong disagreement.

For those who've downloaded the spreadsheet describing the format, I've noticed one small error in the 11 March 2018 version of the spreadsheet: Cell B129 should read "Interleave depth B1 US". I'll be uploading a revised version dated 9 April 2018 shortly.

[Later] Revised spreadsheet now available from: http://dslstats.me.uk/files/UploadStats.ods
« Last Edit: April 09, 2018, 11:17:44 AM by roseway »
Logged
  Eric

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 26924
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: DSLstats data store/upload format
« Reply #1 on: April 09, 2018, 04:26:09 PM »

Thank you. Updated spread-sheet downloaded and replaced the prior version.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

broadstairs

  • Kitizen
  • ****
  • Posts: 3187
Re: DSLstats data store/upload format
« Reply #2 on: April 12, 2018, 07:59:15 AM »

Eric I'm just looking to add CRCs to my script but the spreadsheet does not seem to have a CRC US value at all only a DS one.

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39471
  • Penguins CAN fly
    • DSLstats
Re: DSLstats data store/upload format
« Reply #3 on: April 12, 2018, 10:06:45 AM »

Stuart, CRC is the same as OHFErr, so CRC delta US is available from field 128 (OHFErr delta US). This is one of the historical hangovers from the way the format for the stats file was changed over time.
Logged
  Eric

broadstairs

  • Kitizen
  • ****
  • Posts: 3187
Re: DSLstats data store/upload format
« Reply #4 on: April 12, 2018, 10:15:00 AM »

Thanks Eric, I knew there was an explanation.

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220

broadstairs

  • Kitizen
  • ****
  • Posts: 3187
Re: DSLstats data store/upload format
« Reply #5 on: April 12, 2018, 10:35:22 AM »

Eric another question, is there a delta value for SES somewhere called something else?

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39471
  • Penguins CAN fly
    • DSLstats
Re: DSLstats data store/upload format
« Reply #6 on: April 12, 2018, 10:43:53 AM »

DSLstats does calculate delta SES values for display in its own graphs, but they are not currently included in the Stats.log file. It would be simple to add them (I suggest fields 48 and 49) if that's helpful.
Logged
  Eric

broadstairs

  • Kitizen
  • ****
  • Posts: 3187
Re: DSLstats data store/upload format
« Reply #7 on: April 12, 2018, 10:53:48 AM »

If that's easy to add Eric then yes please. Currently it only displays the total values from the other fields but there is no (easy) way to know when that count started but showing a per minute vale I think is better.

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Netgear D6220

tiffy

  • Reg Member
  • ***
  • Posts: 549
Re: DSLstats data store/upload format
« Reply #8 on: May 11, 2018, 09:39:50 AM »

There are a number of threads on the go at present ref. DSLStats display in various web interface variations, don't wish to start another so hope this is the best location choice to post my question.

I'am trying to make use of the "WS Upload" feature of DSLStats to upload stats as selected in the "Webserver" tab, running DSLStats on a RPi 3B+ successfully storing the currently selected stats in "/home/pi/.dslstats/Webserver" directory, RPi is currently Wi-Fi connected to ZyXEL VMG1312-B10A modem/router.

Have creatred a FTP server on my Win 8.1 desktop PC using xampp & FileZilla server, have tested the created account access using FileZilla client.

My issue is, can't get the webserver files to upload from the RPi to the PC FTP server, the "test" facility is failing.
Believe the problem is with the format of the "Upload URL:" I am using, have tried using the PC IP address and the default xampp server IP address (127.0.0.1) with various permutations but no joy.
The "Upload Directory:" is the xampp default <drive letter:>\xampp\htdocs, username & password as per FTP server account, FTP port 21 and Auto TLS selected.

My AV/Firewall software on PC is Bitdefender Internet Security 2018, don't think this is part of the issue but could be wrong ?
Any suggestions appreciated.

Logged
Plusnet 40/10 - DSLStats/RPi - ZyXEL VMG1312-B10A - Huawei Cab - MDWS (sadly gone 12/03/18)

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39471
  • Penguins CAN fly
    • DSLstats
Re: DSLstats data store/upload format
« Reply #9 on: May 11, 2018, 10:33:44 AM »

The latest version of DSLstats is quite fussy about the format of the upload URL and directory. I suspect in this case that you're putting the full path in the upload directory, whereas you should only put the extra part beyond the point where you go when first logging in.

In any event, I've completely rewritten the FTP upload code using an external program (Curl) to perform the upload, because this also supports SFTP (which I couldn't get working any other way). So with apologies for your wasted time, I can only ask that you wait another week or so until I'm ready to release the next version.
Logged
  Eric

d2d4j

  • Reg Member
  • ***
  • Posts: 866
Re: DSLstats data store/upload format
« Reply #10 on: May 11, 2018, 11:20:20 AM »

Hi Roseway

Kudos to you

I am most likely entirely wrong, so apologies in advance, but I think the issue maybe connected with the server trust of rsa2 keys.  See picture as taken from using scp test

Also, it maybe that sftp usually runs on a different port, which on our platforms is 24

Many thanks

John
Logged

tiffy

  • Reg Member
  • ***
  • Posts: 549
Re: DSLstats data store/upload format
« Reply #11 on: May 11, 2018, 11:23:59 AM »

Eric,

Many thanks for your response, much appreciated.
Your comments on the pending revision noted, no need for apologies, my time is currently cheap !

As per advice, now using "/Uploads" as Upload directory having created same in xampp/htdocs.

Still very confused on the format of the Upload URL required:
Have tried "http://192.168.0.2" (my PC IP add.)
Have tried "http://127.0.0.1" (web server local host IP)
Have tried "192.168.0.2" (this locks up DSLStats for approx 2 min's)
Have tried "127.0.0.1" (registers failure)
Reading you configuration help, "Data Store / Data Upload" section I note you prefix the Upload URL with "ftp.", have also tried this with above permutations.

I accept and very much appreciate you are currently working on an update so please don't feel obliged to respond further.
Logged
Plusnet 40/10 - DSLStats/RPi - ZyXEL VMG1312-B10A - Huawei Cab - MDWS (sadly gone 12/03/18)

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39471
  • Penguins CAN fly
    • DSLstats
Re: DSLstats data store/upload format
« Reply #12 on: May 11, 2018, 11:45:38 AM »

@d2d4j: Many thanks for the suggestion, but my problem was more basic than that - I couldn't find an SFTP library which I could even build into DSLstats without compiler errors. There are commercial libraries, but they're quite expensive. Curl works very well, and versions with SSH support are readily available now.
Logged
  Eric

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39471
  • Penguins CAN fly
    • DSLstats
Re: DSLstats data store/upload format
« Reply #13 on: May 11, 2018, 11:52:22 AM »

@tiffy: The URL should be just the domain name without any prefixes. Certainly not http:// because that would be specifying the http protocol. Internally DSLstats adds ftp:// to the domain name when connecting. However your FTP server on the LAN is something I hadn't envisaged, so I'd need to think more about that.
Logged
  Eric

vic0239

  • Reg Member
  • ***
  • Posts: 374
Re: DSLstats data store/upload format
« Reply #14 on: May 11, 2018, 11:59:29 AM »

At the moment I am uploading to my NAS on the local LAN and the upload URL is simply the DNS name of the NAS which resolves to 192.168.1.xx. I've seen a lock up, but it was down to incorrect specification of the subsequent FTP parameters.
Logged
Pages: [1] 2
 

anything