Kitz Forum

Broadband Related => Router Monitoring Software => Topic started by: roseway on May 21, 2018, 10:53:51 AM

Title: DSLstats pre-release version 6.5.6
Post by: roseway on May 21, 2018, 10:53:51 AM
The main change in this version is that SFTP uploads are now working. An external program (curl) is used for this. Linux and Raspberry Pi users must install curl on their systems for this to work. This needs to be built with SSH support, and it's assumed that this will be the case with most modern Linux systems. An appropriate version of curl is included in the Windows version, and all the files in the archive should be copied to their destination.

Full list of changes since v6.5.2:
http://dslstats.me.uk/downloads.html
Title: Re: DSLstats pre-release version 6.5.6
Post by: broadstairs on May 21, 2018, 12:51:07 PM
Running here OK although only using plain FTP currently.

Stuart
Title: Re: DSLstats pre-release version 6.5.6
Post by: tiffy on May 21, 2018, 12:57:31 PM
@roseway:
Eric, many thanks for your latest program revision and hard work on the project, very much appreciated.

Ref. installing "curl" on RPi's:
Currently running a RPi 3B+ with Raspbian Stretch on my home system, tried to install curl by the simple method:
sudo apt-get update
sudo apt-get install php5-curl  (errors produced, would not install)

Googled RPi support forums and this was suggested:
sudo apt-get update
sudo apt-get install libcurl4-openssl-dev  (this installed without errors)

Tried the same "curl" install process on one of my remote RPi Zero W's still running Raspbian Jessie:
Both "curl" install options worked without any errors, installed the later "libcurl4" last so presume it will be active.

Unfortunately, I still have the same issue with WS Upload to my LAN FTP server on my Win 8.1 desktop PC, all permutations of FTP, Upload URL & Upload Dir still produce a test failure although the previously reported program "lock up" when using PC IP address has now been cured.

Otherwise the program revision is up and running on all my RPi's with no noted anomalies to date.
Title: Re: DSLstats pre-release version 6.5.6
Post by: marjohn56 on May 21, 2018, 02:18:20 PM
Nice one Eric.. excellent work, thank you very much.
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 21, 2018, 03:41:52 PM
@tiffy: To install curl, this is what you should do:

Code: [Select]
sudo apt-get update
sudo apt-get install curl

This installs curl and libcurl3. You don't need the other packages you mentioned.
Title: Re: DSLstats pre-release version 6.5.6
Post by: marjohn56 on May 21, 2018, 03:48:39 PM
Eric, I appear to have an issue with data upload, The stats.log is being uploaded into the data folder, i.e. "server/DSLstats_data" not "server/DSLstats_data/2018-05-21", are you aware of this?

Also I am seeing fec.html and crc.html in the DSLstats_data folder too...
Title: Re: DSLstats pre-release version 6.5.6
Post by: tiffy on May 21, 2018, 03:51:39 PM
@tiffy: To install curl, this is what you should do:

Code: [Select]
sudo apt-get update
sudo apt-get install curl

This installs curl and libcurl3.

OK Eric, even more simple and obvious !
Was not sure on the format so Googled and got the options quoted.
Just making the point that Raspbian Stretch and Jessie appeared to react differently to the first option.

Will try the obvious option you suggested on my second remote RPi Zero W which is not updated yet.
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 21, 2018, 04:16:04 PM
Eric, I appear to have an issue with data upload, The stats.log is being uploaded into the data folder, i.e. "server/DSLstats_data" not "server/DSLstats_data/2018-05-21", are you aware of this?

Also I am seeing fec.html and crc.html in the DSLstats_data folder too...

On the second point, those two files are uploaded when you press the Test button. I meant to change their names to test1 and test2 to make their function clear, but didn't get around to it.

On the first point, I didn't specifically test the data store uploads this time because I didn't think the behaviour had changed in this version, but it looks as though I was wrong. I'll look into this and report back.
Title: Re: DSLstats pre-release version 6.5.6
Post by: broadstairs on May 21, 2018, 05:16:29 PM
Yes the FTP of stats.log goes to the datastore directory not the daily directory as it used to, however the local file is correctly in the datastore directory in a daily directory.

Stuart
Title: Re: DSLstats pre-release version 6.5.6
Post by: marjohn56 on May 21, 2018, 05:38:19 PM
On the second point, those two files are uploaded when you press the Test button. I meant to change their names to test1 and test2 to make their function clear, but didn't get around to it.

You're just trying to confuse me.. :)

On the first point, I didn't specifically test the data store uploads this time because I didn't think the behaviour had changed in this version, but it looks as though I was wrong. I'll look into this and report back.

Cool. I think Ned has PM'd you with more info so I know your on the case.

KR
Title: Re: DSLstats pre-release version 6.5.6
Post by: Chrysalis on May 22, 2018, 01:11:47 PM
Special thanks from me to Eric on the continued work he is putting into dslstats.

The SFTP support significantly lowers resource usage on both client side and server side for uploads vs standard FTP as well as speeding it up, so this will help greatly when we get a new centralised service up and running.
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 22, 2018, 01:29:29 PM
Thanks Chris.
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 22, 2018, 07:11:03 PM
Concerning the Data Store upload issue, I've located the cause and fixed it for the next release.
Title: Re: DSLstats pre-release version 6.5.6
Post by: g3uiss on May 22, 2018, 09:28:59 PM
Stuart

I found but after  one upload using SFTP , DSLstats hung, needing a forced restart. Then I found all the SFTP settings were blank, I've tried twice with the same results

However Iíve just realised my hosting needs FTPS I assume thatís not included. I can connect and upload using P21 and the middle option, but as above crashes after one upload.

Tony
Title: Re: DSLstats pre-release version 6.5.6
Post by: Browni on May 22, 2018, 10:29:56 PM
This one crashes out with the error "" is an invalid integer, looks like that's been there since build 6.5.2

I believe you narrowed that down to pbparams and the figures I see on 6.5 using a G.fast connection are here

Code: [Select]
bcm_xdslctl info --pbParams
bcm_xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    0
Last initialization procedure status:   0
Max:    Upstream rate = 34767 Kbps, Downstream rate = 198057 Kbps
Bearer: 0, Upstream rate = 34651 Kbps, Downstream rate = 198057 Kbps

Discovery Phase (Initial) Band Plan
US: (368,1930)
DS: (368,1930)
Medley Phase (Final) Band Plan
US: (368,1339)
DS: (368,1211)
VDSL Port Details               Upstream                Downstream
Attainable Net Data Rate:        34767 kbps             198057 kbps
Actual Aggregate Tx Power:          2.8 dBm                 0.0 dBm
====================================================================================
  VDSL Band Status U0 U1 U2 U3 U4 D1 D2 D3
  Line Attenuation(dB): N/A 0.0 N/A N/A N/A 37.7 N/A N/A
Signal Attenuation(dB): N/A 0.0 N/A N/A N/A 37.7 N/A N/A
        SNR Margin(dB): N/A 0.0 N/A N/A N/A 0.0 N/A N/A
         TX Power(dBm): N/A 0.0 N/A N/A N/A 0.0 N/A N/A
        kl0(dBx100):      N/A     N/A     N/A     N/A     N/A     N/A     N/A     N/A   
/ #


New territory!
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 23, 2018, 07:27:33 AM
As you say, this is new territory. G.Fast isn't VDSL2, so I guess that my formal answer at present has to be that DSLstats doesn't support G.Fast. And to be honest I don't know if I've got the energy/capability/time to add that support. I will see if I can locate the source of that error and trap it, but I'm afraid it will take some time.
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 23, 2018, 11:22:20 AM
Stuart

I found but after  one upload using SFTP , DSLstats hung, needing a forced restart. Then I found all the SFTP settings were blank, I've tried twice with the same results

However Iíve just realised my hosting needs FTPS I assume thatís not included. I can connect and upload using P21 and the middle option, but as above crashes after one upload.

Tony

It was my understanding that FTPS is deprecated these days. I can add SSL support if it's really required, but I would appreciate advice on its security status.
Title: DSLstats pre-release version 6.5.6
Post by: d2d4j on May 23, 2018, 11:37:32 AM
Hi roseway

Kudos to you for sftp/curl which works lovely on our platforms

I thought only implicit ftps was deprecated and explicit still ongoing

I suppose in reality, there will always be a mixture but using explicit ftps, could allow for insecure ftp to be used if secure handshake fail, whereas implicit ftps would drop connection

We do not use ftps, only sftp as sftp is to us, a better system

Many thanks

John
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 23, 2018, 12:25:07 PM
Thanks for that, John.
Title: Re: DSLstats pre-release version 6.5.6
Post by: g3uiss on May 23, 2018, 12:55:06 PM
It was my understanding that FTPS is deprecated these days. I can add SSL support if it's really required, but I would appreciate advice on its security status.

Well I use 1&1 on a hosted MS Server 2016, and the only permitted access is FTPS on Port 990. I'm not sure on the security, but its quite a major provider and this is a "Pro" package. They changed from FTP around 1 year ago

Tony
Title: Re: DSLstats pre-release version 6.5.6
Post by: d2d4j on May 23, 2018, 01:37:09 PM
Hi g3uiss

Thatís a window server so explicit ftps would be used, so it has nothing to do with size of provider

This is most likely why explicit ftps is still in use, and as you stated, fallback to 21 works

Many thanks

John
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 23, 2018, 03:15:26 PM
Thanks for the extra information. I'll add SSL support in the next release.
Title: Re: DSLstats pre-release version 6.5.6
Post by: g3uiss on May 23, 2018, 05:53:19 PM
Hi g3uiss

Thatís a window server so explicit ftps would be used, so it has nothing to do with size of provider

This is most likely why explicit ftps is still in use, and as you stated, fallback to 21 works

Many thanks

John

John. Thanks. I was meaning to say exactly that and suggested that I would expect that provider to have the required security. I believe they moved from Windows2012 which still supported FTP or rather my Server 2012 does 😄

Thanks to Eric in advance for adding the support.

Tony
Title: Re: DSLstats pre-release version 6.5.6
Post by: marjohn56 on May 25, 2018, 08:27:33 AM
Minor issue raised its head yesterday, something prevented the sftp sessions from connecting. After four attempts dslstats gives up.

Would it be possible to send a warning email when this happens?

@roseway - I realise the list of requests and additions is never ending and this is not a major issue.
Title: Re: DSLstats pre-release version 6.5.6
Post by: roseway on May 25, 2018, 09:59:58 AM
@marjohn56: I'll see what I can do. The logical thing would be to add it to the alerts system.