Kitz Forum
Broadband Related => Router Monitoring Software => Topic started by: roseway on March 31, 2018, 03:00:12 PM
-
This version fixes all the reported issues on v6.3.4, and in addition it completes the FTP upload feature. All the files saved in the data store (or a selection of them) can be uploaded to an FTP server of the user's choice. Full list of changes:
- fixed transitional webserver code which sometimes omitted the path delimiter when upgrading from a previous version
- fixed duplicate event log messages about location of webserver files
- FTP test upload files are now named "FTPtest1" and "FTPtest2" and are not deleted after upload
- fixed the extra " in the error message "File .... failed to upload"
- fixed the error in graph time per page when closing and restarting DSLstats
- the names of the files in the data store now have the format "name-hours-minutes.zip"
- the default Snapshots folder is now <config files location>/Snapshots
- manual editing of snapshots, webserver and data store locations is now disallowed
- regular FTP upload of all stats is now enabled using daily folders on the server
- some small cosmetic changes
The FTP uploads are stored in daily folders on the server. DSLstats creates these daily folders as required, but the designated upload directory must exist before starting. The main stats are uploaded after each sample, as a single file for the day which grows after each sample. The optional per-tone zip files are uploaded once per hour.
I want this to be a release candidate for the next full release, so I would appreciate it if you could test it and report your findings.
Downloads available from http://dslstats.me.uk/downloads.html
-
Windows version running now. So far all OK. I have tested FTP and the test works. All directories are as expected. The FTP upload is working OK for Stats but windows complains it does not know what to open the file with as it has no extension, would be good to add one, also the data in the file is OK but alignment is off - there are two lines of data on the first line and all subsequent lines after first have an additional space in column 1 which might be problematic in parsing in future by program.
Also is there somewhere a document which shows what each value is in the Stats file?
Stuart
-
Eric, many thanks for the latest program revision.
Just a few initial observations while testing on a RPi 3B+ test bed before updating to my "working" RPi's:
The snapshots new default folder as stated /home/pi/.dslstats/Snapshots is not created, in the configuration / snapshots window the path (not now changable) is /home/pi/.dslstats, on forcing a few test snapshots they are indeed stored in /home/pi/.dslstats folder.
Have tried manually creating a "Snapshots" folder in file manager but snapshots are still written to /home/pi/.dslstats..
Note that there are now 2 folders in /home/pi/.dslstats, "webserver" and "Webserver", obviously the "Webserver" folder has been created by this program revision and the "webserver" folder by a previous program revision.
Can the "webserver" folder and it's contents now be deleated ?
Presumably the kitz webserver files would now have to be copied to the "Webserver" folder ?
On a more positive note, the graph time per page, hours anomaly after program re-starting which I reported is definately sorted, many thanks for that.
-
Version 6.3.5 has been downloaded and examined. I see that the CHANGELOG and mime.types files both have the execution bit unnecessarily set.
Actual testing is about to be started. :)
-
@Stuart: I've sent you a PM with an email address to send me a copy of your stats file. The reason is that I don't see the anomalies which you mention.
You can download a spreadsheet describing the format of the stats file at:
http://dslstats.me.uk/files/UploadStats.ods (http://dslstats.me.uk/files/UploadStats.ods)
Regarding the name of the stats file, I'd forgotten about Windows' funny habits. :) I'll change the name to 'stats.log' if that's OK with everyone.
-
@b*cat: I don't know how that happened, but noted.
-
@tiffy: The change in the snapshots folder is only the default, i.e. the value which is assigned to a new installation of DSLstats. If you already have an ini file from previous use, then your stored value will continue to be used.
The old 'webserver' folder can be deleted with all its contents. But anything you've manually added to it, such as the Kitz script, should be copied to the new folder first.
Thanks for confirming that the time per page saving and reloading works correctly.
-
there are two lines of data on the first line and all subsequent lines after first have an additional space in column 1 which might be problematic in parsing in future by program.
Just to confirm I have the same issue with my RPI version at least with 6.3.4 not tried with .5 yet. I noticed the extra spaces at the beginning of each line when first added but not noticed the two "lines" on the first line that is definitely the more troublesome one.
Edit: I guess if space is being used to indicate a value the extra spaces at the beginning of he lines are troublesome also.
-
@tiffy: The change in the snapshots folder is only the default, i.e. the value which is assigned to a new installation of DSLstats. If you already have an ini file from previous use, then your stored value will continue to be used.
Eric, thanks for the response.
I most likely have not expressed myself very well, what I am trying to say:
In Configuration / Snapshots / Snapshots Directory: the default path stated is now /home/pi/.dslstats/ I would have expected this to be /home/pi/.dslstats/Snapshots as per your revision description statement of 03:00 hrs. today.
In the previous program revision, v.6.3.4 and all previous revisions, I had edited the snapshots directory path to /home/pi/.dslstats/snapshots for better segregation of the snapshot files, just my preference.
I have checked the current dslstats.ini file and under [snapshots] the narrative is "Snapshotdir=/home/pi/.dslstats/" as would be expected with the current, default file path you have set.
I have verified that snapshots do currently save to /home/pi/.dslstats, again, as would be expected.
I'am very happy with your new default snapshot save location, was always my preference anyway, however, how do I get there now as I can no longer edit the file save path ?
Is there a possibility you could have just ommited to add the "/Snapshots" suffix to your default save path ?
-
Thanks eric. I downloaded it shortly after you posted and its been running ok here for the past 6hrs whilst I've been playing with getting something working so that it will run from a remote website.
I too noticed that the first 2 lines were differently spaced in the datastorage stats file that Stuart mentioned.
Presumably the kitz webserver files would now have to be copied to the "Webserver" folder ?
I just left that alone and only copied over the .exe and changelog file to my existing directory.
I always do it that way and only ever replace any files with a new date as I find it quicker to update that way.
-
I, too, see that the first two lines of the "DataStore/date/Stats" file are not separated by a new-line character and all subsequent lines beginning with a space character. Here follows the first three lines from my "Stats" file --
31/03/2018 19:18 5120 763 7.0 12.9 0.0 12.7 6008 992 ATM 0.00 0.00 0 0 0 0 21150 0 0 0 0 5.875 0 0 0 0 0 0 0 0 0 0 0 0 5451 0 0 32 8 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Showtime 8000 -1 L0 N/A ON ON No Showtime 44.5 27.1 160 95 1 1 1 1 16 8 1.0000 3.9809 1416 209 0 0 0 0 0 0 0 0 0 0 0 54 54 1.00 1.00 0 0 8 8 0 0 629 26 ADSL2 0 0 0 0 44.5 27.1 44.7 27.7 6008 992 0 0 0 0 0 0 31/03/2018 19:19 5120 763 6.7 12.8 0.0 12.7 5916 992 ATM 0.00 0.00 0 0 0 0 21209 0 0 32 0 5.891 0 0 32 0 0 0 0 0 0 0 0 0 5451 0 0 32 8 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Showtime 8000 -1 L0 N/A ON ON No Showtime 44.5 27.1 160 95 1 1 1 1 16 8 1.0000 3.9809 1416 209 0 0 0 0 0 0 0 0 0 0 0 54 54 1.00 1.00 0 0 8 8 0 0 651 26 ADSL2 22 0 0 0 44.5 27.1 44.7 27.7 5916 992 0 0 0 0 0 0
31/03/2018 19:20 5120 763 7.3 12.9 0.0 12.7 6048 992 ATM 0.00 0.00 0 0 0 0 21269 0 0 47 0 5.908 0 0 47 0 0 0 0 0 0 0 0 0 5451 0 0 32 8 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Showtime 8000 -1 L0 N/A ON ON No Showtime 44.5 27.1 160 95 1 1 1 1 16 8 1.0000 3.9809 1416 209 0 0 0 0 0 0 0 0 0 0 0 54 54 1.00 1.00 0 0 8 8 0 0 672 28 ADSL2 21 2 0 0 44.5 27.1 44.7 27.7 6048 992 0 0 0 0 0
As the "Stats" file is pure ASCII text, perhaps it could have a ".txt" extension?
-
Thanks, Eric. The work you are putting in is much appreciated.
Downloaded (clean install) and running on RPi Zero W, as expected.
Snapshots and Datastore set to save daily at 23:59, so have not observed new-line or extra space anomaly reported by others.
Snapshots, Datastore and Webserver files saved to appropriate folders on local NAS. Browsing to the correct directory when setting the Configuration was easy, and performed flawlessly.
Thank you again.
-
Eric the format of my stats file is the same as that posted by burakkucat, I'll email my file anyway.
Stuart
-
I most likely have not expressed myself very well, what I am trying to say:
In Configuration / Snapshots / Snapshots Directory: the default path stated is now /home/pi/.dslstats/ I would have expected this to be /home/pi/.dslstats/Snapshots as per your revision description statement of 03:00 hrs. today.
Did you delete (or rename) the present dslstats.ini file before running v6.3.5? As I said before, 'default' means running the program as a new user, without any existing configuration. I just tried this on my system, and it responded as described, allocating "/home/eric/.dslstats/Snapshots" as the snapshots directory. This was on a Linux system, not the Raspberry Pi, but the code is the same so the result on the RPi should be the same.
In the previous program revision, v.6.3.4 and all previous revisions, I had edited the snapshots directory path to /home/pi/.dslstats/snapshots for better segregation of the snapshot files, just my preference.
I have checked the current dslstats.ini file and under [snapshots] the narrative is "Snapshotdir=/home/pi/.dslstats/" as would be expected with the current, default file path you have set.
No, as I said, that isn't how it's set.
I have verified that snapshots do currently save to /home/pi/.dslstats, again, as would be expected.
I'am very happy with your new default snapshot save location, was always my preference anyway, however, how do I get there now as I can no longer edit the file save path ?
You can no longer manually edit the directory, but you can still click on the directory icon at the end of the box and select your preferred directory there.
-
The FTP upload is working OK for Stats but windows complains it does not know what to open the file with as it has no extension, would be good to add one, also the data in the file is OK but alignment is off - there are two lines of data on the first line and all subsequent lines after first have an additional space in column 1 which might be problematic in parsing in future by program.
I, too, see that the first two lines of the "DataStore/date/Stats" file are not separated by a new-line character and all subsequent lines beginning with a space character.
I too noticed that the first 2 lines were differently spaced in the datastorage stats file that Stuart mentioned.
The issue here is not the file, but the way you're reading it. There are no newline characters in the file, it's just a long string with the fields delineated with spaces. If you open it in a GUI text editor this will break lines according to its own rules. Stuart and Kitz - if you open it in Wordpad you won't see any leading spaces. Or any of you could open it in Libre Office Writer, and again there won't be any leading spaces. Change the font size and you'll see that the line breaks are in different places.
-
I'm using context editor which is usually quite good at this stuff and having opened it in wordpad I see the same - first block is two records then each record is a block of data on its own starting with a blank, just as context shows it. So I'm not sure what to think now every which way I open it the same format shows.
Stuart
-
I have tried opening the "Stats" file in "Libre Office Calc" and it, too, shows the first two records concatenated together . . .
This is what "file" and "wc" utilities tell me about the "Stats" file --
[Duo2 2018-03-31]$ file Stats
Stats: ASCII text, with very long lines
[Duo2 2018-03-31]$ wc -c Stats
118111 Stats
[Duo2 2018-03-31]$ wc -w Stats
39198 Stats
[Duo2 2018-03-31]$ wc -l Stats
278 Stats
[Duo2 2018-03-31]$
Notice that the "file" utility states "with very long lines" (plural) and that the "wc" utility confirms that the file contains 278 lines. Lines which end with a character with the hexadecimal value 0x0a. Confirmed by examining the "Stats" file with "hexdump -C". ???
-
I open the file in nano on RPI (save each minute in same file option is enabled still .4 release), on the first line there is 31/03/2018 00:00 ...values.... and 31/03/2018 00:01 ....values.... entries and then on 2nd line the 31/03/2018 00:02 starts at the 3rd character. The 3rd line starts at the 3rd character with 31/03/2018 00:03 and repeat. I will open up the file in Notepad++ on a Windows machine to check what line endings there are and whether they are wrong in the morning?
-
I have opened the stats file in LibreOffice on a Pi.
Each data sample begins with a date and time. Between the first and second data sample there are 3 space characters. Between the second and third data sample there is a space, a paragraph marker, then two further spaces. The same applies between all following data samples.
The last data sample recorded concludes with a space and a paragraph marker, and I assume the next data sample to be appended will start with two spaces.
It seems that the second and subsequent data samples conclude with a paragraph marker as they are added, but the first data sample doesn't.
I'm sorry to contradict you, Eric, but there does appear to be a bug. :(
-
The issue here is not the file, but the way you're reading it. There are no newline characters in the file, it's just a long string with the fields delineated with spaces. If you open it in a GUI text editor this will break lines according to its own rules. Stuart and Kitz - if you open it in Wordpad you won't see any leading spaces. Or any of you could open it in Libre Office Writer, and again there won't be any leading spaces. Change the font size and you'll see that the line breaks are in different places.
I don't use wordpad. I use various text editors which don't add their own spacing such as notepad++ and glogg.
No matter what I do, I always see the same format.
I've tried stretching the page width. Ive tried changing the fonts. Ive turned font wrap on and off. I even dug out word pad to view in that but its still the same.
I attach below various screen caps which show the issue.
Attachments
1) Viewing in glogg. Note 00:00 & 00:01 are always on the same line. I can stretch and change the format but the issue remains
2) Viewing in notepad++ with wordwrap off
3) Installed wordpad. Issue still there
cont in next post//
4) Viewing in notepad++ showing 00:00 and 00:01 on same line. CRLF separates for 00:02 onwards
5) Viewing in notepad++ showing 00:00 and 00:01 has three spaces
-
///cont... its taken me ages to capture all these and no-one else's reply was there when I started.. but since I took the time to capture them may as well post :)
-
I'm sorry, we've been talking at cross-purposes. I was talking about the single Stats file representing one sample, so I misunderstood what you were all saying.
The format of the long Stats file representing up to a day's worth of samples is supposed to separate the samples with two spaces, and within each sample the fields are separated by single spaces. Looking at my own Stats files I see that the intended format is adhered to sometimes, but at other times a line feed (hex 0A) has crept in. That's on a Linux system which uses LF as line breaks, so I can believe that on Windows systems CRLF appears sometimes. I was wrong, so I'm sorry. :-[
I'm going to take a bit of time now to resolve this. Please continue to report any other issues.
-
Did you delete (or rename) the present dslstats.ini file before running v6.3.5? As I said before, 'default' means running the program as a new user, without any existing configuration. I just tried this on my system, and it responded as described, allocating "/home/eric/.dslstats/Snapshots" as the snapshots directory. This was on a Linux system, not the Raspberry Pi, but the code is the same so the result on the RPi should be the same.
No, did not delete, rename or modify the existing v.6.3.4 dslstats.ini file as per my normal program change procedure.
You can no longer manually edit the directory, but you can still click on the directory icon at the end of the box and select your preferred directory there.
Ahh, while realising that the manual path edit was now disabled as per your revision just assumed that the directory icon would now also be disabled !
As you have said, not the case, path now changed to /home/pi/.dslstats/Snapshots and test snapshots save location verified.
Apologies for adding to your already substantial work load associated with the new program revision and taking up time with my minor queries, your support is very much appreciated.
-
No apology needed, I probably didn't explain clearly enough. :)
-
Has anyone else got an issue with the 2 day ES graph? Mine is not showing any data for yesterday but it has been working fine? Has the change over to the 1st of the month caused an issue?
I do not know how it is supposed to be working but can I suggest that each minute of data is stored on a separate line instead of one huge line? A huge line is fine for a computer but for me it is much easier to compare values scrolling down.
DSLstats is great and thanks for the continued development.
-
Hi. Same here No ES for yesterday following midnight. I guess this is related to the month end cross over. I expect it would correct today. Somthing im sure Eric can sort in due course as it’s another 30 days until next month 😄😄
Tony
-
Yup same here failng to show 31st March data.
Stuart
-
Yes, same here on my 3 active RPi's (different lines/locations), currently still running v.6.3.4 before updating today.
-
Yes, I agree. It will be fixed within 30 days. :)
By the way, looking back over the previous 100 days, I see that the graph works properly at the beginning of January, February and March. It's only today (1 April) that it fails to show the last day of the previous month.
-
I do not know how it is supposed to be working but can I suggest that each minute of data is stored on a separate line instead of one huge line? A huge line is fine for a computer but for me it is much easier to compare values scrolling down.
This will be referring to the data store Stats file. What do others think? At this stage I can do it either way.
-
Eric I think the single line per minute method works well for both human readers and parsing the file with something like PHP especially as the data is space delimited.
Stuart
-
I think one sample per line would make the file much easier to import or use with other programs.
-
There's one small thing to be aware of: Linux systems use LF for a line break, but Windows uses CRLF. So if you're collecting the data using a Raspberry Pi but processing it using Windows, you'll have to deal with LF line breaks. Is that a problem?
-
If anyone gets to the stage of wanting to draw their own graphs from google from the data then you'd need something like a CRLF delimiter to break down the data anyhow.
I haven't properly thought that far in advance yet.... but if reading data from the file then I'd use space delimiter to break down the various elements into an array and a CRLF delimiter to identify and separate the various arrays for each data row. If it was all spaces then it will be harder to search for the end of each separate row of data before breaking down the arrays into relevant fields.
TBH I thought you'd used CRLF on purpose because it makes it easier programming wise to do something with the data.. and easier on the human eye to read it.
----
ETA
Another example of how long it takes me to type these days.. there were no other responses about using single line methods when I started. Sorry for continually being late to the party :(
-
There's one small thing to be aware of: Linux systems use LF for a line break, but Windows uses CRLF. So if you're collecting the data using a Raspberry Pi but processing it using Windows, you'll have to deal with LF line breaks. Is that a problem?
I dont believe it is an issue especially when using PHP, at least I've not found one ;)
Stuart
-
OK, thanks for all the comments. I'll go ahead with line breaks as dividers between the sets of data.
-
In the "Event Log", is there a reason for the blank line that separates the "Recording Started" and the "IP address is now . . ." lines?
Example --
01 Apr 2018 16:04:39 Recording started
01 Apr 2018 16:04:43 IP address is now W.X.Y.Z
-
If there was a reason at one time I don't remember what it was. I'll put it on the list to correct, but it will be at the end of the list. :)
-
Thank you. :)
-
I found myself staring at the "Event Log" as something seemed to be slightly askew. Here is a screen-scrape --
01 Apr 2018 16:03:17 Configuration files stored in /home/bcat/.dslstats/
01 Apr 2018 16:03:17 The configured modem/router is Zyxel VMG1312-B10A
01 Apr 2018 16:03:17 Snapshots folder is /home/bcat/lib/dslstats64L-6.3.5/Snapshots/
01 Apr 2018 16:03:17 Webserver folder is /home/bcat/lib/dslstats64L-6.3.5/Webserver/
01 Apr 2018 16:03:17 Data store location is /home/bcat/lib/dslstats64L-6.3.5/DataStore/
01 Apr 2018 16:04:39 Recording started
01 Apr 2018 16:04:43 IP address is now W.X.Y.Z
Eventually I realised that it was the difference between the lines referencing the "Snapshots" & "Webserver" folders and the "Data store location".
For consistency, could those three lines use the same teminology? For example --
01 Apr 2018 16:03:17 Snapshots folder is /home/bcat/lib/dslstats64L-6.3.5/Snapshots/
01 Apr 2018 16:03:17 Webserver folder is /home/bcat/lib/dslstats64L-6.3.5/Webserver/
01 Apr 2018 16:03:17 DataStore folder is /home/bcat/lib/dslstats64L-6.3.5/DataStore/
-- or --
01 Apr 2018 16:03:17 Snapshots location is /home/bcat/lib/dslstats64L-6.3.5/Snapshots/
01 Apr 2018 16:03:17 Webserver location is /home/bcat/lib/dslstats64L-6.3.5/Webserver/
01 Apr 2018 16:03:17 DataStore location is /home/bcat/lib/dslstats64L-6.3.5/DataStore/
-
OK, it's on the list.
-
Thank you. :)