Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Chrysalis]

Ok I respect this request may not be doable, but here it is.

If i leave the log to grow for months on end and then try to create graphs, it takes ages, whilst disk is thrashed.  The logs in my case are stored on a windows software raid1, which is split over 2 WD red drives.  I watched resource monitor whilst waiting for one of the logs to be read (700meg log for 10 months of stats), and the disk queue was 30k for both hdd's, an insane queue.  If its a uncached read the log will take about 4 minutes to be read, meaning if I am creating a graph, it takes 4 mins also.

So my request is to add an optional monthly log rotation. (or even weekly).  Perhaps also for hdd space maintenance make it so when it does a rotation to delete the oldest logs, so e.g. if configured to keep 3 logs, then the 4th log gets deleted when a rotation is made.  I expect what will make this a pain tho if making a graph where the data is split over 2 logs.

[Chrysalis]

ok here is possibly how it could be done, an idea to make the request easy to implement.

If monthly rotation enabled, make the first stats run on the 1st of month rename the current log and create a new log.
When running ongoing stats, add an optional question to ask which log to use, the default (if nothing entered is current log) 0 would be current log, 1 would be previous month log and so on.

Sounds good?

[Ronski]

Are you sure you only have 10 months of data in your log?

I split mine on a yearly basis - I was adding this as a feature to the GUI but simply haven't had time recently to finish it. Anyway my yearly logs for 2013 & 2014 are both just under 300MB, my current 2015 log is 158MB in size. This suggests your log contains more than 10 months of data.

Eventually I would like to add the ability to the GUI to graph a given period, but I don't know when I'll get time to do this.

[Bald_Eagle1]

Adding the G.INP related has significantly increased the data stored for each minute's sample.

Pre-G.INP, my connection stored around 630 bytes of data per row.
With G.INP active, around 900 bytes per sample are stored.

900 bytes per sample x 1440 samples per day x 365 days per year equates to around 473,040,000 bytes per year.

Divided by 1024, that equates to around 462 MB per year.


Ronski's connection doesn't use G.INP (ECI Dslam), so his every minute samples will be smaller than mine (simple zero values used where G.INP related data would go if his connection could use it).


Splitting modem_stats.log at pre-set intervals IS on my things to do list (when I eventually get round to it - it's a long list).
Ronski does very politely & kindly remind me about this every now & then 


As each row of data is a variable length, graphpd.exe has to firstly read every row to count the total number of rows & then go back to the row where the data to be plotted starts.
e.g. 1440 rows from the end for 1 day or 720 rows from the end for 12 hours etc. etc.




It doesn't take long on my PC though:-

 The whole log file contains 226320 rows

 Please wait......

 DSL_MODE = VDSL2

 Please wait......

 row_num        1 - 23:23:05.746

 1440 samples processed.

 row_num   226320 - 23:23:07.072



This is how long the whole process took to plot the data for the latest 24 hours:-

07/06/2015 23:28:45.872 - Start of [graphpd.exe] v 5.1.0.1
07/06/2015 23:29:19.896 - End of graphpd. 1440 samples i.e. 1 Day of data should have been plotted.



& this long for the latest 30 days:-

07/06/2015 23:31:33.096 - Start of [graphpd.exe] v 5.1.0.1
07/06/2015 23:34:07.989 - End of graphpd. 43200 samples i.e. 30 Days of data should have been plotted.


How many days of data do you plot at a time?

It's actually the graphing program used by graphpd.exe (gnuplot) that takes the most time.
Reading the log & creating the temporary data file that is used for the plotting should only take a few seconds.


I will get round to splitting modem_stats.log into smaller chunks at some stage, but TBH, it's not my highest priority just at the moment.

 

[NewtronStar]

He could turn off both ticks in the extra debugging info logging tab.

[Chrysalis]

both ticks are already off newt and I was graphing 1 day.

The amount of data been graphed actually has no impact, as my cpu is fast, its simply having to read the log that is so slow.

Also its the ongoing log I think causing the issue not the error log.

I did look for tools that can rotate files for windows, ideally I prefer not to use them and hope the tool can include such a function.

Since I still have the log backed up I loaded it in notepad2 to check the # of lines and is 415k lines.

Part of the problem may well be down to windows software raid which I think is bad for performance, but it is what it is currently, I am not ditching the raid.  It is fine when the log file is cached as just now it took about a second to open in notepad2 but when it isnt cached wow the amount of i/o to load it is insane.

[Bald_Eagle1]

Part of the problem may well be down to windows software raid which I think is bad for performance, but it is what it is currently, I am not ditching the raid.  It is fine when the log file is cached as just now it took about a second to open in notepad2 but when it isnt cached wow the amount of i/o to load it is insane.

I'm not at all familiar with RAID systems.
A quick Google tells me that some versions are used for mirroring, some used for striping etc. etc. etc, using hardware or software controllers.



What are the advantages/disadvantages of using RAID & why do you need to keep using whatever flavour that you personally employ?


4 minutes to plot 1 day's worth of ongoing data is a ridiculous amount of time, especially as I am aware that you have a very fast PC (CPU-wise) for which my data harvesting programs need slowing down!

The graphing programs (snapshot and ongoing) are not slowed down at all.


Just wondering if it might be preferable to store the ongoing modem_stats.log on a USB memory stick or a USB hard drive (that is not part of the RAID setup) to help speed things up?

[Chrysalis]

As I said BE the entire log has to be read regardless of if its 1 hour graph, 1 day graph or 1 month graph.  The slowdown is i/o related not cpu related.

If I make a graph using a small ongoing log (or if its cached) its fast, if its a large ongoing log (e.g. 1 year) and is not cached it is very slow.

Its raid1 I am using, I use it for my data that I want redundancy on such as my modem stat logs.  Using a usb stick is not a good idea as the data gets written every minute and flash memory has limited writes.

[Bald_Eagle1]

O.K. I'll see what I can do with this.

It may be some time before I get round to it though.

[Chrysalis]

thanks, I am considering making a batch file to have the ramdisk host the log, but it would be very complex and me subject to losing updates.

I will do some tests to see how fast the log file can be read uncached from a ssd as well as a standalone hdd.

[Bald_Eagle1]

If you have the daily graphing option enabled, could you please also manually run a 30 day graphing session using the attached version of graphpd.exe?

Then post an extract from your graphpd_ERROR.LOG that includes the scheduled run AND the manual 30 day run?

The attached version corrects a bug where the wrong data was being used for Upstream SES graphs and it should provide details of exactly where the slowdown occurs.
This will help me to focus on specific parts of the program to try to improve matters.


The attached log shows typical timing using my modem_stats.log which is around 181,307 KB in size.

[Chrysalis]

so if I put my old log back and tell it do 32 days, thats ok? (as last 2 days not on the old log)

I assume you prefer me to reboot first as well to make sure its not in the cache.

[Bald_Eagle1]

it doesn't have to be exactly 30 days.
That was just my suggestion of a reasonably large sample period to compare against mine to help identify the issue.

30 days would equate to 43,200 samples.
Anywhere around that number of rows should give a good indication.

I hadn't though about rebooting to clear the cache, but it does seem like a good idea now you mention it.




The way the program currently works is this:-

* The whole log is read to count the number of rows.

* The number of rows to be plotted is then deducted from that total to provide the starting row for graphing.

* The log is then read from the first row again until that point is reached.

* The rest of the log is then read row by row, adding each row to a temporary file until the end of the log is reached.

* The graphs are then generated from the data in the temporary file then the temporary file is deleted.



That's quite an inefficient way to do things, but as it usually only takes a few seconds on my 'ordinary' specced PC, I never considered it to be a big issue (until you mentioned it).


I have some ideas about how to make it more efficient, especially when using a large log file, but I'd just like to see the timestamps in graphpd_ERROR.LOG to be 100% certain that I'm looking at the right places.


Please remind me which version of Windows you are using as it may/may not be relevant.

[Ronski]

Just a thought, but could we not graph a given period.

The log file is read line by line
When the start date/time is reached we start writing to the temporary file counting the lines as we go if required
we stop writing each line to the temporary file when the finish date/time is reached

The above would be quicker, and would only graph the period required,  the current method is rather hit and miss if you don't graph 24/7. IE if you only log 12 hours a day graphing one day actually graphs two days stats.

Obviously if we're splitting logs then that would need to be taken into account.

[NewtronStar]

Have just done a 30 day graphpd it took 2 minutes and 15 seconds this is using 5.1.0.0 the lastest version.