Kitz Forum
Broadband Related => Router Monitoring Software => Topic started by: Chrysalis on June 07, 2015, 09:52:28 PM
-
Ok I respect this request may not be doable, but here it is.
If i leave the log to grow for months on end and then try to create graphs, it takes ages, whilst disk is thrashed. The logs in my case are stored on a windows software raid1, which is split over 2 WD red drives. I watched resource monitor whilst waiting for one of the logs to be read (700meg log for 10 months of stats), and the disk queue was 30k for both hdd's, an insane queue. If its a uncached read the log will take about 4 minutes to be read, meaning if I am creating a graph, it takes 4 mins also.
So my request is to add an optional monthly log rotation. (or even weekly). Perhaps also for hdd space maintenance make it so when it does a rotation to delete the oldest logs, so e.g. if configured to keep 3 logs, then the 4th log gets deleted when a rotation is made. I expect what will make this a pain tho if making a graph where the data is split over 2 logs.
-
ok here is possibly how it could be done, an idea to make the request easy to implement.
If monthly rotation enabled, make the first stats run on the 1st of month rename the current log and create a new log.
When running ongoing stats, add an optional question to ask which log to use, the default (if nothing entered is current log) 0 would be current log, 1 would be previous month log and so on.
Sounds good?
-
Are you sure you only have 10 months of data in your log?
I split mine on a yearly basis - I was adding this as a feature to the GUI but simply haven't had time recently to finish it. Anyway my yearly logs for 2013 & 2014 are both just under 300MB, my current 2015 log is 158MB in size. This suggests your log contains more than 10 months of data.
Eventually I would like to add the ability to the GUI to graph a given period, but I don't know when I'll get time to do this.
-
Adding the G.INP related has significantly increased the data stored for each minute's sample.
Pre-G.INP, my connection stored around 630 bytes of data per row.
With G.INP active, around 900 bytes per sample are stored.
900 bytes per sample x 1440 samples per day x 365 days per year equates to around 473,040,000 bytes per year.
Divided by 1024, that equates to around 462 MB per year.
Ronski's connection doesn't use G.INP (ECI Dslam), so his every minute samples will be smaller than mine (simple zero values used where G.INP related data would go if his connection could use it).
Splitting modem_stats.log at pre-set intervals IS on my things to do list (when I eventually get round to it - it's a long list).
Ronski does very politely & kindly remind me about this every now & then ;) :)
As each row of data is a variable length, graphpd.exe has to firstly read every row to count the total number of rows & then go back to the row where the data to be plotted starts.
e.g. 1440 rows from the end for 1 day or 720 rows from the end for 12 hours etc. etc.
It doesn't take long on my PC though:-
The whole log file contains 226320 rows
Please wait......
DSL_MODE = VDSL2
Please wait......
row_num 1 - 23:23:05.746
1440 samples processed.
row_num 226320 - 23:23:07.072
This is how long the whole process took to plot the data for the latest 24 hours:-
07/06/2015 23:28:45.872 - Start of [graphpd.exe] v 5.1.0.1
07/06/2015 23:29:19.896 - End of graphpd. 1440 samples i.e. 1 Day of data should have been plotted.
& this long for the latest 30 days:-
07/06/2015 23:31:33.096 - Start of [graphpd.exe] v 5.1.0.1
07/06/2015 23:34:07.989 - End of graphpd. 43200 samples i.e. 30 Days of data should have been plotted.
How many days of data do you plot at a time?
It's actually the graphing program used by graphpd.exe (gnuplot) that takes the most time.
Reading the log & creating the temporary data file that is used for the plotting should only take a few seconds.
I will get round to splitting modem_stats.log into smaller chunks at some stage, but TBH, it's not my highest priority just at the moment.
-
He could turn off both ticks in the extra debugging info logging tab.
-
both ticks are already off newt and I was graphing 1 day.
The amount of data been graphed actually has no impact, as my cpu is fast, its simply having to read the log that is so slow.
Also its the ongoing log I think causing the issue not the error log.
I did look for tools that can rotate files for windows, ideally I prefer not to use them and hope the tool can include such a function.
Since I still have the log backed up I loaded it in notepad2 to check the # of lines and is 415k lines.
Part of the problem may well be down to windows software raid which I think is bad for performance, but it is what it is currently, I am not ditching the raid. It is fine when the log file is cached as just now it took about a second to open in notepad2 but when it isnt cached wow the amount of i/o to load it is insane.
-
Part of the problem may well be down to windows software raid which I think is bad for performance, but it is what it is currently, I am not ditching the raid. It is fine when the log file is cached as just now it took about a second to open in notepad2 but when it isnt cached wow the amount of i/o to load it is insane.
I'm not at all familiar with RAID systems.
A quick Google tells me that some versions are used for mirroring, some used for striping etc. etc. etc, using hardware or software controllers.
What are the advantages/disadvantages of using RAID & why do you need to keep using whatever flavour that you personally employ?
4 minutes to plot 1 day's worth of ongoing data is a ridiculous amount of time, especially as I am aware that you have a very fast PC (CPU-wise) for which my data harvesting programs need slowing down!
The graphing programs (snapshot and ongoing) are not slowed down at all.
Just wondering if it might be preferable to store the ongoing modem_stats.log on a USB memory stick or a USB hard drive (that is not part of the RAID setup) to help speed things up?
-
As I said BE the entire log has to be read regardless of if its 1 hour graph, 1 day graph or 1 month graph. The slowdown is i/o related not cpu related.
If I make a graph using a small ongoing log (or if its cached) its fast, if its a large ongoing log (e.g. 1 year) and is not cached it is very slow.
Its raid1 I am using, I use it for my data that I want redundancy on such as my modem stat logs. Using a usb stick is not a good idea as the data gets written every minute and flash memory has limited writes.
-
O.K. I'll see what I can do with this.
It may be some time before I get round to it though.
-
thanks, I am considering making a batch file to have the ramdisk host the log, but it would be very complex and me subject to losing updates.
I will do some tests to see how fast the log file can be read uncached from a ssd as well as a standalone hdd.
-
If you have the daily graphing option enabled, could you please also manually run a 30 day graphing session using the attached version of graphpd.exe?
Then post an extract from your graphpd_ERROR.LOG that includes the scheduled run AND the manual 30 day run?
The attached version corrects a bug where the wrong data was being used for Upstream SES graphs and it should provide details of exactly where the slowdown occurs.
This will help me to focus on specific parts of the program to try to improve matters.
The attached log shows typical timing using my modem_stats.log which is around 181,307 KB in size.
-
so if I put my old log back and tell it do 32 days, thats ok? (as last 2 days not on the old log)
I assume you prefer me to reboot first as well to make sure its not in the cache.
-
it doesn't have to be exactly 30 days.
That was just my suggestion of a reasonably large sample period to compare against mine to help identify the issue.
30 days would equate to 43,200 samples.
Anywhere around that number of rows should give a good indication.
I hadn't though about rebooting to clear the cache, but it does seem like a good idea now you mention it.
The way the program currently works is this:-
* The whole log is read to count the number of rows.
* The number of rows to be plotted is then deducted from that total to provide the starting row for graphing.
* The log is then read from the first row again until that point is reached.
* The rest of the log is then read row by row, adding each row to a temporary file until the end of the log is reached.
* The graphs are then generated from the data in the temporary file then the temporary file is deleted.
That's quite an inefficient way to do things, but as it usually only takes a few seconds on my 'ordinary' specced PC, I never considered it to be a big issue (until you mentioned it).
I have some ideas about how to make it more efficient, especially when using a large log file, but I'd just like to see the timestamps in graphpd_ERROR.LOG to be 100% certain that I'm looking at the right places.
Please remind me which version of Windows you are using as it may/may not be relevant.
-
Just a thought, but could we not graph a given period.
The log file is read line by line
When the start date/time is reached we start writing to the temporary file counting the lines as we go if required
we stop writing each line to the temporary file when the finish date/time is reached
The above would be quicker, and would only graph the period required, the current method is rather hit and miss if you don't graph 24/7. IE if you only log 12 hours a day graphing one day actually graphs two days stats.
Obviously if we're splitting logs then that would need to be taken into account.
-
Have just done a 30 day graphpd it took 2 minutes and 15 seconds this is using 5.1.0.0 the lastest version.
-
I will get back to you as soon as possible BE but last day or so I been very busy.
But it is important whenever you test i/o performance to make sure is uncached Reading an entire log is indeed inefficient.
Will try to get you the info you requested by end of tonight.
-
Please try the attached version of graphpd.exe.
This version now starts at the end of the file & works backward to find the start point of the data to be used for the period specified.
In other words, it does not read the whole file to count the total rows & then read it again until it finds the starting row for the period specified.
It is slightly quicker, but it doesn't seem to make all that much difference on my system when working with a modem_stats.log that is around 185 MB in size.
It might make a bigger difference on your RAID system though as the file reading element is now much reduced.
I'd be very interested to see your graphpd_ERROR.LOG entries for the old version AND this new version in order to compare the timings.
-
will do as soon as I can, as I need to reboot to flush the cache but cant do so with open unsaved work.
-
At BE1's request I have run some tests on my server. My server is an i3 540 runnging at 3ghz
C: drive is an Intel SSD and the OS drive, this is where HG612 stats normally lives
G: drive is a single Samsung HDD 5400rpm
F: drive is two of the above in Raid 1
I: drive is 6 x 3TB WD Reds in raid 6
Both raids are software, the raid 1 is on the motherboard, and the raid 6 is on a Highpoint controller.
My server was not rebooted between tests, I don't have the time to do that and others may be watching recordings stored on the server, I simply cut and pasted the folder between drives and then plotted the same data.
I plotted 60 days using the same data each time.
Drive Original program New Program
C: 391s 255
G: 239s 265s
F: 260s 215s
I: 298s 261s
1 day
C: 23s 26s
Now it's quite possible the log file was still in some cache due to the cut and paste so the above results could be flawed. Plotting 60 days on my office PC SSD using a log that hasn't been accessed since the last reboot only took 127 seconds. It's clearly the plotting that takes the time, reading the file is very quick when watching the DOS window.
Chrysalis, if your system is taking 4 minutes to plot one days worth of graphs then I think something is clearly wrong, have you run a disc benchmark against your raid 1 setup? Are you using Intel ports on your motherboard and the correct Intel drivers?
-
Ronski as I said before the amount I plot is completely irrelevant, whats relevant is the size of the log, as BE confirmed the entire log is read.
I could do tests now but they will be flawed as the file will be cached, I need to wait until I can reboot this system.
The hdd's are on my sata3 intel controller using intel's drivers, but intel now optimise their drivers for ssd's.
I have benchmarked the drives individually and they are running within expected performance, however what I havent done yet is benchmark the raid1 filesystem.
When the file is cached I can open it in under a second.
Also bear in mind due to the nature of how the log is written to, the chances are its very fragmented on my hdd and hence lots of random seeks to read it.
-
I have tried to flush without a reboot using a tool but it stays cached.
note if you make a copy of the file and then read the copy, its more likely to be a sequential read with faster performance and as such isnt a reliable test.
I will try to do this tommorow or maybe even later tonight.
renaming a file is ok, but not copy/move.
-
I think something is clearly wrong, have you run a disc benchmark against your raid 1 setup? Are you using Intel ports on your motherboard and the correct Intel drivers?
Disk corruption will show up as excessive reads on a disk it's trying to read data that's damaged or the segment were the file resides on the disk is damaged, i would run a chkdsk /f from the safemode boot options, as no matter how large the file is be it cached or not it should not be thrashing away to read data.
-
ok attached is 30 day run, old debug graphd exe and with file cached. I will follow this up doing same test after reboot, then will reboot a 2nd time and test with the new exe.
13/06/2015 10:57:55.306 - Start of [graphpd.exe] v 5.1.0.2
13/06/2015 10:57:55.306 - The program full path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\GraphPD.exe
13/06/2015 10:57:55.306 - The program directory path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\
13/06/2015 10:57:55.306 - DEBUG_COMMENTS = YES
13/06/2015 10:57:58.397 - You entered (or it was determined) a period of 30 d
13/06/2015 10:57:58.397 - Log file name wasn't entered - Using default modem_stats.log and subfolders
13/06/2015 10:57:58.398 - F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\modem_stats.log opened for reading
13/06/2015 10:57:58.400 - Log file & path not specified. Creating SUBFOLDER: F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1057-30Days
13/06/2015 10:57:58.400 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1057-30Days\GDATA$$
13/06/2015 10:57:59.501 - GDATA$$ created
13/06/2015 10:58:00.915 - Data from GDATA$$ starts at 08/05/2015 07:59 & ends at 07/06/2015 21:45
13/06/2015 10:58:00.916 - START of piped plotting with gnuplot
13/06/2015 10:58:34.291 - END of piped plotting with gnuplot
13/06/2015 10:58:34.431 - START of creating montage(s) with gnuplot
13/06/2015 10:58:35.543 - END of creating montage(s) with gnuplot
13/06/2015 10:58:35.548 - GDATA$$ deleted
13/06/2015 10:58:35.548 - End of graphpd. 43200 samples i.e. 30 Days of data should have been plotted
-
ok now 30 day graph with same log, after reboot.
13/06/2015 11:07:39.020 - Start of [graphpd.exe] v 5.1.0.2
13/06/2015 11:07:39.020 - The program full path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\GraphPD.exe
13/06/2015 11:07:39.020 - The program directory path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\
13/06/2015 11:07:39.020 - DEBUG_COMMENTS = YES
13/06/2015 11:07:43.824 - You entered (or it was determined) a period of 30 d
13/06/2015 11:07:43.824 - Log file name wasn't entered - Using default modem_stats.log and subfolders
13/06/2015 11:07:43.825 - F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\modem_stats.log opened for reading
13/06/2015 11:07:43.825 - Log file & path not specified. Creating SUBFOLDER: F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1107-30Days
13/06/2015 11:07:43.825 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1107-30Days\GDATA$$
13/06/2015 11:10:05.142 - GDATA$$ created
13/06/2015 11:10:06.557 - Data from GDATA$$ starts at 08/05/2015 07:59 & ends at 07/06/2015 21:45
13/06/2015 11:10:06.557 - START of piped plotting with gnuplot
13/06/2015 11:10:39.881 - END of piped plotting with gnuplot
13/06/2015 11:10:40.474 - START of creating montage(s) with gnuplot
13/06/2015 11:10:41.938 - END of creating montage(s) with gnuplot
13/06/2015 11:10:41.940 - GDATA$$ deleted
13/06/2015 11:10:41.940 - End of graphpd. 43200 samples i.e. 30 Days of data should have been plotted
-
here is using the new graphd, significantly quicker although was longer then I expected. ?I am about to do a 4th test, I will copy the file, rename the copy to be the current file, reboot and it should be quicker due to sequential reads, assuming my hdd's have enough sequential empty blocks left.
13/06/2015 11:17:18.792 - Start of [graphpd.exe] v 5.1.0.3
13/06/2015 11:17:18.792 - The program full path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\GraphPD.exe
13/06/2015 11:17:18.792 - The program directory path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\
13/06/2015 11:17:18.792 - DEBUG_COMMENTS = YES
13/06/2015 11:17:21.447 - You entered (or it was determined) a period of 30 d
13/06/2015 11:17:21.447 - Log file name wasn't entered - Using default modem_stats.log and subfolders
13/06/2015 11:17:21.447 - F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\modem_stats.log opened for reading
13/06/2015 11:17:21.447 - Log file & path not specified. Creating SUBFOLDER: F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1117-30Days
13/06/2015 11:17:21.447 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1117-30Days\GDATA$$
13/06/2015 11:18:21.090 - GDATA$$ created
13/06/2015 11:18:21.092 - Data from GDATA$$ starts at 08/05/2015 07:59 & ends at 07/06/2015 21:45
13/06/2015 11:18:21.093 - START of piped plotting with gnuplot
13/06/2015 11:18:54.262 - END of piped plotting with gnuplot
13/06/2015 11:18:54.575 - START of creating montage(s) with gnuplot
13/06/2015 11:18:55.788 - END of creating montage(s) with gnuplot
13/06/2015 11:18:55.792 - GDATA$$ deleted
13/06/2015 11:18:55.792 - End of graphpd. 43200 samples i.e. 30 Days of data should have been plotted[/b]
-
finally here is using a copy of the file, after a reboot, with new binary, can see its much quicker due to no or at least less fragmentation. Bear in mind this is not a realistic result, as these large logs several months long will be fragmented. After some household and personal chores I will do some diagnosis and tests on my raid filesystem to see if that has an issue as well.
13/06/2015 11:29:19.284 - Start of [graphpd.exe] v 5.1.0.3
13/06/2015 11:29:19.284 - The program full path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\GraphPD.exe
13/06/2015 11:29:19.284 - The program directory path is F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Scripts\
13/06/2015 11:29:19.284 - DEBUG_COMMENTS = YES
13/06/2015 11:29:21.052 - You entered (or it was determined) a period of 30 d
13/06/2015 11:29:21.052 - Log file name wasn't entered - Using default modem_stats.log and subfolders
13/06/2015 11:29:21.053 - F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\modem_stats.log opened for reading
13/06/2015 11:29:21.053 - Log file & path not specified. Creating SUBFOLDER: F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1129-30Days
13/06/2015 11:29:21.053 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1129-30Days\GDATA$$
13/06/2015 11:29:24.982 - GDATA$$ created
13/06/2015 11:29:24.983 - Data from GDATA$$ starts at 08/05/2015 07:59 & ends at 07/06/2015 21:45
13/06/2015 11:29:24.984 - START of piped plotting with gnuplot
13/06/2015 11:29:58.310 - END of piped plotting with gnuplot
13/06/2015 11:29:58.950 - START of creating montage(s) with gnuplot
13/06/2015 11:30:00.207 - END of creating montage(s) with gnuplot
13/06/2015 11:30:00.210 - GDATA$$ deleted
13/06/2015 11:30:00.210 - End of graphpd. 43200 samples i.e. 30 Days of data should have been plotted
-
Before I do my offline stuff, here is some notes.
1 - once it is cached it will stay cached for a while, how long is an unknown as it depends on amount of free ram in the system and pressure on the system to flush it, in my case it does stay cached over night but I know it doesnt stay cached over a few days.
2 - Based on #1 if one is running daily ongoing graphs its possible that serves to refresh the cache and hide the problem.
3 - I do not run scheduled on going graphs, only snapshot graphs. Which means generally when I do a manual run, it is not cached.
-
Ronski as I said before the amount I plot is completely irrelevant, whats relevant is the size of the log, as BE confirmed the entire log is read.
The amount being plotted did have & still does have some relevance, both in time taken to read the ongoing log & the time taken to create the temporary file for the actual plotting.
The temporary file is then read row by row a number of times to obtain the data for each individual graph.
Plotting the data for 1 day uses 1440 samples (rows of data).
Plotting 30 days uses 43200 samples & so on.
A lot more graphs are created for G.INP active connections which will also have added more time for the whole process compared to the time taken for non-G.INP active connections.
-
here is using the new graphd, significantly quicker although was longer then I expected.
13/06/2015 11:17:21.447 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1117-30Days\GDATA$$
13/06/2015 11:18:21.090 - GDATA$$ created
Was that using the log cached or non-cached?
-
I Just noticed when running the GraphPD the DOS text shows up as G.INP enabled which is not the case since 13th of may ???
Its ok it much see the G.INP stats when this line was disabled a month ago :blush:
-
only the first was cached all other 3 tests were uncached, and yeah I did notice 30 days is slower than 1 days, but the delay is after the log is read for that,
-
finally here is using a copy of the file, after a reboot, with new binary, can see its much quicker due to no or at least less fragmentation.
13/06/2015 11:29:21.053 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1129-30Days\GDATA$$
13/06/2015 11:29:24.982 - GDATA$$ created
/quote]
So that only took around 4 seconds to read the log & create the temporary file.
-
yeah the unfragmented file was really quick.
-
only the first was cached all other 3 tests were uncached, and yeah I did notice 30 days is slower than 1 days, but the delay is after the log is read for that,
Yes. But didn't you say earlier that it takes 4 minutes to just read the non-cached log when the program is getting ready to plot just 1 day's data?
-
I might schedule this every week on the log file.
http://techbrew.net/articles/200808/defrag-a-single-file-or-set-of-files-in-windows/
http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx
-
only the first was cached all other 3 tests were uncached, and yeah I did notice 30 days is slower than 1 days, but the delay is after the log is read for that,
Yes. But didn't you say earlier that it takes 4 minutes to just read the non-cached log when the program is getting ready to plot just 1 day's data?
The 4 minutes was me guessing, it actually was nearer to 3 minutes.
But the slow file is fragmented, so the read speed is heavily compromised as its random reads vs sequential reads. When I Was making the copy the hdd was only able to read the file at about 5 meg/sec.
-
I Just noticed when running the GraphPD the DOS text shows up as G.INP enabled which is not the case since 13th of may ???
Its ok it much see the G.INP stats when this line was enabled a month ago :blush:
Which version of graphpd.exe did you use for that?
Maybe I should word the message differently.
With the old version, reading the whole log will report whether G.INP was active regardless of when it was disabled.
With the new version, it should only report it if the flag is present somewhere in the data for the period being plotted.
Please let me know if it still reports it as active with the new version if plotting say just a few days worth of data. i.e. from some date more recent than 13th May.
-
I might schedule this every week on the log file.
http://techbrew.net/articles/200808/defrag-a-single-file-or-set-of-files-in-windows/
http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx
That might help, especially if it is done immediately or pretty recently before plotting the graphs.
Please let us know what difference that makes to the time required to read the log & create the temporary file.
As the temporary file is written more or less sequentially, it probably won't make much difference to the time taken by gnuplot to create the graphs.
I'm afraid there's nothing I can do to speed up the actual plotting though as gnuplot is a 3rd party program.
-
are you going to keep the new improved log reading graphd now?
and I will provide future feedback on the results of contig yes, its a shame its not included in all windows installs then you could have automated it but instead it needs to be manually downloaded by the end user.
-
are you going to keep the new improved log reading graphd now?
More than likely - subject to a bit more testing/monitoring etc.
and I will provide future feedback on the results of contig yes, its a shame its not included in all windows installs then you could have automated it but instead it needs to be manually downloaded by the end user.
The licence doesn't allow me to include it in HG612 Modem Stats either.
Just tried it on my own modem_stats.log & this was the output:-
contig -v "C:\HG612_Modem_Stats\Ongoing_Stats\modem_stats.log"
Contig v1.7 - Makes files contiguous
Copyright (C) 1998-2012 Mark Russinovich
Sysinternals - www.sysinternals.com
------------------------
Processing C:\HG612_Modem_Stats\Ongoing_Stats\modem_stats.log:
Scanning file...
Scanning disk...
Scanning Disk: 0xc0000022
No files fragmented.
So my log wasn't fragmented at all.
-
I Just noticed when running the GraphPD the DOS text shows up as G.INP enabled which is not the case since 13th of may ???
Its ok it much see the G.INP stats when this line was disabled a month ago :blush:
Try the attached version 5.1.0.4.
Plotting data for the last few days should now show it as not active on your connection.
However, if you plot from the time it was active (or even from before then), it should show that it was active "at some point".
-
I'm wondering if Chrysalis is perhaps onto something here, I vaguely remember a long long time ago that my scheduled graphs took a long time to be created, and I'm sure is was around the GDATA bit that the delay was happening. Unfortunately I can't remember when it was, but I did email BE1 about it, but I can't find the relevant email either. It was certainly prior to me splitting my modem log into separate years and also certainly when the server was using a spinning HDD for the OS and Modem Stats. I think the only reason I noticed was that HG612_Stats was having problems running as one instance was waiting for the graphs to be plotted when another started, and I think the programs were altered so HG612_Stats could close, and the graphing could get on and do it's thing.
Unfortunately due to using a text file for the stats there's not very much that could be done to speed things up, and I think what BE1 has come up with is probably one of the best solutions that I'm aware of - remember that neither of us are programmers via profession and have had to learn to code to write our respective programs.
Regarding defragging the file, this is very simple, no additional programs are required, I'm pretty sure that making a copy of the original, then deleting the original and renaming the copy would have the desired affect. But we don't want to be doing this on systems running on SSD's
-
Processing F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\mod
F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\modem_stats.lo
g.old is in 65110 fragments
Summary:
Number of files processed : 1
Average fragmentation : 65110 frags/file
haha
I will defrag it and then retest it but should be fast afterwards.
and yes copy + delete + rename will have the same result. Which is what I did in my test 4.
-
I'm wondering if Chrysalis is perhaps onto something here, I vaguely remember a long long time ago that my scheduled graphs took a long time to be created, and I'm sure is was around the GDATA bit that the delay was happening. Unfortunately I can't remember when it was, but I did email BE1 about it, but I can't find the relevant email either. It was certainly prior to me splitting my modem log into separate years and also certainly when the server was using a spinning HDD for the OS and Modem Stats. I think the only reason I noticed was that HG612_Stats was having problems running as one instance was waiting for the graphs to be plotted when another started, and I think the programs were altered so HG612_Stats could close, and the graphing could get on and do it's thing.
I can't track down your email either.
However, some of the changes that have been made, probably since then, are:-
* HG612_stats.exe now runs at 'normal' priority rather than 'below normal' priority.
* Other programs triggered via HG612_stats.exe can now run & complete normally without holding up HG612_stats.exe.
-
Summary:
Number of files processed : 1
Average fragmentation : 65110 frags/file
haha
I will defrag it and then retest it but should be fast afterwards.
I'm sure (I hope) that defragging the log file will certainly assist in speeding things up at your end.
-
Added a weekly schedule and then hit thr run button no the schedule it worked, but the 2 issues were was the popup cmd window which if I am doing something would be an issue, and that it still took about 2 minutes on a log which I think has only been in use for a week or two. So I changed the schedule to daily, eve on this new log there was 1600 fragments, so I am amazed BE your log is unfragmented.
I noticed in task scheduler is a hidden button, does this stop the task going into the foreground? There must be a way as the stats task that runs every minute doesnt go in the foreground.
-
The normal scheduled task runs the program named HG612_Run.exe every minute.
This program simply makes HG612_stas.exe run hidden every minute.
You could create a batch file named something like Run_Contig.BAT & save it in the same folder as Contig.exe
The contents of the batch file would include whatever command you wish Contig.exe to use.
e.g:-
Contig.exe -a -v "C:\HG612 Modem Stats with Spaces\Ongoing Stats\TESTS - VDSL2\G.INP Enabled\modem_stats.log" >> contig_output.txt
echo ******************************************************** >> contig_output.txt
echo. >> contig_output.txt
That batch file would append the normal screen display when Contig.exe runs to a file named contig_output.TXT.
e.g:-
Contig v1.7 - Makes files contiguous
Copyright (C) 1998-2012 Mark Russinovich
Sysinternals - www.sysinternals.com
------------------------
Processing C:\HG612 Modem Stats with Spaces\Ongoing Stats\TESTS - VDSL2\G.INP Enabled\modem_stats.log:
Scanning file...
[Cluster] Runlength
[0] 39536
File size: 161937571 bytes
C:\HG612 Modem Stats with Spaces\Ongoing Stats\TESTS - VDSL2\G.INP Enabled\modem_stats.log is in 1 fragment
------------------------
Summary:
Number of files processed : 1
Average fragmentation : 1 frags/file
********************************************************
Contig v1.7 - Makes files contiguous
Copyright (C) 1998-2012 Mark Russinovich
Sysinternals - www.sysinternals.com
------------------------
Processing C:\HG612 Modem Stats with Spaces\Ongoing Stats\TESTS - VDSL2\G.INP Enabled\modem_stats.log:
Scanning file...
[Cluster] Runlength
[0] 39536
File size: 161937571 bytes
C:\HG612 Modem Stats with Spaces\Ongoing Stats\TESTS - VDSL2\G.INP Enabled\modem_stats.log is in 1 fragment
------------------------
Summary:
Number of files processed : 1
Average fragmentation : 1 frags/file
********************************************************
To run the program silently, you could create a *.VBS script named something like Run_Contig.VBS
The content of that VBS script (also saved in the same folder as Contig.exe) should be:-
Set WshShell = CreateObject("WScript.Shell")
WshShell.Run chr(34) & "Run_Contig.BAT" & Chr(34), 0
Set WshShell = Nothing
Your Scheduled task should be set up to run the VBS script & then it should all work hidden.
This certainly works on Windows 7 & I initially used the VBS script method before Ronski wrote HG612_Run.exe
Try running the VBS script directly to see if it runs the batch file silently.
-
Try the attached version 5.1.0.4.
Cheers BE1 downloaded and extracted to script folder, Graphpd 5.1.0.4 took 24 seconds to complete a 4 day graph and 30 days took 2mins 56 seconds but have Windows Media Center running and watching country file in HD.
-
Did it correctly state whether G.INP was active or not?
It probably still (incorrectly) says it is v 5.1.0.3 in the event log
-
I got it going in background now, have to set it to run regardless of password or not, I found out how by copying what ronski did :)
For the curious, a 6 hour or so old ongoing stats log on my hdd had 78 fragments.
by the way I set it to start 20 secs after the minute, it doesnt show up in the task list as that but hopefully thats when it starts so no conflict with modem stats.