Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 2 [3] 4

Author Topic: hg612 modem stats request reference log rotation  (Read 11974 times)

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: hg612 modem stats request reference log rotation
« Reply #30 on: June 13, 2015, 08:06:30 PM »

only the first was cached all other 3 tests were uncached, and yeah I did notice 30 days is slower than 1 days, but the delay is after the log is read for that,
Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #31 on: June 13, 2015, 08:07:27 PM »

finally here is using a copy of the file, after a reboot, with new binary, can see its much quicker due to no or at least less fragmentation. 

13/06/2015 11:29:21.053 - Creating F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\Ongoing_Stats_20150613-1129-30Days\GDATA$$
13/06/2015 11:29:24.982 - GDATA$$ created

/quote]

So that only took around 4 seconds to read the log & create the temporary file.

Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: hg612 modem stats request reference log rotation
« Reply #32 on: June 13, 2015, 08:09:38 PM »

yeah the unfragmented file was really quick.
Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #33 on: June 13, 2015, 08:10:25 PM »

only the first was cached all other 3 tests were uncached, and yeah I did notice 30 days is slower than 1 days, but the delay is after the log is read for that,

Yes. But didn't you say earlier that it takes 4 minutes to just read the non-cached log when the program is getting ready to plot just 1 day's data?

 
Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: hg612 modem stats request reference log rotation
« Reply #35 on: June 13, 2015, 08:15:43 PM »

only the first was cached all other 3 tests were uncached, and yeah I did notice 30 days is slower than 1 days, but the delay is after the log is read for that,

Yes. But didn't you say earlier that it takes 4 minutes to just read the non-cached log when the program is getting ready to plot just 1 day's data?

 


The 4 minutes was me guessing, it actually was nearer to 3 minutes.

But the slow file is fragmented, so the read speed is heavily compromised as its random reads vs sequential reads.  When I Was making the copy the hdd was only able to read the file at about 5 meg/sec.
Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #36 on: June 13, 2015, 08:16:30 PM »

I Just noticed when running the GraphPD the DOS text shows up as G.INP enabled which is not the case since 13th of may  ???

Its ok it much see the G.INP stats when this line was enabled a month ago  :blush:

Which version of graphpd.exe did you use for that?



Maybe I should word the message differently.

With the old version, reading the whole log will report whether G.INP was active regardless of when it was disabled.

With the new version, it should only report it if the flag is present somewhere in the data for the period being plotted.


Please let me know if it still reports it as active with the new version if plotting say just a few days worth of data. i.e. from some date more recent than 13th May.


Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #37 on: June 13, 2015, 08:25:39 PM »

I might schedule this every week on the log file.

http://techbrew.net/articles/200808/defrag-a-single-file-or-set-of-files-in-windows/
http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx


That might help, especially if it is done immediately or pretty recently before plotting the graphs.

Please let us know what difference that makes to the time required to read the log & create the temporary file.


As the temporary file is written more or less sequentially, it probably won't make much difference to the time taken by gnuplot to create the graphs.

I'm afraid there's nothing I can do to speed up the actual plotting though as gnuplot is a 3rd party program.



Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: hg612 modem stats request reference log rotation
« Reply #38 on: June 13, 2015, 09:01:37 PM »

are you going to keep the new improved log reading graphd now? 

and I will provide future feedback on the results of contig yes, its a shame its not included in all windows installs then you could have automated it but instead it needs to be manually downloaded by the end user.
Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #39 on: June 13, 2015, 10:01:50 PM »

are you going to keep the new improved log reading graphd now? 


More than likely - subject to a bit more testing/monitoring etc.


Quote
and I will provide future feedback on the results of contig yes, its a shame its not included in all windows installs then you could have automated it but instead it needs to be manually downloaded by the end user.


The licence doesn't allow me to include it in HG612 Modem Stats either.

Just tried it on my own modem_stats.log & this was the output:-

contig  -v "C:\HG612_Modem_Stats\Ongoing_Stats\modem_stats.log"

Contig v1.7 - Makes files contiguous
Copyright (C) 1998-2012 Mark Russinovich
Sysinternals - www.sysinternals.com

------------------------
Processing C:\HG612_Modem_Stats\Ongoing_Stats\modem_stats.log:
Scanning file...
Scanning disk...

Scanning Disk: 0xc0000022

No files fragmented.



So my log wasn't fragmented at all.



Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #40 on: June 13, 2015, 11:26:39 PM »

I Just noticed when running the GraphPD the DOS text shows up as G.INP enabled which is not the case since 13th of may  ???

Its ok it much see the G.INP stats when this line was disabled a month ago  :blush:


Try the attached version 5.1.0.4.

Plotting data for the last few days should now show it as not active on your connection.
However, if you plot from the time it was active (or even from before then), it should show that it was active "at some point".

« Last Edit: June 13, 2015, 11:30:30 PM by Bald_Eagle1 »
Logged

Ronski

  • Moderator
  • Kitizen
  • *
  • Posts: 4300
Re: hg612 modem stats request reference log rotation
« Reply #41 on: June 14, 2015, 09:40:42 AM »

I'm wondering if Chrysalis is perhaps onto something here, I vaguely remember a long long time ago that my scheduled graphs took a long time to be created, and I'm sure is was around the GDATA bit that the delay was happening. Unfortunately I can't remember when it was, but I did email BE1 about it, but I can't find the relevant email either. It was certainly prior to me splitting my modem log into separate years and also certainly when the server was using a spinning HDD for the OS and Modem Stats. I think the only reason I noticed was that HG612_Stats was having problems running as one instance was waiting for the graphs to be plotted when another started, and I think the programs were altered so HG612_Stats could close, and the graphing could get on and do it's thing.

Unfortunately due to using a text file for the stats there's not very much that could be done to speed things up, and I think what BE1 has come up with is probably one of the best solutions that I'm aware of - remember that neither of us are programmers via profession and have had to learn to code to write our respective programs.

Regarding defragging the file, this is very simple, no additional programs are required, I'm pretty sure that making a copy of the original, then deleting the original and renaming the copy would have the desired affect. But we don't want to be doing this on systems running on SSD's
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 7382
  • VM Gig1 - AAISP L2TP
Re: hg612 modem stats request reference log rotation
« Reply #42 on: June 14, 2015, 11:41:42 AM »

Processing F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\mod
F:\Drivers\HG612\HG612_Modem_Stats_Programs-CURRENT\Ongoing_Stats\modem_stats.lo
g.old is in 65110 fragments

Summary:
     Number of files processed   : 1
     Average fragmentation       : 65110 frags/file

haha

I will defrag it and then retest it but should be fast afterwards.

and yes copy + delete + rename will have the same result. Which is what I did in my test 4.
« Last Edit: June 14, 2015, 11:44:14 AM by Chrysalis »
Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #43 on: June 14, 2015, 12:04:04 PM »

I'm wondering if Chrysalis is perhaps onto something here, I vaguely remember a long long time ago that my scheduled graphs took a long time to be created, and I'm sure is was around the GDATA bit that the delay was happening. Unfortunately I can't remember when it was, but I did email BE1 about it, but I can't find the relevant email either. It was certainly prior to me splitting my modem log into separate years and also certainly when the server was using a spinning HDD for the OS and Modem Stats. I think the only reason I noticed was that HG612_Stats was having problems running as one instance was waiting for the graphs to be plotted when another started, and I think the programs were altered so HG612_Stats could close, and the graphing could get on and do it's thing.


I can't track down your email either.

However, some of the changes that have been made, probably since then, are:-

*    HG612_stats.exe now runs at 'normal' priority rather than 'below normal' priority.
*    Other programs triggered via HG612_stats.exe can now run & complete normally without holding up HG612_stats.exe.


Logged

Bald_Eagle1

  • Helpful
  • Kitizen
  • *
  • Posts: 2721
Re: hg612 modem stats request reference log rotation
« Reply #44 on: June 14, 2015, 12:06:22 PM »


Summary:
     Number of files processed   : 1
     Average fragmentation       : 65110 frags/file

haha

I will defrag it and then retest it but should be fast afterwards.



I'm sure (I hope) that defragging the log file will certainly assist in speeding things up at your end.

Logged
Pages: 1 2 [3] 4