Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 2 3 [4] 5 6

Author Topic: DSLstats after the demise of MDWS  (Read 19815 times)

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43432
  • Penguins CAN fly
    • DSLstats
Re: DSLstats after the demise of MDWS
« Reply #45 on: March 08, 2018, 03:15:27 PM »

   Might it be possible to extend the time per page on DSLstats graphs to more than 24 hours?   Up to 7 days or even a month would give more MDWS like visibility of things if you don't look at things every day.  If a week is possible then an option to save snapshots every 7 days would be nice.  If a month is possible I think the monthly option is already there.   This would also make looking a things back in time via the saved snapshots a much easier task.

   I have no idea how easy or hard that might be.    I doubt that it would matter if for graphs longer than 24 hours days the sample times needed to have a bigger minimum value. 

I can certainly do this - the current limits are fairly arbitrary anyway. But I'll need to make sure that increased values don't have any unexpected side effects, so this will have to join the todo list.
Logged
  Eric

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43432
  • Penguins CAN fly
    • DSLstats
Re: DSLstats after the demise of MDWS
« Reply #46 on: March 08, 2018, 03:51:27 PM »

The current state of the development

I've stripped out the MDWS upload capability and changed the handling of the MDWS data so it's saved locally. At present the main data is saved after every sample and the zipped per-tone data is saved once an hour. The files have all got a time stamp in their filenames and they're saved in daily folders with names like 2018-03-08 (yyyy-mm-dd).

With the sample time set at one minute, the amount of data stored per day is about 900 kB for a VDSL2 profile 17a connection. In a full day there will be 1440 files of main stats and 96 files of per-tone data in the daily folder. I'm assuming that the number of files in each folder isn't a problem on any of the supported operating systems, but could someone please let me know if I'm wrong?

The next stage is to add some user control of the frequency with which the data is saved, and I would be interested in views on that.
Logged
  Eric

Westie

  • Kitizen
  • ****
  • Posts: 1596
Re: DSLstats after the demise of MDWS
« Reply #47 on: March 08, 2018, 04:29:25 PM »

At the moment (v6.1.3) there is the option to save snapshots in daily folders or category folders.

I find the latter more useful, as it is then easier to compare how a given parameter has changed over time, because all the time-stamped snapshots for a particular category are collated together. Since I am only saving 24-hour snapshots once per day I find it a great way to get an overview.

I can understand that data saved after each sample might be best stored in daily folders, but if the ability to save daily snapshots by category is retained, I think it's easier to identify which day & category should be examined at raw data level.

Just my personal opinion...
Logged

ktz392837

  • Reg Member
  • ***
  • Posts: 559
Re: DSLstats after the demise of MDWS
« Reply #48 on: March 08, 2018, 04:53:36 PM »

Roseway I would try to keep things simple at first.

Keep the new raw data snapshots separate from the current snapshots (but use /raw/yyyymmdd and /graph/yyyymmdd folders) so retention days can be configured separately.  E.g
 I may have 28d for the current graph like snapshots but I may want 365d of the new minute by minute raw data.

Personally the raw data is such a small amount of disk space per day I would just always do it per minute and the size can be set by the number of days kept.

Anyway I am happy just to have something after mdws so these are just my personal preferences I am sure I will be happy with whatever is decided.

Cheers and thanks again for spending time enhancing your code to partially fill the gap.
« Last Edit: March 08, 2018, 05:01:21 PM by ktz392837 »
Logged

Ronski

  • Moderator
  • Kitizen
  • *
  • Posts: 4300
Re: DSLstats after the demise of MDWS
« Reply #49 on: March 08, 2018, 08:16:37 PM »

I'm assuming that the number of files in each folder isn't a problem on any of the supported operating systems, but could someone please let me know if I'm wrong?

Not a problem, but on some systems 900kB of data in 1440 files may well take up far more space than 900kB, I believe it depends on the sector size of the disc.
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

jelv

  • Helpful
  • Kitizen
  • *
  • Posts: 2054
Re: DSLstats after the demise of MDWS
« Reply #50 on: March 08, 2018, 09:47:10 PM »

The Windows default cluster size on NTFS is 4K so 1440 files will take a minimum of 5760K. That works out to over 2GB for a year's worth of files!
Logged
Broadband and Line rental: Zen Unlimited Fibre 2, Mobile: Vodaphone
Router: Fritz!Box 7530

g3uiss

  • Kitizen
  • ****
  • Posts: 1151
  • You never too old to learn but soon I may be
    • Midas Solutions
Re: DSLstats after the demise of MDWS
« Reply #51 on: March 08, 2018, 10:40:31 PM »

Thank you.    I'm currently trying to wade though the minefield myself to aid better understanding, but there is one area I'm still hazy about.   There's a couple of things I do need to check first, but if you don't mind I may at some time pick your brains if I get stuck.

Fine. Just let me know what clarifications you need.
Logged
Cerebus FTTP 500/70 Draytec 2927 VOXI 4G fallback.

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #52 on: March 09, 2018, 10:50:59 AM »

Yes I was looking at google charts last night. In fact I was actually looking at their Line charts here.

But its not quite so simple as having the charts, you'd also need something which would accept the data from DSLstats, update the data then send it to be able to graph it to the chart.
Unfortunately this isn't going to be a 2 minute job just because there is such a thing as google charts  :(
Yes but as what we want it a way of generating charts and Google Charts is already being used to do this, then Google charts seems the most appropriate way of producing charts for this replacement project.

As I understand it, DSLstats extracts the data on a regular interval and stores it on my PC. It then uploads the data to the MDWS website.

MDWS now accumulates the data and stores it in a way it can be easily charted. This involves asking Google Charts to chart this specified data and output a Chart which is then displayed to the user.

Within this process the data items have already been named by DSLstats.
Logged

d2d4j

  • Kitizen
  • ****
  • Posts: 1103
Re: DSLstats after the demise of MDWS
« Reply #53 on: March 09, 2018, 11:08:27 AM »

Hi

@dray - sorry I thought the naming was completed by tony mdws

I could be wrong sorry but it’s easy to check by looking at the files that are uploaded

Many thanks

John
Logged

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #54 on: March 09, 2018, 11:15:02 AM »

Yes it would be nice to see a list
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #55 on: March 09, 2018, 11:15:45 AM »

Youre correct John, its just raw data.
There's  a sample of raw data in my post here.

There's a lot of code needed to get from that to sending the info to google charts.  As I said in my earlier post there's a lot of other areas that need to be thought about ie


It's currently sent as space delimited file, so first step would be to break down into an array for each item,
You'd then need to identify what each stat in the array is and name it, so you can identify which piece of data you want for the particular graph you want to display....  and that's just the easy bit.

There's lots of other areas you need to think about.
 - How is the data automatically going to be uploaded to your webserver
 - How are you going to keep a permanent record of all the stats so that each new incoming data is added to the record
 - Code for sending an array to the graph...  there's some code here which may help
 - You'd need .html templates to output the graph to.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43432
  • Penguins CAN fly
    • DSLstats
Re: DSLstats after the demise of MDWS
« Reply #56 on: March 09, 2018, 11:17:32 AM »

Quote
Within this process the data items have already been named by DSLstats.

The main stats as uploaded to MDWS are simply a space delimited list of values. They're not in any logical order for historical reasons, so I'll be providing a conversion chart to relate element numbers to what they represent.
Logged
  Eric

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #57 on: March 09, 2018, 11:21:15 AM »

Sounds good :)
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #58 on: March 09, 2018, 11:44:08 AM »

@dray..  but that will only be a reference for who ever does write the code.    The data is always going to have to be sent and stored as either space delimited or csv files otherwise they become very large.
You still then have to approach all the other steps I outlined.  :(
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #59 on: March 09, 2018, 01:42:16 PM »

Hi Kitz,

Just to echo what you asked earlier - can anyone provide a list of steps that can be used by MDWS to create a chart, to keep it simple lets say the SRNM dB chart?
Logged
Pages: 1 2 3 [4] 5 6
 

anything