Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 ... 3 4 [5] 6

Author Topic: DSLstats after the demise of MDWS  (Read 19895 times)

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #60 on: March 09, 2018, 02:09:04 PM »

I know the steps although it may take just a while to type them all out if you mean the actual processes and functions required. (and no I dont mean the code)
I have covered the basic steps already.

The hard part and time consuming part is the bit in the middle between having the raw data and then being able to use that data to output to the graph.  As I said earlier outputting the data to the graph is then easy.

 
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #61 on: March 09, 2018, 02:26:21 PM »

Definitely not the code. And I'm not trying to find work for you to do neither.

Currently we now have the data stored like this: (although I haven't seen it)

I've stripped out the MDWS upload capability and changed the handling of the MDWS data so it's saved locally. At present the main data is saved after every sample and the zipped per-tone data is saved once an hour. The files have all got a time stamp in their filenames and they're saved in daily folders with names like 2018-03-08 (yyyy-mm-dd).

Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #62 on: March 09, 2018, 02:34:13 PM »

Currently we now have the data stored like this: (although I haven't seen it)

I gave a sample of what one row of data contains a few days ago..  and linked to it again this morning :(

Admittedly I got my sample from HG612-stats because that is what I was using to upload to MDWS , but the data will be practically the same whether its coming from DSLstats or HG612stats.

This is just basic line stats.   There's a lot more to cover things like hlog etc.


Code: [Select]
25/07/2016 20:13 79987 20000 7.0 12.5 5.9 5.9 83805 30404 PTM 850 18 0 390 731 0 218562 0 0 0 0 60.712 0 390 0 132124761 76362 0 850 18 0 0 3400835 0 0 0 0 1 1 122 62 12 0 0 0 0 0 0 0 0.7 18.5 28.1 0.0 11.0 24.5 38.5 0.6 18.3 28.1 0.0 10.7 24.4 38.5 14.2 13.1 12.3 0.0 7.0 7.0 7.0 -6.0 -25.6 5.6 0.0 12.0 7.4 7.4 Showtime 0 0 L0 17a ON ON No Showtime 0.0 0.0 239 236 1 1 23 5 0 16 0.0955 0.3771 20104 5410 30 0 30 0 3547761395 0 29704784 0 0 0 0 59 59 0.00 0.00 1.65 6.15 0.00 0.00 116.09 202.87 588 12 VDSL2 0 0 0 0 0.00 0.00 0.00 0.00 0 0 1 0 0 0 7 0 0 0 0 0 0 0 NO_G.INP 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0 0 0.00 0.00 0.00 0.00 0.00 0.00
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #63 on: March 09, 2018, 02:39:19 PM »

Dunno how I missed that. Sorry  :-[
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #64 on: March 09, 2018, 02:50:42 PM »

can anyone provide a list of steps that can be used by MDWS to create a chart, to keep it simple lets say the SRNM dB chart?


  • First you'd need to decide how you are going to store the historic data.   There's 2 ways of approaching this, you can use SQL Tables or store txt files.


There's pro's and cons' for both methods. Setting up the (SQL) database in itself will take some time to create all the tables you need.
What a approach you take could also depend on whether you are using a webserver running on your local PC OR if you are going to be putting the scripts on your website. 
If you have a web-server running on your local PC it gives you a lot more flexibility, but google graphs will I believe only run live because they are parsed by your browser.   
I can understand why Tony needed to run this locally using his own equipment because in some respects it makes some things a bit easier to control. 


-----------------------   

Then move on to the next steps, these are just off the top of my head and there will probably be a lot more functions to code, but I think I've covered the main ones:-


  • Find a way of uploading the DSLstats file to your webserver.   Easiest way would probably be to create some sort of script that would automatically FTP this info to your server.
  • Create a script that will take the raw data file from MDWS and automatically be able to add the raw data to existing database tables.  OR create a script that will append the latest raw data to an existing history file.   
  • Create a program which will be able to pull all the SNRM info from either the database or text files.   This is where storing info as a database may be easier as it saves processing time over a script which loops through a large text file.
  • Create a query which will allow different time values for SNRM to display and add this to your code.   This has to be flexible so you will need to add various parameters.   Because you are getting input from a user then the code has to be secure and make sure that it is protected against SQL injection for eg.   This means more code for validation. 
  • Create the html template for the google graph to be able to display it as a webpage.
  • Add the google chart code to the webpage.


Now create numerous other scripts which will do the same for atten, sync speed etc.   
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #65 on: March 09, 2018, 03:01:54 PM »

Thanks for that. I can't see any advantage in using SQL so text files will be used. I'll currently ignore that last point to remove the pressure point ;)
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #66 on: March 09, 2018, 03:57:15 PM »

I can.   If they are stored in an SQL database then it will be much faster and easier to when it comes to producing the graphs.

Over time the text files will become large and have a lot of data to sort through to just find the SNRM values.  Look at that one row of data I posted and you will see how many values are in the text file...  but you are only searching for one figure out of that whole row.    If stored as text then your program will have to loop through 100's of 1000's of data to find the one figure in each row that it is looking for.

If you create a database that stores all the SNR values in one table, then you can do a direct query on that table and ask it to pull all the data between 'x & y' date and then send those figures to the google chart code.   More code at the beginning to set up all the tables and to create a script which will add the incoming data from DSLstats to your database.   But the graph execution and webpage load will be a heck of a lot quicker.

I don't know if you've ever used HG612stats GUI to produce graphs, but it can take a few mins to generate the graphs, that's because all the data is stored as text files.   Even on a fast PC it can still take a while..  it wouldn't be the best approach if you're running it from a remote web-site.

You need to sit down and do some serious thinking how you are going to be outputting the data... and which method is going to be best depending on how you are going to output that data.  What may work best on a machine running a local web-server, or something to just generate static graphs is going to have different requirements to running it on a remote website.  It also depends on how often you think you are going to view them. 

---

There is really an awful lot of thought that has gone behind what Tony did, its not something that will have taken him a couple of days.   Unless you have to do this sort of thing, then people don't realise how much work goes into being able to present the data.     

To give an example...  my broadband checker gets all the data from the BTw database I do a simple query and it returns all the raw data in xml format.    I immediately have the info that is needed.   
The bit that took me weeks/months to code is being able to break down the raw code and then output the data so that its readable to humans and looks pretty.   With MDWS, google graphs is doing the pretty bit, but there is still an awful lot of code in the middle that will be needed to make sure that the graphs get the right info they need. :(
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

ktz392837

  • Reg Member
  • ***
  • Posts: 559
Re: DSLstats after the demise of MDWS
« Reply #67 on: March 09, 2018, 05:14:34 PM »

I guess everyone is paying particular attention of what bits they use the most in MDWS.

I know roseway has a list over 10 miles long now but from things that haven't been mentioned yet, the traffic light overview is something I look at frequently as it is a quick way of checking status of the line.  I also hope the ES graph gets a rolling 24h like most of the other grqphs.

I am trying to remember to use dslstats web server instead of mdws ready for Monday :(
Logged

Dray

  • Kitizen
  • ****
  • Posts: 2361
Re: DSLstats after the demise of MDWS
« Reply #68 on: March 09, 2018, 05:43:41 PM »

I should just point out that I am not going to be developing a replacement for MDWS.

My ability ran out while getting the  DSLstats webserver to produce a useful page of graphs
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #69 on: March 10, 2018, 12:28:29 AM »

I also hope the ES graph gets a rolling 24h like most of the other grqphs.

I think that is one of the things that a lot of us are going to miss the most because of the way DLM works. 
Err/Secs and to a lesser extent SNRM are about the only thing I really look at on my line.   

Eric has got a lot to do as it is and I'm not sure how hard it would be to add this.   I am a bit concerned that the requests are piling up for him.  :-\
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: DSLstats after the demise of MDWS
« Reply #70 on: March 10, 2018, 12:37:39 AM »

I should just point out that I am not going to be developing a replacement for MDWS.

My ability ran out while getting the  DSLstats webserver to produce a useful page of graphs

I'm bowing out of this convo now as I feel its just distracting this thread from what Eric can do to do with the DSLstats data output. 

Anyone who has the ability will likely know what is going to be involved and I've already put far more time into explaining things than was my original intention as I covered the basic steps a few days ago.  Just to re-iterate, its your browser that renders the google graphs and its easy to put their code into a html web page.  The hard part is getting the data, storing the data, then being able to call it when you want to be able to pass it to the graphs.   
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43467
  • Penguins CAN fly
    • DSLstats
Re: DSLstats after the demise of MDWS
« Reply #71 on: March 10, 2018, 07:23:47 AM »

Quote
I also hope the ES graph gets a rolling 24h like most of the other grqphs.

What MDWS does is put in rolling graphical form the data which DSLstats presents as a daily bar chart. It's the same data, and DSLstats retains 100 days worth. All of this data can be viewed by clicking on the "... days ago" control.

If you really want a per-minute display of ES I could present it in the same form as CRC, FEC, etc. But with the jobs piling up, it's not going to be quick.
Logged
  Eric

ktz392837

  • Reg Member
  • ***
  • Posts: 559
Re: DSLstats after the demise of MDWS
« Reply #72 on: March 10, 2018, 09:14:11 AM »

If you really want a per-minute display of ES I could present it in the same form as CRC, FEC, etc. But with the jobs piling up, it's not going to be quick.

Thanks for replying roseway.

My reasoning was when you are looking at the stats via the web browser if you are looking at the start of the day there is not much information as it resets at midnight so if you look at 0600hr say before you go to work there is only hours 00, 01, 02, 03, 04, 05 visible.  If you had an ES spike at 23 or 18 you cant see it at least via the browser.

I just thought it would be useful if people posted how they used MDWS and then you can decide what is worth your very limited time. 

Thanks for dslstats I get you can't do everything I appreciate you are trying to enhance to fill the gap as much as feasible possible with the time you have available.
Logged

Westie

  • Kitizen
  • ****
  • Posts: 1596
Re: DSLstats after the demise of MDWS
« Reply #73 on: March 10, 2018, 12:27:56 PM »

In DSLstats, if you look at the ES/hour for the previous day, you will see any spikes that occurred before midnight. As I understand it the DLM resets at the end of the day anyway, so a spike the previous evening will either already have had an effect, or won't affect the current day anyway.

With the demise of MDWS, I think it's important for each of us to look for alternative ways to get the information we would like from existing resources as far as possible, rather than just adding to the list of jobs Roseway already has.
Logged

j0hn

  • Kitizen
  • ****
  • Posts: 4093
Re: DSLstats after the demise of MDWS
« Reply #74 on: March 10, 2018, 04:18:23 PM »

Quote
As I understand it the DLM resets at the end of the day anyway, so a spike the previous evening will either already have had an effect, or won't affect the current day anyway.

It works on a rolling 24 hour, not midnight to midnight.
Logged
Talktalk FTTP 550/75 - Speedtest - BQM
Pages: 1 ... 3 4 [5] 6