Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Author Topic: Missing stats randomly - MDWS/HG612modem stats.  (Read 3863 times)

WWWombat

  • Kitizen
  • ****
  • Posts: 1674
Missing stats randomly - MDWS/HG612modem stats.
« on: July 26, 2016, 02:10:33 AM »

As of 13:30 on July 25th (yesterday), MDWS had started receiving a lot of invalid, zero, data (thanks to Tony for the heads-up), and displaying that data is zig-zag graphs (eg the SNRM one).

I'm not sure why - both the machine sampling the stats (via HG612 modem_stats) and the router have been up and running just fine. They are connected via wired LAN, and a ping from PC (Windows 7) to the router (Billion 8800NL) has performed flawlessly for the last 10 hours, since about 2pm).

If I telnet into the modem (on a different computer), the modem responds just fine, and gives responses to commands. All pretty normal.

When I look at the "modem_stats.log" file, the records are missing for the same minutes that MDWS is getting all-zero data uploaded to it. That is - the file has no line for that time, whereas MDWS is receiving zeroed data.

Then ...

I tried telnetting into the modem from the PC which collects the stats. It worked fine at first, and I could issue a "--stats" command. However, while I was scrolling back and forth, the telnet window (I'm using Putty) just disappeared ... no warning whatsoever.

So I brought the Putty window up again, telnetted into the modem, ran a couple of commands ... and after a short while, the window just disappeared again.

I tried a few more times over the afternoon, with the same result.

Then ...

I left things alone, intending to go back in the evening, and all the failures just stopped around 7pm. I wasn't doing anything with that machine at the time things fixed themselves.

Now, when I go back and bring up a Putty terminal, it stays up. It doesn't just disappear. (And, while I'm still logged in, the modem_stats collection continues unabated).

Perhaps this will tell someone something about one reason why invalid data can be sent to MDWS, relating to a crashing Putty terminal.

Unfortunately ...

My PC is still somewhat stuffed with permissions, so it is still running old versions. In fact, something is also mucked up, because the GUI doesn't think there's a stats task configured, but it is running in the background.

I'm about to shift to monitoring via a Raspberry Pi, so I guess I won't get much more ability to try any fixes out. Sorry!

Then ...

I noticed some small gaps in the graphs - for example, they are at 20:14, 21:14, 22:14. Those don't get entries in the modem_stats file either, but they don't seem to cause the zig-zig graphs; just a gap.

Those gaps seem a little too regular, so I suspect they show a different kind of fault.
« Last Edit: July 26, 2016, 02:18:40 AM by WWWombat »
Logged

tbailey2

  • Kitizen
  • ****
  • Posts: 1245
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #1 on: July 26, 2016, 08:02:04 AM »

Thanks!

This phenomena isn't new  - you were doing it on the 9th July for a long time as well. In the past year, only two other users have uploaded zero filled files but for much shorter periods. But it is something that has happened occasionally since MDWS began running. I think BE1 did some investigation early on. MDWS may start sending alert mails (maybe every 5 minutes) while it's happening.

Edit:
I meant to mention that the zig-zag effect is because the zero uploads are mixed in with valid ones/no upload at all...

Coincidentally, I was about to make another post on another reason for some missing uploads so here it is. I've been liaising with roseway on this. I think BE1 may be on holiday.

MISSING UPLOADS

Having been looking at solving this for some time, I've found one reason. This won't solve all such instances but it will some.

For a few people, the missing uploads are caused by uploading the sample from your client almost exactly on the change of minute i.e. at 59 or 00 secs. The client sometimes, presumably due to rounding errors, will state that the minute is ahead or behind of where it should be depending on how close to 00 secs it was. For some reason it's worse on the hour changeover. (I tried to explain this with an example  - I can see it in my head but doesn't translate to words so easily). This generates a Duplicate entry (unique Index of User+DateTime) error in the database table against the previous upload so that minute is not added and the minute goes missing...

It happens with both DSLStats and HG612 Stats although more often with the latter due to the way HG612 Stats is set up and often samples closer to 00 secs. 

The cure for the moment is to pause the client and restart it maybe around between 5 and 55 secs past the hour if you see this happening. If you run both stats programs then make the necessary allowances. In theory it may be possible to correct this on receipt of the log which I will look at when I get a chance... As said many times, your clock needs to be accurate in the first place for the MDWS system to work although this problem is based purely on what your uploaded log says..  Longer term the upload clients could maybe be modified but I am aware of time and other pressures making this difficult. Please bear in mind this is all a collaborated best effort system with no guarantees and debugging can take an awful long time.

As I said. it won't cure all instances of missing uploads, some of which stop and start for no apparent reason . The only other reason found in the past is if the machine hibernates, then on resume if often starts very intermittent uploads with DSLstats, Restarting the latter cures it. Also the Billion 8800NL router seems to have the missing individual uploads a lot more more often than other routers (check the local log for timeouts maybe).

Users affected - yesterday I saw Chrysalis (but okay later as 00 upload has changed to 15 secs), jeffers, jaggyt and spiralarms with 00-related gaps but this could change at any time.
« Last Edit: July 26, 2016, 10:13:51 AM by tbailey2 »
Logged
Tony
My Books!
Plusnet 80/20 - DSLstats - HG612/TG582n - ECI

Ronski

  • Moderator
  • Kitizen
  • *
  • Posts: 4300
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #2 on: July 26, 2016, 10:14:56 AM »

In hg612 stats it's  possible to schedule the task to run at xx seconds, to do this you need to delete the scheduled task and then change the delay on one of the settings tab, then create the task.

I only suggest doing this if you have the problem as tbailey2 states. Also bear in mind if running both stats logging programs you need to make sure they don't clash.
« Last Edit: July 26, 2016, 01:17:29 PM by Ronski »
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43467
  • Penguins CAN fly
    • DSLstats
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #3 on: July 26, 2016, 10:48:18 AM »

Concerning the last point, when both programs are running and the "co-operation" option is set in DSLstats, then DSLstats samples between 35 and 40 seconds after the minute. So there shouldn't be any conflict if HG612_Stats is delayed by a few seconds. Don't forget that only one of the programs should be set to upload to MDWS.
Logged
  Eric

tbailey2

  • Kitizen
  • ****
  • Posts: 1245
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #4 on: July 26, 2016, 10:52:39 AM »

Also bare  :-X  in mind if running both stats logging programs you need to make sure they don't clash.
It's bear in this context ..... Unless you are imagining someone with no clothes on in which case you may be correct  ::)
Logged
Tony
My Books!
Plusnet 80/20 - DSLstats - HG612/TG582n - ECI

skyeci

  • Kitizen
  • ****
  • Posts: 1383
    • Line stats
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #5 on: July 26, 2016, 11:47:55 AM »

Hi Tony

seems to me its probably modem related.  From my experience of using the 8800nl and the vmg8924 both suffer from missing minutes from time to time but my vmg8324 never seems to skip any time at all (using hg612 stats), the only time it has lost the odd minute is when there has been an identified issue at the mydsl end. Its always at 100% unless there is a problem at the server

It seems to be a good one to look at to see if there are any actual issues with uploads gaps etc..

cheers

Ronski

  • Moderator
  • Kitizen
  • *
  • Posts: 4300
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #6 on: July 26, 2016, 01:18:54 PM »

It's bear in this context ..... Unless you are imagining someone with no clothes on in which case you may be correct  ::)

I'm going to blame my tablet  keyboard, rather than my poor English!
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

NewtronStar

  • Kitizen
  • ****
  • Posts: 4898
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #7 on: July 26, 2016, 07:07:24 PM »

Very odd these random gaps it's has only started recently the date seems to point to 17th of July before that it was always 100% on the 8800NL this also happened when using the HG612 it would be 100% for a month or so unless I intervened and then random gaps in the SNRm graph

The worst gaps or no uploading so far is 10 minutes with no errors in the logs on DSLstats and it happens when I'm not monitoring the client
« Last Edit: July 26, 2016, 07:37:09 PM by NewtronStar »
Logged

WWWombat

  • Kitizen
  • ****
  • Posts: 1674
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #8 on: July 28, 2016, 11:52:38 AM »

Some extra information discovered.

Note about how the modem was interrogated:

a) My Win7 PC had both DSLstats and HG612 modem_stats running, with them configured to not interfere with each other. I think that meant modem_stats ran at 5-seconds past the minute (and had the MDWS upload duty), while DSLstats ran at around 40 seconds past the minute.

b) At the time, I had also my Raspberry Pi up and running for a while, before giving it prime duty for uploading to MDWS.
That means it was interrogating the modem too, but with no coordination with the other machine.

All 3 seem to (normally) coexist just fine.

The 8800NL seems to allow multiple logins. When I've left myself manually logged in (and putty hasn't killed things), the presence of that login hasn't stopped any stats gathering.

Anyway...

I discovered that the DSLstats Event log on the Raspberry Pi had a lot of events like this:
25 Jul 2016 03:37:29    Auto snapshots taken
25 Jul 2016 11:37:29    Auto snapshots taken
25 Jul 2016 12:37:19    Timeout while collecting SNR per tone
25 Jul 2016 12:37:19    Telnet timeout
25 Jul 2016 12:39:19    Timeout while collecting SNR per tone
25 Jul 2016 12:39:19    Telnet timeout
25 Jul 2016 12:45:19    Timeout while collecting SNR per tone
25 Jul 2016 12:45:19    Telnet timeout
25 Jul 2016 12:47:19    Timeout while collecting SNR per tone
25 Jul 2016 12:47:19    Telnet timeout
25 Jul 2016 12:50:19    Timeout while collecting SNR per tone
25 Jul 2016 12:50:19    Telnet timeout


Or this:
25 Jul 2016 14:23:19    Timeout while collecting SNR per tone
25 Jul 2016 14:23:19    Telnet timeout
25 Jul 2016 14:24:19    Timeout while collecting SNR per tone
25 Jul 2016 14:24:19    Telnet timeout
25 Jul 2016 14:25:19    Timeout while collecting SNR per tone
25 Jul 2016 14:25:19    Unable to collect stats 3 or more times in succession
25 Jul 2016 14:25:20    Telnet timeout
25 Jul 2016 14:26:19    Timeout while collecting SNR per tone
25 Jul 2016 14:26:19    Telnet timeout
25 Jul 2016 14:27:19    Timeout while collecting bitloading data
25 Jul 2016 14:27:20    Telnet timeout
25 Jul 2016 14:28:19    Timeout while collecting SNR per tone
25 Jul 2016 14:28:20    Telnet timeout
25 Jul 2016 14:29:19    Timeout while collecting bitloading data
25 Jul 2016 14:29:20    Telnet timeout


or eventually this:
25 Jul 2016 16:53:19    Timeout while collecting bitloading data
25 Jul 2016 16:53:19    Telnet timeout
25 Jul 2016 17:07:19    Timeout while collecting bitloading data
25 Jul 2016 17:07:19    Telnet timeout
25 Jul 2016 17:27:19    Timeout while retrieving stats
25 Jul 2016 17:27:19    No stats received
25 Jul 2016 17:38:19    Timeout while retrieving stats
25 Jul 2016 17:38:19    No stats received
25 Jul 2016 17:42:19    Timeout while retrieving stats
25 Jul 2016 17:42:19    No stats received
25 Jul 2016 17:43:19    Timeout while retrieving stats
25 Jul 2016 17:43:19    No stats received
25 Jul 2016 17:50:19    Timeout while retrieving stats
25 Jul 2016 17:50:19    No stats received
25 Jul 2016 18:06:19    Timeout while retrieving stats
25 Jul 2016 18:06:19    No stats received
25 Jul 2016 19:37:29    Auto snapshots taken


A lot of those events started out as failures during SNR reading, then morphed  into failures during bitloading, then morphed into failures during stats retrieval.

These events were all on DSLstats on the RPi at around the same time that the MDWS upload was having problems - though that was still back on the Win7 PC at the time, with HG612 modem_stats in charge.

I can't check if the DSLstats on Win7 saw the same thing (machine offline right now).
Logged

William Grimsley

  • Kitizen
  • ****
  • Posts: 1489
    • Newton Poppleford Weather
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #9 on: August 01, 2016, 08:54:19 AM »

Been trying out Tony's somewhat of a fix for missing uploads, sometimes it works and sometimes it doesn't.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #10 on: August 19, 2016, 01:10:34 AM »

There's also the known issue with windows task scheduler which can cause HG612 modemstats to do silly things.

What happens is that every hour (not on the hour) it triggers 2 instances to run.   HG612modemstats detects 2 instances and neither are uploaded or occasionally HG612stats can get stuck.

There is a windows 7 hotfix for this
http://forum.kitz.co.uk/index.php/topic,17613.msg322495/topicseen.html#msg322495

Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

tbailey2

  • Kitizen
  • ****
  • Posts: 1245
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #11 on: August 19, 2016, 07:14:14 AM »

@William's problem isn't and never has been due to duplicate client uploads which is what moving the upload time fixes...

Eric has fixed DSLstats to stop it sampling before about :05 and I believe you can tell HG612 Stats (which is the worst offender) to delay uploading for x secs past :00 which will fix that particular problem.
Logged
Tony
My Books!
Plusnet 80/20 - DSLstats - HG612/TG582n - ECI

Ronski

  • Moderator
  • Kitizen
  • *
  • Posts: 4300
Re: Missing stats randomly - MDWS/HG612modem stats.
« Reply #12 on: August 19, 2016, 08:03:47 AM »

Eric has fixed DSLstats to stop it sampling before about :05 and I believe you can tell HG612 Stats (which is the worst offender) to delay uploading for x secs past :00 which will fix that particular problem.

You can set the delay on the additional settings tab, but you'll need to delete the scheduled task first, which can be done from the tasks menu option and then set the task up again after changing the delay.
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D
 

anything