Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 [2] 3 4

Author Topic: Serving (slightly) better stats directly from Zyxel modems  (Read 11002 times)

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #15 on: July 03, 2018, 11:39:48 PM »

Request: could we make all web output XHTML instead of html, so that I can process the results with an XML parser? Would make it miles easier for the likes of to write tools that filter or query the output because I could pipe the output into an XSLT program.

(There is of course an XHTML dialect corresponding to HTML5, which is up-to-date, as well as the old favourite XHTML 1.0.) There might be some subtle interactions with javascript code in web pages because of the difference between xml and html in this respect.

If you think it is too much pain then I can appreciate that. It would mean running pages through a validator and checking that they are well-formed XML as well as XHTML5 or XHTML1.0.

A possibly bad idea: Some people generate an alternative XML page at a different URL, the Firebrick router does this, I think. Seems a bit wasteful to me because if it is done very cleanly and properly then the page can be easy enough to process, especially if extra id and class values are supplied as ‘hooks’ to help locate elements, and all unnecessary presentational crud is left out and implemented by CSS. The two-files idea is to give the user one that is pretty to look at for eyeballs and another that is lean and mean and easy to parse for machines. But with really good design it should be possible to do both in one to a great extent.
« Last Edit: July 04, 2018, 02:52:33 AM by Weaver »
Logged

johnson

  • Reg Member
  • ***
  • Posts: 838
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #16 on: July 04, 2018, 01:34:58 AM »

You're giving me flash backs to a first year web dev course where the exam had a final question along the lines of "now rewrite everything you have done using XHTML and XSL"... nobody was a fan, then I never encountered it again! The lecturer was a fairly old school type, was adamant we all had to sign up to some weird british computing society, made a point of showing him bring up his network interfaces on the CLI after plugging in the projector... I digress.  :D

What benefit does using XML to define webpages really have? Doesn't seem to be in wide use.

All I have at the moment in terms of HTML is a few named canvas elements, all the rest is my nasty excuse for javascript for parsing the output of xdslctl and structuring the data so it works with Chart.js.

Edit: If you mean producing an XML version of the data from the xdslctl output then thats doable I guess.
« Last Edit: July 04, 2018, 01:37:50 AM by johnson »
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #17 on: July 04, 2018, 02:48:08 AM »

XHTML is just XML that uses html elements out of a particular namespace.

All that it means for the author is just writing the source according to the very simple xhtml rules,
* always use utf8,
* lowercase element names,
* always close empty elements with <element /> type syntax eg <hr /> <br />,
* otherwise always having a matching closing tag for every start tag eg <p>blah</p> and never omitting the closing tag </p>,
* it needs <html xmlns="http://www.w3.org/1999/xhtml">
* and it needs to be validated with the w3c's checker really (web-based) and an xml validator (many good web-based ones can be googled) as any authoring mistakes defeat the point of making it machine-readable and xml-parser-friendly.

It is that easy. There are some considerations with javascript that I have not covered here. If you are ok with using the stricter markup syntax then that would be excellent, otherwise don’t worry about it.

Some useful stuff at:
https://www.w3.org/TR/html-polyglot/#writing
https://www.w3.org/TR/xhtml1/guidelines.html

The main advantage is for users. XHTML is far more machine-readable because its syntax is very rigid with no peculiar quirks as in tag soup markup or html5. Users can use XML-speaking tools to process the page, so if a user wants to extract information from it it is so much easier because these tools can find content, strip it down or filter it very conveniently. This is doable because a large range of high quality XML parsing tools are available and the XSLT programming language can be used to read, filter and generate xml and xhtml because xhtml is an application of xml.

Html5 as well as html4 can be written according to the above few simply xhtml syntax authoring rules and then the source is XHTML5 too and so also is automatically XML without anything further being needed.

One benefit to the author is that authoring mistakes can be spotted quickly
« Last Edit: July 04, 2018, 03:07:25 AM by Weaver »
Logged

johnson

  • Reg Member
  • ***
  • Posts: 838
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #18 on: October 14, 2018, 12:02:52 PM »

Have made a VMG1312 firmware with the most basic of SNRM monitoring. It logs SNRM, FEC & CRC numbers every 30 seconds, but also has the output of various xdslctl commands available with the most basic form of an API.

At modem-ip:8000 is the SNRM graph. A GET request to modem-ip:8000/getdata will update files that can be accessed from a browser (or Workflow etc) under modem-ip:8000/data/, just browse there to see a directory listing.

The SNRM graph uses the modems current time vs the browsers to give the correct labels, if you change the time manually or by NTP some time after boot things will get screwy.

I have run various versions of this for months on a VMG8324, and this specific firmware overnight with a live line on a VMG1312.

Apart from the stats server this is vanilla 1312 AAJZ16 with the jumbo frame patches.

https://www.dropbox.com/s/y6f6pcgnwyz8dn4/1312-B10A-AAJZ16-SNRM-log.bin?dl=0
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #19 on: October 15, 2018, 12:24:14 AM »

Have NOT blown this into a couple of modems. They work fine, but I just get an error if I http-connect to port 8000 decimal. (Not 8080.)

I also tried 0x8000. If I remember correctly, my old boss once invited me round to his house, gave me the address with the house number in hex, without thinking about it. When I got there, I suddenly realised, and then went to the correct house which was #0x10.

I tried Safari and the NetToolbox app’s http test function, and then snooped on the whole conversation using the ‘Charles’ app which is an http-proxy debugger and saw the following response:

method GET
Error code NSPOSIXErrorDomainCode=61 "Connection refused" - error in connect() function

or similar, stupid thing would it let me copy-paste the result.
« Last Edit: October 16, 2018, 02:54:51 AM by Weaver »
Logged

johnson

  • Reg Member
  • ***
  • Posts: 838
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #20 on: October 15, 2018, 06:50:05 PM »

Oh thats disappointing!

I have tried it with the zyxel firewall set to high from wired and wireless connections from PC/tablet/phone, so was sure it just "worked".  :(

I did indeed mean port 8000 decimal. A little google of "NSPOSIXErrorDomainCode=61 Connection refused" suggests its just a generic response to being unable to find the host.

If you are game for troubleshooting the first this would be to make sure the server is running on the device. Telnet in and gain real shell with "xtm && sh". The search the running processes for "stats":

Code: [Select]
~ # ps | grep stats
  262 supervis  1148 S    stats-server-bin
  263 supervis  2148 S    /bin/bash /var/tmp/stats/./stats-logging.sh

If it the stats server is running then the next thing to check the right files exist in /var/tmp/stats:

Code: [Select]
~ # cd /var/tmp/stats
/var/tmp/stats # ls
Chart.min.js      getdata           jquery.min.js
data              getstats.sh       moment.min.js
getCurTime.sh     index.html        stats-logging.sh
/var/tmp/stats #

Check the log file for data:

Code: [Select]
/var/tmp/stats # tail data/logfile
1514907468 4.3 6.5 8558 0 0 0
1514907498 4.4 6.4 3752 0 0 0
1514907528 4.3 6.4 6155 0 0 0
1514907559 4.3 6.4 5115 0 0 0
1514907589 4.3 6.4 3528 0 0 0
1514907619 4.3 6.4 2893 0 0 0
1514907649 4.2 6.5 3072 0 0 0
1514907680 4.4 6.5 3377 0 0 0
1514907710 4.3 6.5 2842 0 0 0
1514907740 4.3 6.4 2981 0 0 4
/var/tmp/stats #

Have you tried accessing the data folder from a browser, i.e "192.168.2.1:8000/data" ?

Apologies for supplying something that hasn't worked, and no problem if you'd rather not troubleshoot, I know how tedious it can be.
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #21 on: October 16, 2018, 12:17:13 AM »

No stats process running.

Error connect trying to get access to http://ip:8000/data

No file found data/logfile

I will be more than delighted to do some debugging. A thought: I can send you the horrible mongrel config which I have inherited, if that will help ? Or I can send you a modem, not that there would be much advantage that I can see. Final daft thought, with a lot of faffing about I could perhaps map a routable global IPv4 address so that inbound traffic to that address gets redirected to the modem and that way you could access it directly across the internet.

Another thought. I ought to confirm that I have successfully flashed the modem at all. The whole thing is all a bit weird what with always losing contact in the web UI when reflashing. I ought to rule out some strange cockup that caused the update process to abort.

I was wondering before about altering version strings somehow.

I am trying to think about things that you have working that I don’t - facilities, services etc both due to the config in my modem and due to the characteristics of my network environment.
« Last Edit: October 16, 2018, 12:24:38 AM by Weaver »
Logged

johnson

  • Reg Member
  • ***
  • Posts: 838
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #22 on: October 16, 2018, 12:40:45 AM »

If the stats-server-bin is not even running thats where to start.

You make a good point about altering the version string, I'll have to remember where it is to edit and use for further versions.

To make sure it has definitely been flashed you can search the /etc/profile file for the lines that start the server:

Code: [Select]
~ # grep stats /etc/profile
mkdir /var/tmp/stats
mkdir /var/tmp/stats/data
cp /webs/stats-server/* /var/tmp/stats/
chmod +x /var/tmp/stats/getstats.sh
chmod +x /var/tmp/stats/getCurTime.sh
chmod +x /var/tmp/stats/stats-logging.sh
touch /var/tmp/stats/data/logfile
...

Or simply see if "stats-server-bin" is in the PATH:

Code: [Select]
~ # stats-server-bin

I'm pretty sure I still have your default config from a while ago in a PM if it hasn't changed much since then, but as I say, making sure of a successful flash is first on the list.
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #23 on: October 16, 2018, 02:00:07 AM »

Yes, it had failed to blow the image.

All good now.

When I am using the web UI, I get a swirling useless progress indicator (why no proper percentage-done indicator, byte count, clock or growing bar) and that just goes on for ever and ever. Eventually I try hitting refresh. Then nothing much of significance is displayed, just the normal main web page. But it I go back into the upgrade page in the menus, it then immediately says "rebooting" and then no feedback again either, just sits there, so in the end when you do yet another refresh you see the login prompt which shows it has indeed rebooted. Booting still takes an eternity. In fact I know it has booted when an SMS or a Tweet arrives from AA telling me that it has connected to the internet! So anyway, this is how I have managed to screw up upgrades in the past, a lot of guesswork about what is going on. I just need to learn to follow a strict reflashing routine that works.

Logging in to the modem’s usual admin UI programmatically via http might be a bit of a pain, would have to look into it, but if I sorted that out, I could perhaps write a Shortcuts/Workflow program to reliably blow these modems. (I did write a very successful Workflow thing to push a new config into the Firebrick, and optionally tell it to reboot it too.)
Logged

johnson

  • Reg Member
  • ***
  • Posts: 838
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #24 on: October 16, 2018, 03:47:57 AM »

Ah, well good stuff thats it working now. Version stringed releases from now on!

Wish I could offer some reason that the GUI is such a pain in your setup, but have never experienced it myself. Its not the snappiest by any means, but it never bugs out or hangs on the firmware upload page.

Is the SNRM graph drawing itself ok? Have left it auto ranging as it gives pretty good results with the range my line produces.
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #25 on: October 16, 2018, 04:06:46 AM »

It’s just superb. Just so incredibly useful.

Re: the stupid standard ZyXEL web UI - I wonder if their JavaScript or possibly CSS just doesn’t work in Safari iOS or doesn’t work in machines that don’t have a normal pointing device, or both. The dsl status page thing has the bottom of the report clipped off because it is behind the icon bar that is drawn in front of it at the bottom, and the whole thing refuses to scroll. I am sure some js and css browser behaviour wizard who is familiar with Safari iOS could fix ZyXel’s junk and make it behave properly.

Can I just check - which low-level modem driver is in this new version? I ask because the sync speeds are awful - downstream sync down by 10% - nowadays on the two modems that I have reflashed. But it has to be due to the number of resyncs upsetting DLM or my choice of bad times for resynching, because the speeds were already bad when I had actually failed to reflash the device. There is no way it can be due to this new release.

Just need to think how to get the performance back, and the only thing I can think of is a wise choice of time to resynch, that or more mysterious long naps in the powered-off state.

I have changed the settings, or am trying to, as BTW are taking forever, so that the link’s interleaving setting is ‘on’ (ie max, hopefully) instead of ‘auto’. It’s a feature available in AA’s control server.
Logged

johnson

  • Reg Member
  • ***
  • Posts: 838
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #26 on: October 16, 2018, 04:19:15 AM »

Its the standard default A2pv6F039v adsl_phy. Thats the one you use normally right?

All I can suggest is to wait till you have 24 hours of SNR data and pick an optimum resync time (via an xdslctl configure --snr reduction maybe) and hope for the best. I assume at least the most recent flash and resync was during darkness hours so probably suboptimal.
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #27 on: October 16, 2018, 04:28:30 AM »

> the most recent flash and resync was during darkness hours so probably suboptimal.

Absolutely. That is what I was thinking. I don’t know what else there is, what other factors, that make a difference over time. Why exactly might my crosstalk vary - if indeed it does vary ? Why might rf interference from distant sources vary? Attenuation has changed over time, 0.3dB change to upstream attenuation was noticed by AA staff. So perhaps it’s to do with the electromagnetic environment around the cable, guessing, maybe water present or absent - who knows?
Logged

dee.jay

  • ISP Rep
  • Reg Member
  • *
  • Posts: 952
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #28 on: October 16, 2018, 08:06:30 AM »

I recently installed a VMG1312-B10A on my dad's VDSL line that's connected to an ECI cabinet, hence no G.INP etc.

Would that be of any use to you?
Logged
Starlink and AAISP L2TP combo routed by opnSense on proxmox

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Serving (slightly) better stats directly from Zyxel modems
« Reply #29 on: October 16, 2018, 10:40:08 PM »

And you do not need it?
Logged
Pages: 1 [2] 3 4