Kitz Forum

Broadband Related => Broadband Hardware => Topic started by: johnson on June 07, 2018, 01:04:47 AM

Title: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 07, 2018, 01:04:47 AM
Thought I should split this off from the other thread talking about firmware as its so off topic.

Have got a simple webserver compiled and working from boot on a 1312, uses the relatively small mongoose (https://github.com/cesanta/mongoose) server (160K) with the simplest possible setup, running a script to gather stats and then serving a directory under /var/tmp (from ram).

Code: [Select]
#include "mongoose.h"

static const char *s_http_port = "8000";
static struct mg_serve_http_opts s_http_server_opts;

static void ev_handler(struct mg_connection *nc, int ev, void *p) {
  if (ev == MG_EV_HTTP_REQUEST) {
    system("/var/tmp/stats/./getstats.sh");
    mg_serve_http(nc, (struct http_message *) p, s_http_server_opts);
  }
}

int main(void) {
  struct mg_mgr mgr;
  struct mg_connection *nc;

  mg_mgr_init(&mgr, NULL);
  printf("Starting web server on port %s\n", s_http_port);
  nc = mg_bind(&mgr, s_http_port, ev_handler);
  if (nc == NULL) {
    printf("Failed to create listener\n");
    return 1;
  }

  // Set up HTTP server parameters
  mg_set_protocol_http_websocket(nc);
  s_http_server_opts.document_root = "/var/tmp/stats/";  // Serve current directory
  s_http_server_opts.enable_directory_listing = "yes";

  for (;;) {
    mg_mgr_poll(&mgr, 1000);
  }
  mg_mgr_free(&mgr);

  return 0;
}


Arbitrary commands can be placed in "getstats.sh" and their outputs are accessible on routerip:8000.

Not any better than the xDSL Statistics section of the normal GUI except for being able to gather SNR/QLN etc. These could be made into pretty graphs client side, but taking a stab with 5 minutes of googling javascript plotting has only resulted in ugly graphs covered in tooltips and other pointless stuff. If anyone knows of an ultra lightweight plotting library for js that would be great.


This is in no way meant to even remotely compete with the other excellent stats monitoring tools available, but I though it would be nice to be able to glance at things like SNR per tone, bitloading etc from a URL served by the modem itself. I guess a small log of other data could work, like the last 24 hours SNR graph or similar.

Totally stupid idea? Probably! But its fun to play with these things. 
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: banger on June 07, 2018, 03:12:29 AM
Good work. :)
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: burakkucat on June 07, 2018, 05:14:07 PM
Totally stupid idea? Probably! But its fun to play with these things.

No, not stupid by any means. Quite an interesting idea.

What might be worthwhile would be to "serve up" the data used to create the four snapshot plots (Bit Loading, Hlog, QLN & SNR) and the data returned by "info --pbParams" & "info --stats".
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 08, 2018, 02:45:03 AM
Got to messing with chart.js this evening and have some reasonable looking graphs, was starting to try and implement a nice way of choosing ranges, pbParams should work well for selected a sensible range as well separating upload and download sections of SNR/bitloading. But it occurred to me, does pbParams even work with ADSL? How about lines without G.INP, are they missing the whole "Bearer 1" line? If so just using offsets to get to the right data isnt going to work.

So I guess I could go digging through posts to find examples of xdslctl output for various lines and probably should. But can anyone with an ADSL line or VDSL without G.INP provide a few outputs? E.g. --pbParams, the first 10 or so lines of --SNR or --Bits or --Hlog (I assume they are the same).

Much appreciated if anyone can be bothered!  :D
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on June 08, 2018, 11:12:28 AM
pbParams does not work on my ADSL line that has no G.INP (but does we think have Broadcom PhyR their proprietary equivalent)
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: roseway on June 08, 2018, 11:36:20 AM
For the various ADSL flavours there are only two sets of tones - a range for upstream starting at near to 0, and the remainder for downstream starting at different points depending on the spcific protocol. Look at this article (https://en.wikipedia.org/wiki/Asymmetric_digital_subscriber_line#ADSL_standards) for the details. In the UK only Annex A and Annex M need to be considered.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 08, 2018, 11:58:06 AM
Input from the maestro! Thanks for the the links and info, sure I can concoct some ADSL detection and apply the right ranges then.

OT, but I'd really like to thank you for DSLstats, its an invaluable tool. 
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: spring on June 08, 2018, 03:04:36 PM
https://forum.kitz.co.uk/index.php?topic=20691.0
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 08, 2018, 06:16:22 PM
pbParams does not work on my ADSL line that has no G.INP (but does we think have Broadcom PhyR their proprietary equivalent)

Thank you!

The output of xdslctl --SNR or --Bits would still be useful Weaver, only the first 10 or so lines down to where the tones/values start.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on June 09, 2018, 04:59:35 PM
@johnson no luck with xdslctl --SNR or -Bits on my VMG 1312-B10A with your second firmware - jumbo + updated low-level driver BLOB.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: burakkucat on June 09, 2018, 07:01:45 PM
Here follows my list of "useful commands" for a VMG1312-B10A device --

xdslctl info --state
xdslctl info --show
xdslctl info --stats
xdslctl info --SNR
xdslctl info --QLN
xdslctl info --Hlog
xdslctl info --Hlin
xdslctl info --HlinS
xdslctl info --Bits
xdslctl info --pbParams
xdslctl info --linediag
xdslctl info --linediag1
xdslctl info --vendor
xdslctl info --cfg
xdslctl info --webstats
xdslctl info --vectoring
xdslctl profile --show
xdslctl --version


I have deliberately emboldened two of the lines . . .  ;)
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on June 09, 2018, 09:20:22 PM
Ah, I got the syntax wrong: it requires the word "info" not just a -parameter. Bits and SNR are below:
Code: [Select]
VMG1312-B10A
Login: admin
Password:
 > xdslctl info --Bits
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    8000
Last initialization procedure status:   0
Max:    Upstream rate = 440 Kbps, Downstream rate = 3152 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps

Tone number      Bit Allocation
   0            0
   1            0
   2            0
   3            0
   4            0
   5            0
   6            0
   7            8
   8            9
   9            10
   10           10
   11           9
   12           9
   13           9
   14           9
   15           8
   16           8
   17           8
   18           8
   19           7
   20           7
   21           7
   22           7
   23           7
   24           6
   25           6
   26           6
   27           6
   28           5
   29           5
   30           5
   31           4
   32           0
   33           8
   34           8
   35           9
   36           10
   37           10
   38           11
   39           11
   40           11
   41           11
   42           11
   43           11
   44           11
   45           11
   46           11
   47           11
   48           11
   49           11
   50           2
   51           11
   52           11
   53           11
   54           11
   55           11
   56           11
   57           11
   58           11
   59           11
   60           11
   61           11
   62           11
   63           11
   64           11
   65           11
   66           11
   67           10
   68           10
   69           11
   70           10
   71           10
   72           10
   73           10
   74           8
   75           9
   76           9
   77           10
   78           9
   79           10
   80           10
   81           9
   82           9
   83           9
   84           9
   85           9
   86           9
   87           9
   88           9
   89           8
   90           8
   91           8
   92           8
   93           8
   94           8
   95           8
   96           7
   97           8
   98           7
   99           7
   100          7
   101          7
   102          7
   103          7
   104          7
   105          7
   106          6
   107          6
   108          6
   109          6
   110          6
   111          6
   112          6
   113          6
   114          5
   115          5
   116          5
   117          5
   118          5
   119          5
   120          4
   121          4
   122          4
   123          3
   124          0
   125          1
   126          0
   127          2
   128          2
   129          3
   130          3
   131          3
   132          3
   133          3
   134          3
   135          3
   136          3
   137          3
   138          3
   139          3
   140          3
   141          2
   142          2
   143          2
   144          2
   145          2
   146          1
   147          2
   148          2
   149          1
   150          1
   151          1
   152          1
   153          1
   154          1
   155          0
   156          0
   157          0
   158          0
   159          0
   160          0
   161          0
   162          0
   163          0
   164          0
   165          0
   166          0
   167          0
   168          0
   169          0
   170          0
   171          0
   172          0
   173          0
   174          0
   175          0
   176          0
   177          0
   178          0
   179          0
   180          0
   181          0
   182          0
   183          0
   184          0
   185          0
   186          0
   187          0
   188          0
   189          0
   190          0
   191          0
   192          0
   193          0
   194          0
   195          0
   196          0
   197          0
   198          0
   199          0
   200          0
   201          0
   202          0
   203          0
   204          0
   205          0
   206          0
   207          0
   208          0
   209          0
   210          0
   211          0
   212          0
   213          0
   214          0
   215          0
   216          0
   217          0
   218          0
   219          0
   220          0
   221          0
   222          0
   223          0
   224          0
   225          0
   226          0
   227          0
   228          0
   229          0
   230          0
   231          0
   232          0
   233          0
   234          0
   235          0
   236          0
   237          0
   238          0
   239          0
   240          0
   241          0
   242          0
   243          0
   244          0
   245          0
   246          0
   247          0
   248          0
   249          0
   250          0
   251          0
   252          0
   253          0
   254          0
   255          0
 > xdslctl info --QLN
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    8000
Last initialization procedure status:   0
Max:    Upstream rate = 440 Kbps, Downstream rate = 3144 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps

Tone number      QLN
   0            -150.5000
   1            -150.5000
   2            -150.5000
   3            -150.5000
   4            -150.5000
   5            -150.5000
   6            -150.5000
   7            -150.5000
   8            -150.5000
   9            -150.5000
   10           -150.5000
   11           -150.5000
   12           -150.5000
   13           -150.5000
   14           -150.5000
   15           -150.5000
   16           -150.5000
   17           -150.5000
   18           -150.5000
   19           -150.5000
   20           -150.5000
   21           -150.5000
   22           -150.5000
   23           -150.5000
   24           -150.5000
   25           -150.5000
   26           -150.5000
   27           -150.5000
   28           -150.5000
   29           -150.5000
   30           -150.5000
   31           -150.5000
   32           -150.5000
   33           -124.0000
   34           -127.5000
   35           -130.0000
   36           -132.0000
   37           -133.0000
   38           -135.0000
   39           -136.0000
   40           -136.0000
   41           -136.5000
   42           -136.5000
   43           -136.5000
   44           -137.5000
   45           -137.0000
   46           -137.0000
   47           -137.5000
   48           -137.5000
   49           -137.5000
   50           -137.5000
   51           -138.0000
   52           -138.5000
   53           -138.0000
   54           -138.5000
   55           -139.0000
   56           -140.0000
   57           -139.0000
   58           -137.5000
   59           -138.5000
   60           -140.0000
   61           -139.5000
   62           -140.5000
   63           -140.5000
   64           -141.0000
   65           -141.0000
   66           -141.5000
   67           -141.5000
   68           -142.0000
   69           -143.0000
   70           -142.5000
   71           -142.5000
   72           -142.0000
   73           -143.0000
   74           -137.5000
   75           -141.5000
   76           -142.5000
   77           -143.5000
   78           -144.5000
   79           -145.5000
   80           -145.5000
   81           -145.5000
   82           -145.0000
   83           -145.0000
   84           -145.5000
   85           -146.5000
   86           -146.0000
   87           -146.5000
   88           -146.0000
   89           -146.5000
   90           -146.0000
   91           -146.5000
   92           -146.5000
   93           -146.5000
   94           -147.0000
   95           -146.5000
   96           -147.0000
   97           -148.0000
   98           -147.0000
   99           -148.5000
   100          -148.0000
   101          -147.5000
   102          -148.0000
   103          -147.5000
   104          -147.0000
   105          -148.0000
   106          -148.0000
   107          -148.0000
   108          -148.0000
   109          -148.0000
   110          -147.5000
   111          -148.5000
   112          -148.0000
   113          -148.0000
   114          -148.0000
   115          -148.5000
   116          -148.0000
   117          -148.5000
   118          -148.5000
   119          -147.5000
   120          -147.5000
   121          -148.0000
   122          -146.5000
   123          -144.5000
   124          -143.0000
   125          -137.5000
   126          -136.5000
   127          -143.5000
   128          -145.0000
   129          -145.0000
   130          -146.0000
   131          -147.5000
   132          -148.0000
   133          -148.5000
   134          -149.0000
   135          -149.0000
   136          -148.0000
   137          -148.0000
   138          -148.5000
   139          -149.0000
   140          -149.5000
   141          -149.5000
   142          -149.5000
   143          -149.5000
   144          -149.0000
   145          -149.5000
   146          -149.0000
   147          -149.5000
   148          -149.5000
   149          -150.0000
   150          -149.5000
   151          -149.0000
   152          -149.5000
   153          -149.5000
   154          -149.0000
   155          -149.5000
   156          -150.0000
   157          -149.5000
   158          -149.5000
   159          -149.5000
   160          -149.0000
   161          -148.5000
   162          -149.0000
   163          -149.0000
   164          -149.5000
   165          -149.0000
   166          -149.5000
   167          -149.0000
   168          -149.5000
   169          -149.5000
   170          -150.0000
   171          -149.0000
   172          -149.5000
   173          -148.5000
   174          -150.0000
   175          -149.5000
   176          -149.5000
   177          -148.5000
   178          -149.5000
   179          -149.5000
   180          -149.5000
   181          -149.5000
   182          -149.5000
   183          -149.5000
   184          -149.0000
   185          -150.0000
   186          -149.0000
   187          -148.0000
   188          -142.5000
   189          -149.0000
   190          -150.0000
   191          -149.5000
   192          -149.0000
   193          -149.0000
   194          -149.0000
   195          -149.0000
   196          -147.5000
   197          -149.0000
   198          -149.0000
   199          -149.0000
   200          -149.5000
   201          -148.5000
   202          -148.5000
   203          -149.0000
   204          -148.0000
   205          -148.5000
   206          -148.0000
   207          -148.5000
   208          -149.0000
   209          -147.0000
   210          -148.5000
   211          -147.5000
   212          -148.0000
   213          -149.5000
   214          -150.0000
   215          -149.0000
   216          -146.5000
   217          -148.0000
   218          -149.0000
   219          -148.5000
   220          -148.5000
   221          -149.5000
   222          -149.5000
   223          -149.5000
   224          -148.0000
   225          -148.5000
   226          -146.0000
   227          -148.5000
   228          -149.5000
   229          -149.0000
   230          -149.0000
   231          -148.5000
   232          -148.0000
   233          -149.0000
   234          -148.0000
   235          -147.5000
   236          -144.5000
   237          -149.5000
   238          -149.0000
   239          -147.5000
   240          -149.5000
   241          -149.0000
   242          -148.5000
   243          -149.0000
   244          -149.0000
   245          -148.5000
   246          -147.5000
   247          -149.0000
   248          -149.0000
   249          -146.5000
   250          -148.0000
   251          -150.0000
   252          -149.5000
   253          -149.5000
   254          -148.0000
   255          -148.5000
 > xdslctl info --pbParams
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    8000
Last initialization procedure status:   0
Max:    Upstream rate = 440 Kbps, Downstream rate = 3148 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps

Currently not in VDSL modulation --pbParams is only for VDSL mode
 > xdslctl info --SNR     
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:    8000
Last initialization procedure status:   0
Max:    Upstream rate = 440 Kbps, Downstream rate = 3152 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps

Tone number      SNR
   0            0.0000
   1            0.0000
   2            0.0000
   3            0.0000
   4            0.0000
   5            0.0000
   6            0.0000
   7            0.0000
   8            0.0000
   9            0.0000
   10           0.0000
   11           0.0000
   12           0.0000
   13           0.0000
   14           0.0000
   15           0.0000
   16           0.0000
   17           0.0000
   18           0.0000
   19           0.0000
   20           0.0000
   21           0.0000
   22           0.0000
   23           0.0000
   24           0.0000
   25           0.0000
   26           0.0000
   27           0.0000
   28           0.0000
   29           0.0000
   30           0.0000
   31           0.0000
   32           0.0000
   33           26.8750
   34           29.5625
   35           32.1875
   36           34.3750
   37           35.6250
   38           36.6250
   39           37.3125
   40           37.8125
   41           38.2500
   42           38.5625
   43           38.7500
   44           39.0000
   45           39.1875
   46           38.4375
   47           39.3125
   48           39.6875
   49           39.4375
   50           37.4375
   51           39.5000
   52           39.2500
   53           39.1875
   54           39.2500
   55           38.7500
   56           38.4375
   57           38.2500
   58           37.1875
   59           36.8125
   60           37.2500
   61           37.5000
   62           37.3750
   63           36.8750
   64           37.0625
   65           36.9375
   66           36.6875
   67           35.7500
   68           35.8750
   69           36.5625
   70           36.3750
   71           35.6250
   72           34.3125
   73           34.5625
   74           28.6875
   75           33.0000
   76           32.6875
   77           34.3750
   78           33.8125
   79           33.8750
   80           33.5625
   81           33.1250
   82           32.6250
   83           32.2500
   84           32.2500
   85           32.0000
   86           31.3750
   87           30.8750
   88           30.7500
   89           30.0625
   90           30.3125
   91           29.8750
   92           29.4375
   93           29.1875
   94           28.9375
   95           28.4375
   96           28.0000
   97           27.7500
   98           27.2500
   99           26.8125
   100          26.6250
   101          26.1875
   102          26.0625
   103          25.6875
   104          24.9375
   105          24.9375
   106          24.4375
   107          24.0000
   108          23.5625
   109          23.0625
   110          22.8750
   111          22.4375
   112          22.0000
   113          21.5625
   114          20.5625
   115          20.8125
   116          20.3750
   117          19.7500
   118          19.6250
   119          19.1875
   120          17.5000
   121          17.1875
   122          16.7500
   123          13.4375
   124          7.1875
   125          8.3750
   126          8.2500
   127          11.7500
   128          10.1250
   129          14.3125
   130          13.1875
   131          14.6250
   132          13.8750
   133          13.8750
   134          13.3125
   135          13.5625
   136          13.1250
   137          12.9375
   138          12.5625
   139          12.2500
   140          12.0000
   141          11.5625
   142          11.7500
   143          11.0625
   144          11.1250
   145          10.6875
   146          10.5625
   147          10.0000
   148          9.6250
   149          9.5625
   150          9.6250
   151          9.1250
   152          9.0000
   153          8.5625
   154          8.3125
   155          0.7500
   156          8.2500
   157          7.6875
   158          7.3750
   159          7.1250
   160          6.8125
   161          4.9375
   162          0.0000
   163          0.0000
   164          0.0000
   165          0.0000
   166          0.0000
   167          0.0000
   168          0.0000
   169          0.0000
   170          0.0000
   171          0.0000
   172          0.0000
   173          0.0000
   174          0.0000
   175          0.0000
   176          0.0000
   177          0.0000
   178          0.0000
   179          0.0000
   180          0.0000
   181          0.0000
   182          0.0000
   183          0.0000
   184          0.0000
   185          0.0000
   186          0.0000
   187          0.0000
   188          0.0000
   189          0.0000
   190          0.0000
   191          0.0000
   192          0.0000
   193          0.0000
   194          0.0000
   195          0.0000
   196          0.0000
   197          0.0000
   198          0.0000
   199          0.0000
   200          0.0000
   201          0.0000
   202          0.0000
   203          0.0000
   204          0.0000
   205          0.0000
   206          0.0000
   207          0.0000
   208          0.0000
   209          0.0000
   210          0.0000
   211          0.0000
   212          0.0000
   213          0.0000
   214          0.0000
   215          0.0000
   216          0.0000
   217          0.0000
   218          0.0000
   219          0.0000
   220          0.0000
   221          0.0000
   222          0.0000
   223          0.0000
   224          0.0000
   225          0.0000
   226          0.0000
   227          0.0000
   228          0.0000
   229          0.0000
   230          0.0000
   231          0.0000
   232          0.0000
   233          0.0000
   234          0.0000
   235          0.0000
   236          0.0000
   237          0.0000
   238          0.0000
   239          0.0000
   240          0.0000
   241          0.0000
   242          0.0000
   243          0.0000
   244          0.0000
   245          0.0000
   246          0.0000
   247          0.0000
   248          0.0000
   249          0.0000
   250          0.0000
   251          0.0000
   252          0.0000
   253          0.0000
   254          0.0000
   255          0.0000
 >
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 10, 2018, 08:26:12 AM
Ah, I got the syntax wrong: it requires the word "info" not just a -parameter. Bits and SNR are below:

My bad, sorry for the incomplete command, and thank you for the example outputs.  ;D
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 14, 2018, 04:53:00 PM
Bit of keyboard mashing today and have the graphs coming along.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on June 30, 2018, 10:35:42 AM
So have tried to come up with the most simple way of logging data over time, after getting reacquainted with awk and angry at different line endings:

Code: [Select]
#!/bin/bash

logSize=2879

logFile="/var/tmp/stats/logfile"

statsFile="/var/tmp/stats/varStats"

updateInterval=30


lastFecDown=0
lastFecUp=0
lastCrcDown=0
lastCrcUp=0


while [ 1 ]
do

    xdslctl info --stats > $statsFile

    lineActive=$(awk '/^Status/ {if ($2 == "Showtime") print "true";
                        else print "false";exit}' $statsFile)

    numlines=$(wc -l $logFile | awk '{ print $1 }')
    timeStamp=$(date +"%s")

    if [ $lineActive = "true" ]
    then
        #echo "Got showtime"

        snrValues=$(awk '/^SNR/ {print $3,$4; exit}' $statsFile)
        fecValues=$(awk '/^FEC/ {print $2,$3; exit}' $statsFile)
        crcValues=$(awk '/^CRC/ {print $2,$3; exit}' $statsFile)


        fecDown=$(echo $fecValues | awk '{ print $1 }')
        fecDownDiff=$((fecDown-lastFecDown))
        lastFecDown=$fecDown

        fecUp=$(echo $fecValues | awk -v RS='\r' '{ print $2 }')
        fecUpDiff=$((fecUp-lastFecUp))
        lastFecUp=$fecUp

        crcDown=$(echo $crcValues | awk '{ print $1 }')
        crcDownDiff=$((crcDown-lastCrcDown))
        lastCrcDown=$crcDown

        crcUp=$(echo $crcValues | awk -v RS='\r' '{ print $2 }')
        crcUpDiff=$((crcUp-lastCrcUp))
        lastCrcUp=$crcUp

        if [[ $numlines -gt $logSize ]]
        then
            tail -n +2 $logFile > "${logFile}Tmp" && mv "${logFile}Tmp" $logFile
        fi

        echo "$timeStamp $snrValues $fecValues $crcValues $fecDownDiff $fecUpDiff $crcDownDiff $crcUpDiff" >> $logFile


        sleep $updateInterval

    else

        if [[ $numlines -gt $logSize ]]
        then
            tail -n +2 $logFile > "${logFile}Tmp" && mv "${logFile}Tmp" $logFile
        fi

        echo "$timeStamp NOSYNC" >> $logFile

        sleep $updateInterval
    fi


done
 

Reads xdslctl stats into a file, checks for an active sync then parses out the goodies and writes them to a file. File grows to fixed size, well number of lines - 24hrs of data with 30s readings in this case, then starts removing oldest line and appending new.

Had thought I'd got clever using "sed -i" for inplace removal of the oldest line, but it turns out that it just uses temporary files anyway and its quicker to just use tail and manage the temp file yourself. Would really like to have it do it inplace, maybe something with some meta data at the start of the file with references to the start and end line, maybe something in a real language to make a proper circular buffer. But as this is all in memory copying 100K or so around every 30s doesnt seem like a huge problem.

This still has extraneous data in (the total FEC & CRC rather than just the difference) and a 24hr log sampled every 30s comes in at a smidge over 150K. I have successfully written 10Mbytes to the var/tmp folder on a 1312 with no negative results, so even with the stats server etc there is plenty of room for more monitored values and a longer time period.

Saving old logs to a USB drive is also an easy thing to do, so longer term persistent data could be gathered.

Have run this for 36 hours or so on an 8324 (lots more ram than a 1312) with no ill effects, even while still using DSLStats on another system with the same sample rate. Have just put it together with the stats server and javascript to make the graphs into a firmware and flashed to a 1312 and placed on a live line.

Heres some same output from connection:

Code: [Select]
1528004830 NOSYNC
1528004860 NOSYNC
1528004890 NOSYNC
1528004920 NOSYNC
1528004950 3.6 6.1 1442 0 0 0 1442 0 0 0
1528004981 3.6 6.7 5215 0 0 0 3773 0 0 0
1528005011 3.7 6.8 8394 0 0 0 3179 0 0 0
1528005041 3.7 6.7 12516 0 0 0 4122 0 0 0
1528005071 3.8 6.8 17640 0 0 0 5124 0 0 0
1528005102 3.6 6.8 21211 0 0 0 3571 0 0 0
1528005132 3.8 6.8 26134 0 0 0 4923 0 0 0
1528005162 3.6 6.8 30954 0 0 0 4820 0 0 0
1528005192 3.8 6.7 34950 0 0 0 3996 0 0 0
1528005222 3.6 6.6 38539 0 0 0 3589 0 0 0
1528005252 3.8 6.7 42789 0 0 0 4250 0 0 0
1528005283 3.8 6.7 47399 0 0 0 4610 0 0 0
1528005313 3.8 6.7 51470 0 0 0 4071 0 0 0
1528005343 3.8 6.8 55014 0 0 0 3544 0 0 0
1528005373 3.7 6.7 59714 0 0 0 4700 0 0 0
1528005403 3.8 6.8 64423 0 0 0 4709 0 0 0
1528005434 3.8 6.6 68081 0 0 0 3658 0 0 0
1528005464 3.8 6.8 72278 0 0 0 4197 0 0 0
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on July 03, 2018, 11:39:48 PM
Request: could we make all web output XHTML instead of html, so that I can process the results with an XML parser? Would make it miles easier for the likes of to write tools that filter or query the output because I could pipe the output into an XSLT program.

(There is of course an XHTML dialect corresponding to HTML5, which is up-to-date, as well as the old favourite XHTML 1.0.) There might be some subtle interactions with javascript code in web pages because of the difference between xml and html in this respect.

If you think it is too much pain then I can appreciate that. It would mean running pages through a validator and checking that they are well-formed XML as well as XHTML5 or XHTML1.0.

A possibly bad idea: Some people generate an alternative XML page at a different URL, the Firebrick router does this, I think. Seems a bit wasteful to me because if it is done very cleanly and properly then the page can be easy enough to process, especially if extra id and class values are supplied as ‘hooks’ to help locate elements, and all unnecessary presentational crud is left out and implemented by CSS. The two-files idea is to give the user one that is pretty to look at for eyeballs and another that is lean and mean and easy to parse for machines. But with really good design it should be possible to do both in one to a great extent.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on July 04, 2018, 01:34:58 AM
You're giving me flash backs to a first year web dev course where the exam had a final question along the lines of "now rewrite everything you have done using XHTML and XSL"... nobody was a fan, then I never encountered it again! The lecturer was a fairly old school type, was adamant we all had to sign up to some weird british computing society, made a point of showing him bring up his network interfaces on the CLI after plugging in the projector... I digress.  :D

What benefit does using XML to define webpages really have? Doesn't seem to be in wide use.

All I have at the moment in terms of HTML is a few named canvas elements, all the rest is my nasty excuse for javascript for parsing the output of xdslctl and structuring the data so it works with Chart.js.

Edit: If you mean producing an XML version of the data from the xdslctl output then thats doable I guess.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on July 04, 2018, 02:48:08 AM
XHTML is just XML that uses html elements out of a particular namespace.

All that it means for the author is just writing the source according to the very simple xhtml rules,
* always use utf8,
* lowercase element names,
* always close empty elements with <element /> type syntax eg <hr /> <br />,
* otherwise always having a matching closing tag for every start tag eg <p>blah</p> and never omitting the closing tag </p>,
* it needs <html xmlns="http://www.w3.org/1999/xhtml">
* and it needs to be validated with the w3c's checker really (web-based) and an xml validator (many good web-based ones can be googled) as any authoring mistakes defeat the point of making it machine-readable and xml-parser-friendly.

It is that easy. There are some considerations with javascript that I have not covered here. If you are ok with using the stricter markup syntax then that would be excellent, otherwise don’t worry about it.

Some useful stuff at:
https://www.w3.org/TR/html-polyglot/#writing
https://www.w3.org/TR/xhtml1/guidelines.html

The main advantage is for users. XHTML is far more machine-readable because its syntax is very rigid with no peculiar quirks as in tag soup markup or html5. Users can use XML-speaking tools to process the page, so if a user wants to extract information from it it is so much easier because these tools can find content, strip it down or filter it very conveniently. This is doable because a large range of high quality XML parsing tools are available and the XSLT programming language can be used to read, filter and generate xml and xhtml because xhtml is an application of xml.

Html5 as well as html4 can be written according to the above few simply xhtml syntax authoring rules and then the source is XHTML5 too and so also is automatically XML without anything further being needed.

One benefit to the author is that authoring mistakes can be spotted quickly
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on October 14, 2018, 12:02:52 PM
Have made a VMG1312 firmware with the most basic of SNRM monitoring. It logs SNRM, FEC & CRC numbers every 30 seconds, but also has the output of various xdslctl commands available with the most basic form of an API.

At modem-ip:8000 is the SNRM graph. A GET request to modem-ip:8000/getdata will update files that can be accessed from a browser (or Workflow etc) under modem-ip:8000/data/, just browse there to see a directory listing.

The SNRM graph uses the modems current time vs the browsers to give the correct labels, if you change the time manually or by NTP some time after boot things will get screwy.

I have run various versions of this for months on a VMG8324, and this specific firmware overnight with a live line on a VMG1312.

Apart from the stats server this is vanilla 1312 AAJZ16 with the jumbo frame patches.

https://www.dropbox.com/s/y6f6pcgnwyz8dn4/1312-B10A-AAJZ16-SNRM-log.bin?dl=0
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on October 15, 2018, 12:24:14 AM
Have NOT blown this into a couple of modems. They work fine, but I just get an error if I http-connect to port 8000 decimal. (Not 8080.)

I also tried 0x8000. If I remember correctly, my old boss once invited me round to his house, gave me the address with the house number in hex, without thinking about it. When I got there, I suddenly realised, and then went to the correct house which was #0x10.

I tried Safari and the NetToolbox app’s http test function, and then snooped on the whole conversation using the ‘Charles’ app which is an http-proxy debugger and saw the following response:

method GET
Error code NSPOSIXErrorDomainCode=61 "Connection refused" - error in connect() function

or similar, stupid thing would it let me copy-paste the result.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on October 15, 2018, 06:50:05 PM
Oh thats disappointing!

I have tried it with the zyxel firewall set to high from wired and wireless connections from PC/tablet/phone, so was sure it just "worked".  :(

I did indeed mean port 8000 decimal. A little google of "NSPOSIXErrorDomainCode=61 Connection refused" suggests its just a generic response to being unable to find the host.

If you are game for troubleshooting the first this would be to make sure the server is running on the device. Telnet in and gain real shell with "xtm && sh". The search the running processes for "stats":

Code: [Select]
~ # ps | grep stats
  262 supervis  1148 S    stats-server-bin
  263 supervis  2148 S    /bin/bash /var/tmp/stats/./stats-logging.sh

If it the stats server is running then the next thing to check the right files exist in /var/tmp/stats:

Code: [Select]
~ # cd /var/tmp/stats
/var/tmp/stats # ls
Chart.min.js      getdata           jquery.min.js
data              getstats.sh       moment.min.js
getCurTime.sh     index.html        stats-logging.sh
/var/tmp/stats #

Check the log file for data:

Code: [Select]
/var/tmp/stats # tail data/logfile
1514907468 4.3 6.5 8558 0 0 0
1514907498 4.4 6.4 3752 0 0 0
1514907528 4.3 6.4 6155 0 0 0
1514907559 4.3 6.4 5115 0 0 0
1514907589 4.3 6.4 3528 0 0 0
1514907619 4.3 6.4 2893 0 0 0
1514907649 4.2 6.5 3072 0 0 0
1514907680 4.4 6.5 3377 0 0 0
1514907710 4.3 6.5 2842 0 0 0
1514907740 4.3 6.4 2981 0 0 4
/var/tmp/stats #

Have you tried accessing the data folder from a browser, i.e "192.168.2.1:8000/data" ?

Apologies for supplying something that hasn't worked, and no problem if you'd rather not troubleshoot, I know how tedious it can be.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on October 16, 2018, 12:17:13 AM
No stats process running.

Error connect trying to get access to http://ip:8000/data

No file found data/logfile

I will be more than delighted to do some debugging. A thought: I can send you the horrible mongrel config which I have inherited, if that will help ? Or I can send you a modem, not that there would be much advantage that I can see. Final daft thought, with a lot of faffing about I could perhaps map a routable global IPv4 address so that inbound traffic to that address gets redirected to the modem and that way you could access it directly across the internet.

Another thought. I ought to confirm that I have successfully flashed the modem at all. The whole thing is all a bit weird what with always losing contact in the web UI when reflashing. I ought to rule out some strange cockup that caused the update process to abort.

I was wondering before about altering version strings somehow.

I am trying to think about things that you have working that I don’t - facilities, services etc both due to the config in my modem and due to the characteristics of my network environment.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on October 16, 2018, 12:40:45 AM
If the stats-server-bin is not even running thats where to start.

You make a good point about altering the version string, I'll have to remember where it is to edit and use for further versions.

To make sure it has definitely been flashed you can search the /etc/profile file for the lines that start the server:

Code: [Select]
~ # grep stats /etc/profile
mkdir /var/tmp/stats
mkdir /var/tmp/stats/data
cp /webs/stats-server/* /var/tmp/stats/
chmod +x /var/tmp/stats/getstats.sh
chmod +x /var/tmp/stats/getCurTime.sh
chmod +x /var/tmp/stats/stats-logging.sh
touch /var/tmp/stats/data/logfile
...

Or simply see if "stats-server-bin" is in the PATH:

Code: [Select]
~ # stats-server-bin

I'm pretty sure I still have your default config from a while ago in a PM if it hasn't changed much since then, but as I say, making sure of a successful flash is first on the list.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on October 16, 2018, 02:00:07 AM
Yes, it had failed to blow the image.

All good now.

When I am using the web UI, I get a swirling useless progress indicator (why no proper percentage-done indicator, byte count, clock or growing bar) and that just goes on for ever and ever. Eventually I try hitting refresh. Then nothing much of significance is displayed, just the normal main web page. But it I go back into the upgrade page in the menus, it then immediately says "rebooting" and then no feedback again either, just sits there, so in the end when you do yet another refresh you see the login prompt which shows it has indeed rebooted. Booting still takes an eternity. In fact I know it has booted when an SMS or a Tweet arrives from AA telling me that it has connected to the internet! So anyway, this is how I have managed to screw up upgrades in the past, a lot of guesswork about what is going on. I just need to learn to follow a strict reflashing routine that works.

Logging in to the modem’s usual admin UI programmatically via http might be a bit of a pain, would have to look into it, but if I sorted that out, I could perhaps write a Shortcuts/Workflow program to reliably blow these modems. (I did write a very successful Workflow thing to push a new config into the Firebrick, and optionally tell it to reboot it too.)
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on October 16, 2018, 03:47:57 AM
Ah, well good stuff thats it working now. Version stringed releases from now on!

Wish I could offer some reason that the GUI is such a pain in your setup, but have never experienced it myself. Its not the snappiest by any means, but it never bugs out or hangs on the firmware upload page.

Is the SNRM graph drawing itself ok? Have left it auto ranging as it gives pretty good results with the range my line produces.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on October 16, 2018, 04:06:46 AM
It’s just superb. Just so incredibly useful.

Re: the stupid standard ZyXEL web UI - I wonder if their JavaScript or possibly CSS just doesn’t work in Safari iOS or doesn’t work in machines that don’t have a normal pointing device, or both. The dsl status page thing has the bottom of the report clipped off because it is behind the icon bar that is drawn in front of it at the bottom, and the whole thing refuses to scroll. I am sure some js and css browser behaviour wizard who is familiar with Safari iOS could fix ZyXel’s junk and make it behave properly.

Can I just check - which low-level modem driver is in this new version? I ask because the sync speeds are awful - downstream sync down by 10% - nowadays on the two modems that I have reflashed. But it has to be due to the number of resyncs upsetting DLM or my choice of bad times for resynching, because the speeds were already bad when I had actually failed to reflash the device. There is no way it can be due to this new release.

Just need to think how to get the performance back, and the only thing I can think of is a wise choice of time to resynch, that or more mysterious long naps in the powered-off state.

I have changed the settings, or am trying to, as BTW are taking forever, so that the link’s interleaving setting is ‘on’ (ie max, hopefully) instead of ‘auto’. It’s a feature available in AA’s control server.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on October 16, 2018, 04:19:15 AM
Its the standard default A2pv6F039v adsl_phy. Thats the one you use normally right?

All I can suggest is to wait till you have 24 hours of SNR data and pick an optimum resync time (via an xdslctl configure --snr reduction maybe) and hope for the best. I assume at least the most recent flash and resync was during darkness hours so probably suboptimal.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on October 16, 2018, 04:28:30 AM
> the most recent flash and resync was during darkness hours so probably suboptimal.

Absolutely. That is what I was thinking. I don’t know what else there is, what other factors, that make a difference over time. Why exactly might my crosstalk vary - if indeed it does vary ? Why might rf interference from distant sources vary? Attenuation has changed over time, 0.3dB change to upstream attenuation was noticed by AA staff. So perhaps it’s to do with the electromagnetic environment around the cable, guessing, maybe water present or absent - who knows?
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: dee.jay on October 16, 2018, 08:06:30 AM
I recently installed a VMG1312-B10A on my dad's VDSL line that's connected to an ECI cabinet, hence no G.INP etc.

Would that be of any use to you?
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on October 16, 2018, 10:40:08 PM
And you do not need it?
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: dee.jay on October 17, 2018, 07:38:24 AM
Sorry, I was thinking more about the stats collection from it. It's on his line and going now so can't exactly take it away :)

Got a VMG3925-B10B I'm not sure what to do with though.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: tiffy on October 17, 2018, 04:23:59 PM
You could list it on the BB Hardware Swap Shop forum with view to sell, I have used this a few times without any issues, can't see any of this model listed there at present, last one listed, April this year, went for £30 + P&P as far as I can see.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 16, 2020, 11:18:05 PM
Sorry for posting in this old thread however my question is to do with the Stats system.

Is there away to get this to work externally. I have setup the port forwarding for the required port and if i type in the URL on laptop it works without issue however no one externally can view it at all.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on March 16, 2020, 11:49:00 PM
By ‘externally’, you mean remotely, over DSL? Or just stats at a public, global, routable IP address. If the latter, then the modem config can be set up with whatever admin IP address you like - I just happen to have it set to 192.168.n.1 where n is 1,2,3,4 for each one of my four modems.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on March 17, 2020, 12:09:44 AM
Sorry for posting in this old thread however my question is to do with the Stats system.

Is there away to get this to work externally. I have setup the port forwarding for the required port and if i type in the URL on laptop it works without issue however no one externally can view it at all.

I fear you mean allowing the stats server to be accessed from the internet at your IP address. I would strongly advise against this as I would for the regular Zyxel GUI interface.

Bots are bots, and open ports get hammered. The Zyxel interface is years out of date and probably has large holes in, and while the mongoose web server that supplies stats from my firmware builds is meant for IoT devices, I cant guarantee its security either.

If you want remote access to stats I would recommend a VPN server (wireguard is fantastic) that you connect to when not on your home network.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on March 17, 2020, 12:47:02 AM
What Johnson said. That’s a bad idea as you have no control over it. If you really need to access it remotely then configure the modem with an admin IP address that is global/public/routable which means having an ISP that is capable of offering appropriate addressing, and hr. make sure to firewall it appropriately so that only the people you want to be able to access it can do so.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 12:56:11 AM
Oops, sorry didn't make it clear, I am not after the Main WebGUI of the zyxel to be on internet, I am after getting port 8000 Statistics page to be on internet as I am using your custom firmware with the mongoose web server for statistics.

It works internally via my DDNS URL however doesn't work for external connections.

I wouldn't forward port the main WebGUI or even telnet or SSH over the internet as I know it's not wise.


Sent from my SNE-LX1 using Tapatalk
(https://uploads.tapatalk-cdn.com/20200317/722805248239017db876fed5a5325f7b.jpg)
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on March 17, 2020, 01:10:16 AM
Your router might be able to handle redirecting that ?
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 01:11:15 AM
Your router might be able to handle redirecting that ?

As I can't PM you it, will have to be vague on setup but hopefully enough for you to get the jist.

Basically I setup the port forwarding like I did.

Now if I use the LAN side and use the DDNS URL I have it works

Firewall/Portforward is set like

External Port: 12345 (Not the port being used)

Internal Port: 8000
Source: WAN
Destination: LAN (IP address to modem)
Protocol: TCP
Hash limit: 60/minute

Now with this setup which was used on my Raspi 3B+ using DSLStats it worked fine however I can't get any external connections to work for the Zyxel mongoose server stats page.

So reason I was asking was just incase mongoose needed more setup or the VMG1312 needed a rule added like the one needed originally for dslstats to work where grouping was required.

Idea is to try and get it working and then when I have issues I can enable the rule so the ISP can see what's going on or if I am heading out for a bit I can have it enabled to keep an eye out on the line.


Sent from my SNE-LX1 using Tapatalk
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on March 17, 2020, 02:09:09 AM
Oh ok, I still wouldn't recommend exposing the mongoose server to the internet, but if its on a different port and you know what you are doing then fill your boots.

I have no experience with punching holes in the zyxel firewall, maybe try exposing 80 for a short time to test?

I assume you are using the zyxel as a full router? If you are using it as a modem with another router there will need to be another layer of jiggerypokery.

Edit: on second read I see it clearly is as a modem, and you are talking about your router settings... I'm scratching my head why that doesnt work.  :-\
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 02:19:39 AM
I am using the Zyxel as a Modem in bridged mode with my Linksys WRT32x with OpenWRT (Custom build / David502) firmware.

Now when i did the DSL-Stats on Raspi 3B+ i used the same settings above but with the address in the signature and i had no issue whats so ever getting it to work externally. However i can't for the love of me get it to work with Modem stats on port 8000, even having it set to port 8000 for source and dest on DNAT it still doesn't work.

Once i have it working it will be only enabled when either ISP needs to see and save the info or when i need to monitor whilst away else it will be off.

I always used different ports so that the internet being the internet will not cause that much of an issue if they try bouncing on SSH / TELNET / HTTP ports etc
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on March 17, 2020, 02:33:18 AM
Yeah sorry for the confusion.

I honestly cant offer a reason for something accessible on the lan with a properly configured port forward on the router not responding.

Do you have any other systems/services on the network that could be exposed (on a suitably random port) that you can try to connect to from an external connection? Just to prove that those are accessible with the same settings on the router.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 02:43:10 AM
Yes basically anything that connects directly to the router LAN port's / USB (using it for webserver), works without issue. However anything thats on the Zyxel to the router is not working. Messing with the iptables now with DNAT and SNAT on modem results in the same issue.

I am using Eth1 on modem to WAN on Router and using Eth4 on modem to Eth1 on Router with the Interface Group ptm0.1 lan1 setup on modem (which was needed for dslstats to work thru router to modem).

in fact if you remove the port from the address on my signature you'll get a basic webpage stating the website your looking for is closed and its being used as a nameserver.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on March 17, 2020, 02:49:48 AM
Is the Eth4 on modem interface on the same subnet as the rest of your LAN? Or are you using SNAT trickery to get through to it?
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 02:50:57 AM
Not using SNAT trickery, both modem and router are on same IP range.

Just had a thought, could it be that the mongoose web server setup be taking the modems up as the listening address instead of 0.0.0.0 (which would make it so external iOS could access it).
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on March 17, 2020, 03:45:35 AM
So I had a play and cant get a zyxel accessible from a remote IP at all myself.

I'm thinking its probably due to a modem mode zyxel not having a default gateway etc, no access to outside of the lan.

Will fiddle more tomorrow.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 03:46:23 AM
Thanks bud, setting up wireguard for now though.

Sent from my SNE-LX1 using Tapatalk

Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: johnson on March 17, 2020, 03:49:38 AM
Thanks bud, setting up wireguard for now though.

Will find a more optimal solution to occasional remote access to the modem, but having a wireguard setup to remote into your network from any location is never a bad thing!  :D
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 03:54:49 AM
Will find a more optimal solution to occasional remote access to the modem, but having a wireguard setup to remote into your network from any location is never a bad thing!  :D
That is very true, however thank you for your time and assistance. Will let you get back to what you was doing.

Update: I tried Wireguard and although I could pull up the router WebGUI I couldn't pull up the modem WebGUI so really stumped.

Sent from my SNE-LX1 using Tapatalk
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on March 17, 2020, 05:46:16 PM
Sincerest apology if you know all this already: Being in the same IP address range doesn’t mean you’re automatically in the same subnet - definition = “broadcast domain”. In my case the main LAN broadcast domain has an 81.187.x.y public/global/routable address in a /26 and each modem’s admin i/f lives in its own broadcast domain at 192.168.n.1 where n=1,2,3,4 is the nth modem (I have four modems). So that’s 1+4=5 broadcast domains=5 subsets, and in my case they are all distinct non-overlapping address ranges, but that’s just coincidental, it’s down to how I configured the modems.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 05:51:57 PM
Sincerest apology if you know all this already: Being in the same IP address range doesn’t mean you’re automatically in the same subnet - definition = “broadcast domain”. In my case the main LAN broadcast domain has an 81.187.x.y public/global/routable address in a /26 and each modem’s admin i/f lives in its own broadcast domain at 192.168.n.1 where n=1,2,3,4 is the nth modem (I have four modems). So that’s 1+4=5 broadcast domains=5 subsets, and in my case they are all distinct non-overlapping address ranges, but that’s just coincidental, it’s down to how I configured the modems.
It's always a great thing to relearn so no need to apologise beside it might help others.

However both modem and router are in same ip range of 192.168.1.0/32 I have even tried 192.168.1.0/24 however It did just remind me my Guest network has been setup on another range with firewall rules set to block WebGUI access and that's on the same vlan going to wan.

Sent from my SNE-LX1 using Tapatalk

Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on March 17, 2020, 06:07:46 PM
I don’t understand your comment about "/32". That’s not large enough to have two IP addresses in it. Surely the modem and the router have to have two distinct IP addresses in their subnet? How does the modem manage if an incoming packet has src_address==dest_address, as when it replies to incoming packets it will be replying to itself? So I don’t see how the modem can be in a subnet that is /32. Your main LAN will typically be in a /24 or /16 if it’s in the usual home user setup with RFC1918 NAT-ed IP addresses?

Each of my modems lives in its own /24, as it happens, and in each subnet the modem is 192.168.n.1 and the router is 192.168.n.254.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 06:18:04 PM
I have Modem on 192.168.1.1 /255.255.255.0 and router on 192.168.1.3 /255.255.255.0 with guest network on 10.0.1.1 / 255.255.0.0 .

Guest network and normal network are on same vlan however guest network is isolated from the main network via firewall rules so they can't access webgui's or main network systems.

Hopefully this helps clear things up.

Sent from my SNE-LX1 using Tapatalk

Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on March 17, 2020, 06:42:25 PM
Right. Some confusion earlier then, as you have a /24 there given it is 255.255.255.0, not a /32.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: hacktrix2006 on March 17, 2020, 06:43:56 PM
Sorry about that networking can be confusing at times lol. Hold up I had wireguard setup up to /32 and not /24 so that might be why it didn't work with wireguard lol. Will mess again later thanks for your time.

Sent from my SNE-LX1 using Tapatalk

Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on May 12, 2020, 05:23:57 PM
Question: on the CRC and FEC const vs time graphs, what are the units on the verticaly axis? I have modem #4 running very very slowly, with a lot of noise and the downstream CRC count is high compared to the other modems. The CRC rate figure shown on the y axis is the count CRC of corrected errors[?] per unit what time[?]

It is a still a new performance nightmare at the moment. The downstream SNRM was 2.5 dB and I forced a resync which brought the d/s SNRM up to 3.1dB. The downstream rate is even slower now, with a downstream sync rate below 2.5Mbps, whereas it was  ~2.9 Mbps before, and in total I have lost about 0.8-1.0Mbps total effective download rate from speed tests with speedtester2.aa.net.uk compared to what it was like a couple of months ago.

Live sync rates:
  #1: down 2807 kbps, up 519 kbps
  #2: down 2700 kbps, up 544 kbps
  #3: down 2964 kbps, up 389 kbps
  #4: down 2457 kbps, up 515 kbps

Modem #4 CRC count vs time graph:

(https://i.postimg.cc/43pJyxQD/C3-F02654-AE06-4-B72-BCB3-84-DB0608-F858.jpg)

Modem #4 stats:

xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason:   8000
Last initialization procedure status:   0
Max:   Upstream rate = 436 Kbps, Downstream rate = 2708 Kbps
Bearer:   0, Upstream rate = 515 Kbps, Downstream rate = 2457 Kbps

Link Power State:   L0
Mode:         ADSL2 Annex A
TPS-TC:         ATM Mode(0x0)
Trellis:      U:ON /D:ON
Line Status:      No Defect
Training Status:   Showtime
      Down      Up
SNR (dB):    3.1       6.0
Attn(dB):    65.0       41.0
Pwr(dBm):    18.4       12.4

         ADSL2 framing
         Bearer 0
MSGc:      52      11
B:      28      62
M:      8      1
T:      3      1
R:      16      12
S:      2.9745      3.8462
L:      667      156
D:      1      8

         Counters
         Bearer 0
SF:      2803623      754342
SFErr:      2593      10
RS:      60978814      473909
RSCorr:      201383      13964
RSUnCorr:   2733      0

ReXmt:      464652      0
ReXmtCorr:   464413      0
ReXmtUnCorr:   2884      0

         Bearer 0
HEC:      3616      17
OCD:      0      0
LCD:      0      0
Total Cells:   263857767      55806956
Data Cells:   5188622      1176787
Drop Cells:   0
Bit Errors:   167670      1432

ES:      219901      278
SES:      121      8
UAS:      4933      4825
AS:      45884

         Bearer 0
INP:      26.00      2.00
INPRein:   0.00      0.00
delay:      8      8
PER:      16.17      16.34
OR:      28.68      8.32
AgR:      2475.44   522.12

Bitswap:   5896/5896      105/105

Total time = 33 days 4 hours 1 min 23 sec
FEC:      21598882      717266
CRC:      347108      447
ES:      219901      278
SES:      121      8
UAS:      4933      4825
LOS:      12      2
LOF:      79      2
LOM:      8      0
Latest 15 minutes time = 1 min 23 sec
FEC:      348      11
CRC:      8      0
ES:      7      0
SES:      0      0
UAS:      0      0
LOS:      0      0
LOF:      0      0
LOM:      0      0
Previous 15 minutes time = 15 min 0 sec
FEC:      3625      274
CRC:      34      0
ES:      32      0
SES:      0      0
UAS:      0      0
LOS:      0      0
LOF:      0      0
LOM:      0      0
Latest 1 day time = 4 hours 1 min 23 sec
FEC:      57112      5415
CRC:      661      8
ES:      631      8
SES:      0      0
UAS:      0      0
LOS:      0      0
LOF:      0      0
LOM:      0      0
Previous 1 day time = 24 hours 0 sec
FEC:      688715      22265
CRC:      9424      4
ES:      8006      4
SES:      0      0
UAS:      54      54
LOS:      0      0
LOF:      0      0
LOM:      0      0
Since Link time = 12 hours 44 min 43 sec
FEC:      201383      13964
CRC:      2593      10
ES:      2252      10
SES:      0      0
UAS:      0      0
LOS:      0      0
LOF:      0      0
LOM:      0      0
NTR: mipsCntAtNtr=0 ncoCntAtNtr=0
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: burakkucat on May 12, 2020, 07:05:59 PM
Question: on the CRC and FEC const vs time graphs, what are the units on the vertical yaxis?

Number of events per minute. I.e. number of CRCs per minute and number of FECs per minute.

Quote
The CRC rate figure shown on the y axis is the count CRC of corrected errors[?] per unit what time[?]

No, not corrected errors but actual CRC errors recorded in the immediate, preceding minute.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: Weaver on May 12, 2020, 07:17:21 PM
Thank you I take it that FEC count is the count of error events corrected by RS ?

So it’s per minute. Where does L2 RETX come in in this count? Is PHY irrelevant here, if retransmission is required then it goes ahead and either succeeds or not but isn’t shown in these particular figures in this graph?

In the raw stats figures, are those first numbers under bearer 0 etc : ReXMT etc - are those just cumulative totals? Those in the first set just up and up? As opposed to being counts per some unit time interval? Which one sees further down in the various time interval sections.
Title: Re: Serving (slightly) better stats directly from Zyxel modems
Post by: burakkucat on May 12, 2020, 07:53:49 PM
Thank you I take it that FEC count is the count of error events corrected by RS ?

Yes, RS and any other "goodness" that was capable of making the correction

Quote
Where does L2 RETX come in in this count? Is PHY irrelevant here, if retransmission is required then it goes ahead and either succeeds or not but isn’t shown in these particular figures in this graph?

Sorry, that is something I do not know.

Quote
In the raw stats figures, are those first numbers under bearer 0 etc : ReXMT etc - are those just cumulative totals? Those in the first set just up and up? As opposed to being counts per some unit time interval? Which one sees further down in the various time interval sections.

They are cumulative totals. The delta is created by subtracting the previous cumulative total (at time n - 1 minutes) from the current cumulative total (at time n minutes).