Kitz Forum
Broadband Related => Broadband Hardware => Topic started by: johnson on June 07, 2018, 01:04:47 AM
-
Thought I should split this off from the other thread talking about firmware as its so off topic.
Have got a simple webserver compiled and working from boot on a 1312, uses the relatively small mongoose (https://github.com/cesanta/mongoose) server (160K) with the simplest possible setup, running a script to gather stats and then serving a directory under /var/tmp (from ram).
#include "mongoose.h"
static const char *s_http_port = "8000";
static struct mg_serve_http_opts s_http_server_opts;
static void ev_handler(struct mg_connection *nc, int ev, void *p) {
if (ev == MG_EV_HTTP_REQUEST) {
system("/var/tmp/stats/./getstats.sh");
mg_serve_http(nc, (struct http_message *) p, s_http_server_opts);
}
}
int main(void) {
struct mg_mgr mgr;
struct mg_connection *nc;
mg_mgr_init(&mgr, NULL);
printf("Starting web server on port %s\n", s_http_port);
nc = mg_bind(&mgr, s_http_port, ev_handler);
if (nc == NULL) {
printf("Failed to create listener\n");
return 1;
}
// Set up HTTP server parameters
mg_set_protocol_http_websocket(nc);
s_http_server_opts.document_root = "/var/tmp/stats/"; // Serve current directory
s_http_server_opts.enable_directory_listing = "yes";
for (;;) {
mg_mgr_poll(&mgr, 1000);
}
mg_mgr_free(&mgr);
return 0;
}
Arbitrary commands can be placed in "getstats.sh" and their outputs are accessible on routerip:8000.
Not any better than the xDSL Statistics section of the normal GUI except for being able to gather SNR/QLN etc. These could be made into pretty graphs client side, but taking a stab with 5 minutes of googling javascript plotting has only resulted in ugly graphs covered in tooltips and other pointless stuff. If anyone knows of an ultra lightweight plotting library for js that would be great.
This is in no way meant to even remotely compete with the other excellent stats monitoring tools available, but I though it would be nice to be able to glance at things like SNR per tone, bitloading etc from a URL served by the modem itself. I guess a small log of other data could work, like the last 24 hours SNR graph or similar.
Totally stupid idea? Probably! But its fun to play with these things.
-
Good work. :)
-
Totally stupid idea? Probably! But its fun to play with these things.
No, not stupid by any means. Quite an interesting idea.
What might be worthwhile would be to "serve up" the data used to create the four snapshot plots (Bit Loading, Hlog, QLN & SNR) and the data returned by "info --pbParams" & "info --stats".
-
Got to messing with chart.js this evening and have some reasonable looking graphs, was starting to try and implement a nice way of choosing ranges, pbParams should work well for selected a sensible range as well separating upload and download sections of SNR/bitloading. But it occurred to me, does pbParams even work with ADSL? How about lines without G.INP, are they missing the whole "Bearer 1" line? If so just using offsets to get to the right data isnt going to work.
So I guess I could go digging through posts to find examples of xdslctl output for various lines and probably should. But can anyone with an ADSL line or VDSL without G.INP provide a few outputs? E.g. --pbParams, the first 10 or so lines of --SNR or --Bits or --Hlog (I assume they are the same).
Much appreciated if anyone can be bothered! :D
-
pbParams does not work on my ADSL line that has no G.INP (but does we think have Broadcom PhyR their proprietary equivalent)
-
For the various ADSL flavours there are only two sets of tones - a range for upstream starting at near to 0, and the remainder for downstream starting at different points depending on the spcific protocol. Look at this article (https://en.wikipedia.org/wiki/Asymmetric_digital_subscriber_line#ADSL_standards) for the details. In the UK only Annex A and Annex M need to be considered.
-
Input from the maestro! Thanks for the the links and info, sure I can concoct some ADSL detection and apply the right ranges then.
OT, but I'd really like to thank you for DSLstats, its an invaluable tool.
-
https://forum.kitz.co.uk/index.php?topic=20691.0
-
pbParams does not work on my ADSL line that has no G.INP (but does we think have Broadcom PhyR their proprietary equivalent)
Thank you!
The output of xdslctl --SNR or --Bits would still be useful Weaver, only the first 10 or so lines down to where the tones/values start.
-
@johnson no luck with xdslctl --SNR or -Bits on my VMG 1312-B10A with your second firmware - jumbo + updated low-level driver BLOB.
-
Here follows my list of "useful commands" for a VMG1312-B10A device --
xdslctl info --state
xdslctl info --show
xdslctl info --stats
xdslctl info --SNR
xdslctl info --QLN
xdslctl info --Hlog
xdslctl info --Hlin
xdslctl info --HlinS
xdslctl info --Bits
xdslctl info --pbParams
xdslctl info --linediag
xdslctl info --linediag1
xdslctl info --vendor
xdslctl info --cfg
xdslctl info --webstats
xdslctl info --vectoring
xdslctl profile --show
xdslctl --version
I have deliberately emboldened two of the lines . . . ;)
-
Ah, I got the syntax wrong: it requires the word "info" not just a -parameter. Bits and SNR are below:
VMG1312-B10A
Login: admin
Password:
> xdslctl info --Bits
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 440 Kbps, Downstream rate = 3152 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps
Tone number Bit Allocation
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 8
8 9
9 10
10 10
11 9
12 9
13 9
14 9
15 8
16 8
17 8
18 8
19 7
20 7
21 7
22 7
23 7
24 6
25 6
26 6
27 6
28 5
29 5
30 5
31 4
32 0
33 8
34 8
35 9
36 10
37 10
38 11
39 11
40 11
41 11
42 11
43 11
44 11
45 11
46 11
47 11
48 11
49 11
50 2
51 11
52 11
53 11
54 11
55 11
56 11
57 11
58 11
59 11
60 11
61 11
62 11
63 11
64 11
65 11
66 11
67 10
68 10
69 11
70 10
71 10
72 10
73 10
74 8
75 9
76 9
77 10
78 9
79 10
80 10
81 9
82 9
83 9
84 9
85 9
86 9
87 9
88 9
89 8
90 8
91 8
92 8
93 8
94 8
95 8
96 7
97 8
98 7
99 7
100 7
101 7
102 7
103 7
104 7
105 7
106 6
107 6
108 6
109 6
110 6
111 6
112 6
113 6
114 5
115 5
116 5
117 5
118 5
119 5
120 4
121 4
122 4
123 3
124 0
125 1
126 0
127 2
128 2
129 3
130 3
131 3
132 3
133 3
134 3
135 3
136 3
137 3
138 3
139 3
140 3
141 2
142 2
143 2
144 2
145 2
146 1
147 2
148 2
149 1
150 1
151 1
152 1
153 1
154 1
155 0
156 0
157 0
158 0
159 0
160 0
161 0
162 0
163 0
164 0
165 0
166 0
167 0
168 0
169 0
170 0
171 0
172 0
173 0
174 0
175 0
176 0
177 0
178 0
179 0
180 0
181 0
182 0
183 0
184 0
185 0
186 0
187 0
188 0
189 0
190 0
191 0
192 0
193 0
194 0
195 0
196 0
197 0
198 0
199 0
200 0
201 0
202 0
203 0
204 0
205 0
206 0
207 0
208 0
209 0
210 0
211 0
212 0
213 0
214 0
215 0
216 0
217 0
218 0
219 0
220 0
221 0
222 0
223 0
224 0
225 0
226 0
227 0
228 0
229 0
230 0
231 0
232 0
233 0
234 0
235 0
236 0
237 0
238 0
239 0
240 0
241 0
242 0
243 0
244 0
245 0
246 0
247 0
248 0
249 0
250 0
251 0
252 0
253 0
254 0
255 0
> xdslctl info --QLN
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 440 Kbps, Downstream rate = 3144 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps
Tone number QLN
0 -150.5000
1 -150.5000
2 -150.5000
3 -150.5000
4 -150.5000
5 -150.5000
6 -150.5000
7 -150.5000
8 -150.5000
9 -150.5000
10 -150.5000
11 -150.5000
12 -150.5000
13 -150.5000
14 -150.5000
15 -150.5000
16 -150.5000
17 -150.5000
18 -150.5000
19 -150.5000
20 -150.5000
21 -150.5000
22 -150.5000
23 -150.5000
24 -150.5000
25 -150.5000
26 -150.5000
27 -150.5000
28 -150.5000
29 -150.5000
30 -150.5000
31 -150.5000
32 -150.5000
33 -124.0000
34 -127.5000
35 -130.0000
36 -132.0000
37 -133.0000
38 -135.0000
39 -136.0000
40 -136.0000
41 -136.5000
42 -136.5000
43 -136.5000
44 -137.5000
45 -137.0000
46 -137.0000
47 -137.5000
48 -137.5000
49 -137.5000
50 -137.5000
51 -138.0000
52 -138.5000
53 -138.0000
54 -138.5000
55 -139.0000
56 -140.0000
57 -139.0000
58 -137.5000
59 -138.5000
60 -140.0000
61 -139.5000
62 -140.5000
63 -140.5000
64 -141.0000
65 -141.0000
66 -141.5000
67 -141.5000
68 -142.0000
69 -143.0000
70 -142.5000
71 -142.5000
72 -142.0000
73 -143.0000
74 -137.5000
75 -141.5000
76 -142.5000
77 -143.5000
78 -144.5000
79 -145.5000
80 -145.5000
81 -145.5000
82 -145.0000
83 -145.0000
84 -145.5000
85 -146.5000
86 -146.0000
87 -146.5000
88 -146.0000
89 -146.5000
90 -146.0000
91 -146.5000
92 -146.5000
93 -146.5000
94 -147.0000
95 -146.5000
96 -147.0000
97 -148.0000
98 -147.0000
99 -148.5000
100 -148.0000
101 -147.5000
102 -148.0000
103 -147.5000
104 -147.0000
105 -148.0000
106 -148.0000
107 -148.0000
108 -148.0000
109 -148.0000
110 -147.5000
111 -148.5000
112 -148.0000
113 -148.0000
114 -148.0000
115 -148.5000
116 -148.0000
117 -148.5000
118 -148.5000
119 -147.5000
120 -147.5000
121 -148.0000
122 -146.5000
123 -144.5000
124 -143.0000
125 -137.5000
126 -136.5000
127 -143.5000
128 -145.0000
129 -145.0000
130 -146.0000
131 -147.5000
132 -148.0000
133 -148.5000
134 -149.0000
135 -149.0000
136 -148.0000
137 -148.0000
138 -148.5000
139 -149.0000
140 -149.5000
141 -149.5000
142 -149.5000
143 -149.5000
144 -149.0000
145 -149.5000
146 -149.0000
147 -149.5000
148 -149.5000
149 -150.0000
150 -149.5000
151 -149.0000
152 -149.5000
153 -149.5000
154 -149.0000
155 -149.5000
156 -150.0000
157 -149.5000
158 -149.5000
159 -149.5000
160 -149.0000
161 -148.5000
162 -149.0000
163 -149.0000
164 -149.5000
165 -149.0000
166 -149.5000
167 -149.0000
168 -149.5000
169 -149.5000
170 -150.0000
171 -149.0000
172 -149.5000
173 -148.5000
174 -150.0000
175 -149.5000
176 -149.5000
177 -148.5000
178 -149.5000
179 -149.5000
180 -149.5000
181 -149.5000
182 -149.5000
183 -149.5000
184 -149.0000
185 -150.0000
186 -149.0000
187 -148.0000
188 -142.5000
189 -149.0000
190 -150.0000
191 -149.5000
192 -149.0000
193 -149.0000
194 -149.0000
195 -149.0000
196 -147.5000
197 -149.0000
198 -149.0000
199 -149.0000
200 -149.5000
201 -148.5000
202 -148.5000
203 -149.0000
204 -148.0000
205 -148.5000
206 -148.0000
207 -148.5000
208 -149.0000
209 -147.0000
210 -148.5000
211 -147.5000
212 -148.0000
213 -149.5000
214 -150.0000
215 -149.0000
216 -146.5000
217 -148.0000
218 -149.0000
219 -148.5000
220 -148.5000
221 -149.5000
222 -149.5000
223 -149.5000
224 -148.0000
225 -148.5000
226 -146.0000
227 -148.5000
228 -149.5000
229 -149.0000
230 -149.0000
231 -148.5000
232 -148.0000
233 -149.0000
234 -148.0000
235 -147.5000
236 -144.5000
237 -149.5000
238 -149.0000
239 -147.5000
240 -149.5000
241 -149.0000
242 -148.5000
243 -149.0000
244 -149.0000
245 -148.5000
246 -147.5000
247 -149.0000
248 -149.0000
249 -146.5000
250 -148.0000
251 -150.0000
252 -149.5000
253 -149.5000
254 -148.0000
255 -148.5000
> xdslctl info --pbParams
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 440 Kbps, Downstream rate = 3148 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps
Currently not in VDSL modulation --pbParams is only for VDSL mode
> xdslctl info --SNR
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 440 Kbps, Downstream rate = 3152 Kbps
Bearer: 0, Upstream rate = 496 Kbps, Downstream rate = 2861 Kbps
Tone number SNR
0 0.0000
1 0.0000
2 0.0000
3 0.0000
4 0.0000
5 0.0000
6 0.0000
7 0.0000
8 0.0000
9 0.0000
10 0.0000
11 0.0000
12 0.0000
13 0.0000
14 0.0000
15 0.0000
16 0.0000
17 0.0000
18 0.0000
19 0.0000
20 0.0000
21 0.0000
22 0.0000
23 0.0000
24 0.0000
25 0.0000
26 0.0000
27 0.0000
28 0.0000
29 0.0000
30 0.0000
31 0.0000
32 0.0000
33 26.8750
34 29.5625
35 32.1875
36 34.3750
37 35.6250
38 36.6250
39 37.3125
40 37.8125
41 38.2500
42 38.5625
43 38.7500
44 39.0000
45 39.1875
46 38.4375
47 39.3125
48 39.6875
49 39.4375
50 37.4375
51 39.5000
52 39.2500
53 39.1875
54 39.2500
55 38.7500
56 38.4375
57 38.2500
58 37.1875
59 36.8125
60 37.2500
61 37.5000
62 37.3750
63 36.8750
64 37.0625
65 36.9375
66 36.6875
67 35.7500
68 35.8750
69 36.5625
70 36.3750
71 35.6250
72 34.3125
73 34.5625
74 28.6875
75 33.0000
76 32.6875
77 34.3750
78 33.8125
79 33.8750
80 33.5625
81 33.1250
82 32.6250
83 32.2500
84 32.2500
85 32.0000
86 31.3750
87 30.8750
88 30.7500
89 30.0625
90 30.3125
91 29.8750
92 29.4375
93 29.1875
94 28.9375
95 28.4375
96 28.0000
97 27.7500
98 27.2500
99 26.8125
100 26.6250
101 26.1875
102 26.0625
103 25.6875
104 24.9375
105 24.9375
106 24.4375
107 24.0000
108 23.5625
109 23.0625
110 22.8750
111 22.4375
112 22.0000
113 21.5625
114 20.5625
115 20.8125
116 20.3750
117 19.7500
118 19.6250
119 19.1875
120 17.5000
121 17.1875
122 16.7500
123 13.4375
124 7.1875
125 8.3750
126 8.2500
127 11.7500
128 10.1250
129 14.3125
130 13.1875
131 14.6250
132 13.8750
133 13.8750
134 13.3125
135 13.5625
136 13.1250
137 12.9375
138 12.5625
139 12.2500
140 12.0000
141 11.5625
142 11.7500
143 11.0625
144 11.1250
145 10.6875
146 10.5625
147 10.0000
148 9.6250
149 9.5625
150 9.6250
151 9.1250
152 9.0000
153 8.5625
154 8.3125
155 0.7500
156 8.2500
157 7.6875
158 7.3750
159 7.1250
160 6.8125
161 4.9375
162 0.0000
163 0.0000
164 0.0000
165 0.0000
166 0.0000
167 0.0000
168 0.0000
169 0.0000
170 0.0000
171 0.0000
172 0.0000
173 0.0000
174 0.0000
175 0.0000
176 0.0000
177 0.0000
178 0.0000
179 0.0000
180 0.0000
181 0.0000
182 0.0000
183 0.0000
184 0.0000
185 0.0000
186 0.0000
187 0.0000
188 0.0000
189 0.0000
190 0.0000
191 0.0000
192 0.0000
193 0.0000
194 0.0000
195 0.0000
196 0.0000
197 0.0000
198 0.0000
199 0.0000
200 0.0000
201 0.0000
202 0.0000
203 0.0000
204 0.0000
205 0.0000
206 0.0000
207 0.0000
208 0.0000
209 0.0000
210 0.0000
211 0.0000
212 0.0000
213 0.0000
214 0.0000
215 0.0000
216 0.0000
217 0.0000
218 0.0000
219 0.0000
220 0.0000
221 0.0000
222 0.0000
223 0.0000
224 0.0000
225 0.0000
226 0.0000
227 0.0000
228 0.0000
229 0.0000
230 0.0000
231 0.0000
232 0.0000
233 0.0000
234 0.0000
235 0.0000
236 0.0000
237 0.0000
238 0.0000
239 0.0000
240 0.0000
241 0.0000
242 0.0000
243 0.0000
244 0.0000
245 0.0000
246 0.0000
247 0.0000
248 0.0000
249 0.0000
250 0.0000
251 0.0000
252 0.0000
253 0.0000
254 0.0000
255 0.0000
>
-
Ah, I got the syntax wrong: it requires the word "info" not just a -parameter. Bits and SNR are below:
My bad, sorry for the incomplete command, and thank you for the example outputs. ;D
-
Bit of keyboard mashing today and have the graphs coming along.
-
So have tried to come up with the most simple way of logging data over time, after getting reacquainted with awk and angry at different line endings:
#!/bin/bash
logSize=2879
logFile="/var/tmp/stats/logfile"
statsFile="/var/tmp/stats/varStats"
updateInterval=30
lastFecDown=0
lastFecUp=0
lastCrcDown=0
lastCrcUp=0
while [ 1 ]
do
xdslctl info --stats > $statsFile
lineActive=$(awk '/^Status/ {if ($2 == "Showtime") print "true";
else print "false";exit}' $statsFile)
numlines=$(wc -l $logFile | awk '{ print $1 }')
timeStamp=$(date +"%s")
if [ $lineActive = "true" ]
then
#echo "Got showtime"
snrValues=$(awk '/^SNR/ {print $3,$4; exit}' $statsFile)
fecValues=$(awk '/^FEC/ {print $2,$3; exit}' $statsFile)
crcValues=$(awk '/^CRC/ {print $2,$3; exit}' $statsFile)
fecDown=$(echo $fecValues | awk '{ print $1 }')
fecDownDiff=$((fecDown-lastFecDown))
lastFecDown=$fecDown
fecUp=$(echo $fecValues | awk -v RS='\r' '{ print $2 }')
fecUpDiff=$((fecUp-lastFecUp))
lastFecUp=$fecUp
crcDown=$(echo $crcValues | awk '{ print $1 }')
crcDownDiff=$((crcDown-lastCrcDown))
lastCrcDown=$crcDown
crcUp=$(echo $crcValues | awk -v RS='\r' '{ print $2 }')
crcUpDiff=$((crcUp-lastCrcUp))
lastCrcUp=$crcUp
if [[ $numlines -gt $logSize ]]
then
tail -n +2 $logFile > "${logFile}Tmp" && mv "${logFile}Tmp" $logFile
fi
echo "$timeStamp $snrValues $fecValues $crcValues $fecDownDiff $fecUpDiff $crcDownDiff $crcUpDiff" >> $logFile
sleep $updateInterval
else
if [[ $numlines -gt $logSize ]]
then
tail -n +2 $logFile > "${logFile}Tmp" && mv "${logFile}Tmp" $logFile
fi
echo "$timeStamp NOSYNC" >> $logFile
sleep $updateInterval
fi
done
Reads xdslctl stats into a file, checks for an active sync then parses out the goodies and writes them to a file. File grows to fixed size, well number of lines - 24hrs of data with 30s readings in this case, then starts removing oldest line and appending new.
Had thought I'd got clever using "sed -i" for inplace removal of the oldest line, but it turns out that it just uses temporary files anyway and its quicker to just use tail and manage the temp file yourself. Would really like to have it do it inplace, maybe something with some meta data at the start of the file with references to the start and end line, maybe something in a real language to make a proper circular buffer. But as this is all in memory copying 100K or so around every 30s doesnt seem like a huge problem.
This still has extraneous data in (the total FEC & CRC rather than just the difference) and a 24hr log sampled every 30s comes in at a smidge over 150K. I have successfully written 10Mbytes to the var/tmp folder on a 1312 with no negative results, so even with the stats server etc there is plenty of room for more monitored values and a longer time period.
Saving old logs to a USB drive is also an easy thing to do, so longer term persistent data could be gathered.
Have run this for 36 hours or so on an 8324 (lots more ram than a 1312) with no ill effects, even while still using DSLStats on another system with the same sample rate. Have just put it together with the stats server and javascript to make the graphs into a firmware and flashed to a 1312 and placed on a live line.
Heres some same output from connection:
1528004830 NOSYNC
1528004860 NOSYNC
1528004890 NOSYNC
1528004920 NOSYNC
1528004950 3.6 6.1 1442 0 0 0 1442 0 0 0
1528004981 3.6 6.7 5215 0 0 0 3773 0 0 0
1528005011 3.7 6.8 8394 0 0 0 3179 0 0 0
1528005041 3.7 6.7 12516 0 0 0 4122 0 0 0
1528005071 3.8 6.8 17640 0 0 0 5124 0 0 0
1528005102 3.6 6.8 21211 0 0 0 3571 0 0 0
1528005132 3.8 6.8 26134 0 0 0 4923 0 0 0
1528005162 3.6 6.8 30954 0 0 0 4820 0 0 0
1528005192 3.8 6.7 34950 0 0 0 3996 0 0 0
1528005222 3.6 6.6 38539 0 0 0 3589 0 0 0
1528005252 3.8 6.7 42789 0 0 0 4250 0 0 0
1528005283 3.8 6.7 47399 0 0 0 4610 0 0 0
1528005313 3.8 6.7 51470 0 0 0 4071 0 0 0
1528005343 3.8 6.8 55014 0 0 0 3544 0 0 0
1528005373 3.7 6.7 59714 0 0 0 4700 0 0 0
1528005403 3.8 6.8 64423 0 0 0 4709 0 0 0
1528005434 3.8 6.6 68081 0 0 0 3658 0 0 0
1528005464 3.8 6.8 72278 0 0 0 4197 0 0 0
-
Request: could we make all web output XHTML instead of html, so that I can process the results with an XML parser? Would make it miles easier for the likes of to write tools that filter or query the output because I could pipe the output into an XSLT program.
(There is of course an XHTML dialect corresponding to HTML5, which is up-to-date, as well as the old favourite XHTML 1.0.) There might be some subtle interactions with javascript code in web pages because of the difference between xml and html in this respect.
If you think it is too much pain then I can appreciate that. It would mean running pages through a validator and checking that they are well-formed XML as well as XHTML5 or XHTML1.0.
A possibly bad idea: Some people generate an alternative XML page at a different URL, the Firebrick router does this, I think. Seems a bit wasteful to me because if it is done very cleanly and properly then the page can be easy enough to process, especially if extra id and class values are supplied as ‘hooks’ to help locate elements, and all unnecessary presentational crud is left out and implemented by CSS. The two-files idea is to give the user one that is pretty to look at for eyeballs and another that is lean and mean and easy to parse for machines. But with really good design it should be possible to do both in one to a great extent.
-
You're giving me flash backs to a first year web dev course where the exam had a final question along the lines of "now rewrite everything you have done using XHTML and XSL"... nobody was a fan, then I never encountered it again! The lecturer was a fairly old school type, was adamant we all had to sign up to some weird british computing society, made a point of showing him bring up his network interfaces on the CLI after plugging in the projector... I digress. :D
What benefit does using XML to define webpages really have? Doesn't seem to be in wide use.
All I have at the moment in terms of HTML is a few named canvas elements, all the rest is my nasty excuse for javascript for parsing the output of xdslctl and structuring the data so it works with Chart.js.
Edit: If you mean producing an XML version of the data from the xdslctl output then thats doable I guess.
-
XHTML is just XML that uses html elements out of a particular namespace.
All that it means for the author is just writing the source according to the very simple xhtml rules,
* always use utf8,
* lowercase element names,
* always close empty elements with <element /> type syntax eg <hr /> <br />,
* otherwise always having a matching closing tag for every start tag eg <p>blah</p> and never omitting the closing tag </p>,
* it needs <html xmlns="http://www.w3.org/1999/xhtml">
* and it needs to be validated with the w3c's checker really (web-based) and an xml validator (many good web-based ones can be googled) as any authoring mistakes defeat the point of making it machine-readable and xml-parser-friendly.
It is that easy. There are some considerations with javascript that I have not covered here. If you are ok with using the stricter markup syntax then that would be excellent, otherwise don’t worry about it.
Some useful stuff at:
https://www.w3.org/TR/html-polyglot/#writing
https://www.w3.org/TR/xhtml1/guidelines.html
The main advantage is for users. XHTML is far more machine-readable because its syntax is very rigid with no peculiar quirks as in tag soup markup or html5. Users can use XML-speaking tools to process the page, so if a user wants to extract information from it it is so much easier because these tools can find content, strip it down or filter it very conveniently. This is doable because a large range of high quality XML parsing tools are available and the XSLT programming language can be used to read, filter and generate xml and xhtml because xhtml is an application of xml.
Html5 as well as html4 can be written according to the above few simply xhtml syntax authoring rules and then the source is XHTML5 too and so also is automatically XML without anything further being needed.
One benefit to the author is that authoring mistakes can be spotted quickly
-
Have made a VMG1312 firmware with the most basic of SNRM monitoring. It logs SNRM, FEC & CRC numbers every 30 seconds, but also has the output of various xdslctl commands available with the most basic form of an API.
At modem-ip:8000 is the SNRM graph. A GET request to modem-ip:8000/getdata will update files that can be accessed from a browser (or Workflow etc) under modem-ip:8000/data/, just browse there to see a directory listing.
The SNRM graph uses the modems current time vs the browsers to give the correct labels, if you change the time manually or by NTP some time after boot things will get screwy.
I have run various versions of this for months on a VMG8324, and this specific firmware overnight with a live line on a VMG1312.
Apart from the stats server this is vanilla 1312 AAJZ16 with the jumbo frame patches.
https://www.dropbox.com/s/y6f6pcgnwyz8dn4/1312-B10A-AAJZ16-SNRM-log.bin?dl=0
-
Have NOT
blown this into a couple of modems. They work fine, but I just get an error if I http-connect to port 8000 decimal. (Not 8080.)
I also tried 0x8000. If I remember correctly, my old boss once invited me round to his house, gave me the address with the house number in hex, without thinking about it. When I got there, I suddenly realised, and then went to the correct house which was #0x10.
I tried Safari and the NetToolbox app’s http test function, and then snooped on the whole conversation using the ‘Charles’ app which is an http-proxy debugger and saw the following response:
method GET
Error code NSPOSIXErrorDomainCode=61 "Connection refused" - error in connect() function
or similar, stupid thing would it let me copy-paste the result.
-
Oh thats disappointing!
I have tried it with the zyxel firewall set to high from wired and wireless connections from PC/tablet/phone, so was sure it just "worked". :(
I did indeed mean port 8000 decimal. A little google of "NSPOSIXErrorDomainCode=61 Connection refused" suggests its just a generic response to being unable to find the host.
If you are game for troubleshooting the first this would be to make sure the server is running on the device. Telnet in and gain real shell with "xtm && sh". The search the running processes for "stats":
~ # ps | grep stats
262 supervis 1148 S stats-server-bin
263 supervis 2148 S /bin/bash /var/tmp/stats/./stats-logging.sh
If it the stats server is running then the next thing to check the right files exist in /var/tmp/stats:
~ # cd /var/tmp/stats
/var/tmp/stats # ls
Chart.min.js getdata jquery.min.js
data getstats.sh moment.min.js
getCurTime.sh index.html stats-logging.sh
/var/tmp/stats #
Check the log file for data:
/var/tmp/stats # tail data/logfile
1514907468 4.3 6.5 8558 0 0 0
1514907498 4.4 6.4 3752 0 0 0
1514907528 4.3 6.4 6155 0 0 0
1514907559 4.3 6.4 5115 0 0 0
1514907589 4.3 6.4 3528 0 0 0
1514907619 4.3 6.4 2893 0 0 0
1514907649 4.2 6.5 3072 0 0 0
1514907680 4.4 6.5 3377 0 0 0
1514907710 4.3 6.5 2842 0 0 0
1514907740 4.3 6.4 2981 0 0 4
/var/tmp/stats #
Have you tried accessing the data folder from a browser, i.e "192.168.2.1:8000/data" ?
Apologies for supplying something that hasn't worked, and no problem if you'd rather not troubleshoot, I know how tedious it can be.
-
No stats process running.
Error connect trying to get access to http://ip:8000/data
No file found data/logfile
I will be more than delighted to do some debugging. A thought: I can send you the horrible mongrel config which I have inherited, if that will help ? Or I can send you a modem, not that there would be much advantage that I can see. Final daft thought, with a lot of faffing about I could perhaps map a routable global IPv4 address so that inbound traffic to that address gets redirected to the modem and that way you could access it directly across the internet.
Another thought. I ought to confirm that I have successfully flashed the modem at all. The whole thing is all a bit weird what with always losing contact in the web UI when reflashing. I ought to rule out some strange cockup that caused the update process to abort.
I was wondering before about altering version strings somehow.
I am trying to think about things that you have working that I don’t - facilities, services etc both due to the config in my modem and due to the characteristics of my network environment.
-
If the stats-server-bin is not even running thats where to start.
You make a good point about altering the version string, I'll have to remember where it is to edit and use for further versions.
To make sure it has definitely been flashed you can search the /etc/profile file for the lines that start the server:
~ # grep stats /etc/profile
mkdir /var/tmp/stats
mkdir /var/tmp/stats/data
cp /webs/stats-server/* /var/tmp/stats/
chmod +x /var/tmp/stats/getstats.sh
chmod +x /var/tmp/stats/getCurTime.sh
chmod +x /var/tmp/stats/stats-logging.sh
touch /var/tmp/stats/data/logfile
...
Or simply see if "stats-server-bin" is in the PATH:
~ # stats-server-bin
I'm pretty sure I still have your default config from a while ago in a PM if it hasn't changed much since then, but as I say, making sure of a successful flash is first on the list.
-
Yes, it had failed to blow the image.
All good now.
When I am using the web UI, I get a swirling useless progress indicator (why no proper percentage-done indicator, byte count, clock or growing bar) and that just goes on for ever and ever. Eventually I try hitting refresh. Then nothing much of significance is displayed, just the normal main web page. But it I go back into the upgrade page in the menus, it then immediately says "rebooting" and then no feedback again either, just sits there, so in the end when you do yet another refresh you see the login prompt which shows it has indeed rebooted. Booting still takes an eternity. In fact I know it has booted when an SMS or a Tweet arrives from AA telling me that it has connected to the internet! So anyway, this is how I have managed to screw up upgrades in the past, a lot of guesswork about what is going on. I just need to learn to follow a strict reflashing routine that works.
Logging in to the modem’s usual admin UI programmatically via http might be a bit of a pain, would have to look into it, but if I sorted that out, I could perhaps write a Shortcuts/Workflow program to reliably blow these modems. (I did write a very successful Workflow thing to push a new config into the Firebrick, and optionally tell it to reboot it too.)
-
Ah, well good stuff thats it working now. Version stringed releases from now on!
Wish I could offer some reason that the GUI is such a pain in your setup, but have never experienced it myself. Its not the snappiest by any means, but it never bugs out or hangs on the firmware upload page.
Is the SNRM graph drawing itself ok? Have left it auto ranging as it gives pretty good results with the range my line produces.
-
It’s just superb. Just so incredibly useful.
Re: the stupid standard ZyXEL web UI - I wonder if their JavaScript or possibly CSS just doesn’t work in Safari iOS or doesn’t work in machines that don’t have a normal pointing device, or both. The dsl status page thing has the bottom of the report clipped off because it is behind the icon bar that is drawn in front of it at the bottom, and the whole thing refuses to scroll. I am sure some js and css browser behaviour wizard who is familiar with Safari iOS could fix ZyXel’s junk and make it behave properly.
Can I just check - which low-level modem driver is in this new version? I ask because the sync speeds are awful - downstream sync down by 10% - nowadays on the two modems that I have reflashed. But it has to be due to the number of resyncs upsetting DLM or my choice of bad times for resynching, because the speeds were already bad when I had actually failed to reflash the device. There is no way it can be due to this new release.
Just need to think how to get the performance back, and the only thing I can think of is a wise choice of time to resynch, that or more mysterious long naps in the powered-off state.
I have changed the settings, or am trying to, as BTW are taking forever, so that the link’s interleaving setting is ‘on’ (ie max, hopefully) instead of ‘auto’. It’s a feature available in AA’s control server.
-
Its the standard default A2pv6F039v adsl_phy. Thats the one you use normally right?
All I can suggest is to wait till you have 24 hours of SNR data and pick an optimum resync time (via an xdslctl configure --snr reduction maybe) and hope for the best. I assume at least the most recent flash and resync was during darkness hours so probably suboptimal.
-
> the most recent flash and resync was during darkness hours so probably suboptimal.
Absolutely. That is what I was thinking. I don’t know what else there is, what other factors, that make a difference over time. Why exactly might my crosstalk vary - if indeed it does vary ? Why might rf interference from distant sources vary? Attenuation has changed over time, 0.3dB change to upstream attenuation was noticed by AA staff. So perhaps it’s to do with the electromagnetic environment around the cable, guessing, maybe water present or absent - who knows?
-
I recently installed a VMG1312-B10A on my dad's VDSL line that's connected to an ECI cabinet, hence no G.INP etc.
Would that be of any use to you?
-
And you do not need it?
-
Sorry, I was thinking more about the stats collection from it. It's on his line and going now so can't exactly take it away :)
Got a VMG3925-B10B I'm not sure what to do with though.
-
You could list it on the BB Hardware Swap Shop forum with view to sell, I have used this a few times without any issues, can't see any of this model listed there at present, last one listed, April this year, went for £30 + P&P as far as I can see.
-
Sorry for posting in this old thread however my question is to do with the Stats system.
Is there away to get this to work externally. I have setup the port forwarding for the required port and if i type in the URL on laptop it works without issue however no one externally can view it at all.
-
By ‘externally’, you mean remotely, over DSL? Or just stats at a public, global, routable IP address. If the latter, then the modem config can be set up with whatever admin IP address you like - I just happen to have it set to 192.168.n.1 where n is 1,2,3,4 for each one of my four modems.
-
Sorry for posting in this old thread however my question is to do with the Stats system.
Is there away to get this to work externally. I have setup the port forwarding for the required port and if i type in the URL on laptop it works without issue however no one externally can view it at all.
I fear you mean allowing the stats server to be accessed from the internet at your IP address. I would strongly advise against this as I would for the regular Zyxel GUI interface.
Bots are bots, and open ports get hammered. The Zyxel interface is years out of date and probably has large holes in, and while the mongoose web server that supplies stats from my firmware builds is meant for IoT devices, I cant guarantee its security either.
If you want remote access to stats I would recommend a VPN server (wireguard is fantastic) that you connect to when not on your home network.
-
What Johnson said. That’s a bad idea as you have no control over it. If you really need to access it remotely then configure the modem with an admin IP address that is global/public/routable which means having an ISP that is capable of offering appropriate addressing, and hr. make sure to firewall it appropriately so that only the people you want to be able to access it can do so.
-
Oops, sorry didn't make it clear, I am not after the Main WebGUI of the zyxel to be on internet, I am after getting port 8000 Statistics page to be on internet as I am using your custom firmware with the mongoose web server for statistics.
It works internally via my DDNS URL however doesn't work for external connections.
I wouldn't forward port the main WebGUI or even telnet or SSH over the internet as I know it's not wise.
Sent from my SNE-LX1 using Tapatalk
(https://uploads.tapatalk-cdn.com/20200317/722805248239017db876fed5a5325f7b.jpg)
-
Your router might be able to handle redirecting that ?
-
Your router might be able to handle redirecting that ?
As I can't PM you it, will have to be vague on setup but hopefully enough for you to get the jist.
Basically I setup the port forwarding like I did.
Now if I use the LAN side and use the DDNS URL I have it works
Firewall/Portforward is set like
External Port: 12345 (Not the port being used)
Internal Port: 8000
Source: WAN
Destination: LAN (IP address to modem)
Protocol: TCP
Hash limit: 60/minute
Now with this setup which was used on my Raspi 3B+ using DSLStats it worked fine however I can't get any external connections to work for the Zyxel mongoose server stats page.
So reason I was asking was just incase mongoose needed more setup or the VMG1312 needed a rule added like the one needed originally for dslstats to work where grouping was required.
Idea is to try and get it working and then when I have issues I can enable the rule so the ISP can see what's going on or if I am heading out for a bit I can have it enabled to keep an eye out on the line.
Sent from my SNE-LX1 using Tapatalk
-
Oh ok, I still wouldn't recommend exposing the mongoose server to the internet, but if its on a different port and you know what you are doing then fill your boots.
I have no experience with punching holes in the zyxel firewall, maybe try exposing 80 for a short time to test?
I assume you are using the zyxel as a full router? If you are using it as a modem with another router there will need to be another layer of jiggerypokery.
Edit: on second read I see it clearly is as a modem, and you are talking about your router settings... I'm scratching my head why that doesnt work. :-\
-
I am using the Zyxel as a Modem in bridged mode with my Linksys WRT32x with OpenWRT (Custom build / David502) firmware.
Now when i did the DSL-Stats on Raspi 3B+ i used the same settings above but with the address in the signature and i had no issue whats so ever getting it to work externally. However i can't for the love of me get it to work with Modem stats on port 8000, even having it set to port 8000 for source and dest on DNAT it still doesn't work.
Once i have it working it will be only enabled when either ISP needs to see and save the info or when i need to monitor whilst away else it will be off.
I always used different ports so that the internet being the internet will not cause that much of an issue if they try bouncing on SSH / TELNET / HTTP ports etc
-
Yeah sorry for the confusion.
I honestly cant offer a reason for something accessible on the lan with a properly configured port forward on the router not responding.
Do you have any other systems/services on the network that could be exposed (on a suitably random port) that you can try to connect to from an external connection? Just to prove that those are accessible with the same settings on the router.
-
Yes basically anything that connects directly to the router LAN port's / USB (using it for webserver), works without issue. However anything thats on the Zyxel to the router is not working. Messing with the iptables now with DNAT and SNAT on modem results in the same issue.
I am using Eth1 on modem to WAN on Router and using Eth4 on modem to Eth1 on Router with the Interface Group ptm0.1 lan1 setup on modem (which was needed for dslstats to work thru router to modem).
in fact if you remove the port from the address on my signature you'll get a basic webpage stating the website your looking for is closed and its being used as a nameserver.
-
Is the Eth4 on modem interface on the same subnet as the rest of your LAN? Or are you using SNAT trickery to get through to it?
-
Not using SNAT trickery, both modem and router are on same IP range.
Just had a thought, could it be that the mongoose web server setup be taking the modems up as the listening address instead of 0.0.0.0 (which would make it so external iOS could access it).
-
So I had a play and cant get a zyxel accessible from a remote IP at all myself.
I'm thinking its probably due to a modem mode zyxel not having a default gateway etc, no access to outside of the lan.
Will fiddle more tomorrow.
-
Thanks bud, setting up wireguard for now though.
Sent from my SNE-LX1 using Tapatalk
-
Thanks bud, setting up wireguard for now though.
Will find a more optimal solution to occasional remote access to the modem, but having a wireguard setup to remote into your network from any location is never a bad thing! :D
-
Will find a more optimal solution to occasional remote access to the modem, but having a wireguard setup to remote into your network from any location is never a bad thing! :D
That is very true, however thank you for your time and assistance. Will let you get back to what you was doing.
Update: I tried Wireguard and although I could pull up the router WebGUI I couldn't pull up the modem WebGUI so really stumped.
Sent from my SNE-LX1 using Tapatalk
-
Sincerest apology if you know all this already: Being in the same IP address range doesn’t mean you’re automatically in the same subnet - definition = “broadcast domain”. In my case the main LAN broadcast domain has an 81.187.x.y public/global/routable address in a /26 and each modem’s admin i/f lives in its own broadcast domain at 192.168.n.1 where n=1,2,3,4 is the nth modem (I have four modems). So that’s 1+4=5 broadcast domains=5 subsets, and in my case they are all distinct non-overlapping address ranges, but that’s just coincidental, it’s down to how I configured the modems.
-
Sincerest apology if you know all this already: Being in the same IP address range doesn’t mean you’re automatically in the same subnet - definition = “broadcast domain”. In my case the main LAN broadcast domain has an 81.187.x.y public/global/routable address in a /26 and each modem’s admin i/f lives in its own broadcast domain at 192.168.n.1 where n=1,2,3,4 is the nth modem (I have four modems). So that’s 1+4=5 broadcast domains=5 subsets, and in my case they are all distinct non-overlapping address ranges, but that’s just coincidental, it’s down to how I configured the modems.
It's always a great thing to relearn so no need to apologise beside it might help others.
However both modem and router are in same ip range of 192.168.1.0/32 I have even tried 192.168.1.0/24 however It did just remind me my Guest network has been setup on another range with firewall rules set to block WebGUI access and that's on the same vlan going to wan.
Sent from my SNE-LX1 using Tapatalk
-
I don’t understand your comment about "/32". That’s not large enough to have two IP addresses in it. Surely the modem and the router have to have two distinct IP addresses in their subnet? How does the modem manage if an incoming packet has src_address==dest_address, as when it replies to incoming packets it will be replying to itself? So I don’t see how the modem can be in a subnet that is /32. Your main LAN will typically be in a /24 or /16 if it’s in the usual home user setup with RFC1918 NAT-ed IP addresses?
Each of my modems lives in its own /24, as it happens, and in each subnet the modem is 192.168.n.1 and the router is 192.168.n.254.
-
I have Modem on 192.168.1.1 /255.255.255.0 and router on 192.168.1.3 /255.255.255.0 with guest network on 10.0.1.1 / 255.255.0.0 .
Guest network and normal network are on same vlan however guest network is isolated from the main network via firewall rules so they can't access webgui's or main network systems.
Hopefully this helps clear things up.
Sent from my SNE-LX1 using Tapatalk
-
Right. Some confusion earlier then, as you have a /24 there given it is 255.255.255.0, not a /32.
-
Sorry about that networking can be confusing at times lol. Hold up I had wireguard setup up to /32 and not /24 so that might be why it didn't work with wireguard lol. Will mess again later thanks for your time.
Sent from my SNE-LX1 using Tapatalk
-
Question: on the CRC and FEC const vs time graphs, what are the units on the verticaly axis? I have modem #4 running very very slowly, with a lot of noise and the downstream CRC count is high compared to the other modems. The CRC rate figure shown on the y axis is the count CRC of corrected errors[?] per unit what time[?]
It is a still a new performance nightmare at the moment. The downstream SNRM was 2.5 dB and I forced a resync which brought the d/s SNRM up to 3.1dB. The downstream rate is even slower now, with a downstream sync rate below 2.5Mbps, whereas it was ~2.9 Mbps before, and in total I have lost about 0.8-1.0Mbps total effective download rate from speed tests with speedtester2.aa.net.uk compared to what it was like a couple of months ago.
Live sync rates:
#1: down 2807 kbps, up 519 kbps
#2: down 2700 kbps, up 544 kbps
#3: down 2964 kbps, up 389 kbps
#4: down 2457 kbps, up 515 kbps
Modem #4 CRC count vs time graph:
(https://i.postimg.cc/43pJyxQD/C3-F02654-AE06-4-B72-BCB3-84-DB0608-F858.jpg)
Modem #4 stats:
xdslctl: ADSL driver and PHY status
Status: Showtime
Last Retrain Reason: 8000
Last initialization procedure status: 0
Max: Upstream rate = 436 Kbps, Downstream rate = 2708 Kbps
Bearer: 0, Upstream rate = 515 Kbps, Downstream rate = 2457 Kbps
Link Power State: L0
Mode: ADSL2 Annex A
TPS-TC: ATM Mode(0x0)
Trellis: U:ON /D:ON
Line Status: No Defect
Training Status: Showtime
Down Up
SNR (dB): 3.1 6.0
Attn(dB): 65.0 41.0
Pwr(dBm): 18.4 12.4
ADSL2 framing
Bearer 0
MSGc: 52 11
B: 28 62
M: 8 1
T: 3 1
R: 16 12
S: 2.9745 3.8462
L: 667 156
D: 1 8
Counters
Bearer 0
SF: 2803623 754342
SFErr: 2593 10
RS: 60978814 473909
RSCorr: 201383 13964
RSUnCorr: 2733 0
ReXmt: 464652 0
ReXmtCorr: 464413 0
ReXmtUnCorr: 2884 0
Bearer 0
HEC: 3616 17
OCD: 0 0
LCD: 0 0
Total Cells: 263857767 55806956
Data Cells: 5188622 1176787
Drop Cells: 0
Bit Errors: 167670 1432
ES: 219901 278
SES: 121 8
UAS: 4933 4825
AS: 45884
Bearer 0
INP: 26.00 2.00
INPRein: 0.00 0.00
delay: 8 8
PER: 16.17 16.34
OR: 28.68 8.32
AgR: 2475.44 522.12
Bitswap: 5896/5896 105/105
Total time = 33 days 4 hours 1 min 23 sec
FEC: 21598882 717266
CRC: 347108 447
ES: 219901 278
SES: 121 8
UAS: 4933 4825
LOS: 12 2
LOF: 79 2
LOM: 8 0
Latest 15 minutes time = 1 min 23 sec
FEC: 348 11
CRC: 8 0
ES: 7 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 15 minutes time = 15 min 0 sec
FEC: 3625 274
CRC: 34 0
ES: 32 0
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Latest 1 day time = 4 hours 1 min 23 sec
FEC: 57112 5415
CRC: 661 8
ES: 631 8
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
Previous 1 day time = 24 hours 0 sec
FEC: 688715 22265
CRC: 9424 4
ES: 8006 4
SES: 0 0
UAS: 54 54
LOS: 0 0
LOF: 0 0
LOM: 0 0
Since Link time = 12 hours 44 min 43 sec
FEC: 201383 13964
CRC: 2593 10
ES: 2252 10
SES: 0 0
UAS: 0 0
LOS: 0 0
LOF: 0 0
LOM: 0 0
NTR: mipsCntAtNtr=0 ncoCntAtNtr=0
-
Question: on the CRC and FEC const vs time graphs, what are the units on the vertical yaxis?
Number of events per minute. I.e. number of CRCs per minute and number of FECs per minute.
The CRC rate figure shown on the y axis is the count CRC of corrected errors[?] per unit what time[?]
No, not corrected errors but actual CRC errors recorded in the immediate, preceding minute.
-
Thank you I take it that FEC count is the count of error events corrected by RS ?
So it’s per minute. Where does L2 RETX come in in this count? Is PHY irrelevant here, if retransmission is required then it goes ahead and either succeeds or not but isn’t shown in these particular figures in this graph?
In the raw stats figures, are those first numbers under bearer 0 etc : ReXMT etc - are those just cumulative totals? Those in the first set just up and up? As opposed to being counts per some unit time interval? Which one sees further down in the various time interval sections.
-
Thank you I take it that FEC count is the count of error events corrected by RS ?
Yes, RS and any other "goodness" that was capable of making the correction
Where does L2 RETX come in in this count? Is PHY irrelevant here, if retransmission is required then it goes ahead and either succeeds or not but isn’t shown in these particular figures in this graph?
Sorry, that is something I do not know.
In the raw stats figures, are those first numbers under bearer 0 etc : ReXMT etc - are those just cumulative totals? Those in the first set just up and up? As opposed to being counts per some unit time interval? Which one sees further down in the various time interval sections.
They are cumulative totals. The delta is created by subtracting the previous cumulative total (at time n - 1 minutes) from the current cumulative total (at time n minutes).