Kitz Forum
Broadband Related => ISPs => Topic started by: jelv on January 21, 2019, 12:23:03 PM
-
22over7 recently posted a link to his TBB BQM
Plusnet
FTTC 80/20 (static)
Zyxel VMG1312-B10A
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/89493edf03886b2dcd280962c6420627.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/89493edf03886b2dcd280962c6420627)
The packet loss looks wrong to me.
From a PM he sent me his possible reasons are distance from TBB servers, he has a modem in bridge mode to the router which responds to the pings, he's on a Plusnet static IP, he has a Raspberry Pi monitoring the stats.
I don't buy any of those. I would have thought that if it were the packet loss would be constant. I have a similar setup and see no packet loss to AAISP.
-
It may only be packet loss affecting the line between the TBB servers, and has no effect on anything else.
I've seen this on a few BQM's where the user didn't notice any packet loss.
My BQM when I was with BT was often similar but the packet loss shown was not quite as bad.
I definitely didn't have any packet loss though, the line was perfect.
-
Many thanks for your attention jelv (& J0hn). I myself don't buy my clutched-straws.
The only thing I might add is that I doubt it's anything to do with line, which has comparatively
few error'd seconds. My FEC graphs do take a big lurch upwards [c. 250->12000 pm] when some nasty electrical device starts up
in my vicinity (shows up as a smear of yellow above the green bit), but it doesn't seem to correlate in any way
with the red stalactites of packet loss.
j0hn is quite right. I don't really notice any packet loss in real-life, except when mtr'ing some site on the other side of the world.
Throughput is respectably north of 70Mbps.
-
I've seen this on a few BQM's where the user didn't notice any packet loss.
None of the other BQM's that people have posted on here are showing any loss. I did have a "heartbeat" on IPv6 but that was a much smaller loss and at absolutely regular (roughly hourly) intervals. That disappeared with a router/firmware change.
It would be interesting to see a TBB speed test run at a time when the BQM was showing packet loss. I'd suspect the x1 download would be impacted.
-
It would be interesting to see a TBB speed test run at a time when the BQM was showing packet loss. I'd suspect the x1 download would be impacted.
FYI, an attached (I hope) screen shot. [Edit: Damn! Showed my postcode. Withdrawn for some cropping. In any case, x1 looks ok to me.]
The red stalactites are indeed sometimes rather heartbeat-ish. I've running Merlin fork 374.43_2-11E1j9527 (rather ancient, but good wifi) on an ASUS RT-n66u.
-
the graph definitely got my attention and I agree with jelv its not normal, not even close to normal.
The problem with speedtests is you only doing a snapshot, a speedtest lasts about 30 seconds or so and the packetloss is not 24/7.
What is the ES. SES and FEC stats average for past 7 days? Although FEC can probably be excluded as those are fixed errors which should not cause any packet loss. So just ES and SES stats.
Also do you have any anti ddos, or packet inspection type features enabled on your firewall which could be blocking some tbb pings. e.g. the fritzbox devices are known to do this. Asuswrt merlin has 2 firewall settings, one is the basic firewall which should be on, but there is a second option for anti ddos which when I used to use asuswrt as my router device I had that option disabled.
Also aside from the packet loss the latency variance increases at 10pm and then stays increased for the rest of the graph including overnight, I suspect there is possible malware on the network if the other stuff is ruled out, and we assume not seeding torrents or something.
-
What is the ES. SES and FEC stats average for past 7 days? Although FEC can probably be excluded as those are fixed errors which should not cause any packet loss. So just ES and SES stats.
Over the last 7 days, averages, ES(D) = 5, ES(U) = 15. (Computed from a weeks worth of dslstats errors>averages. It has to be said, now and then there can be twice as many, the upstream almost always more ES-prone.)
I have not seen ANY SES for a very long time. (Yes, I mean that. ) In the last year, I had one resync for a power brown-out, AFAICR.
Also do you have any anti ddos, or packet inspection type features enabled on your firewall which could be blocking some tbb pings. e.g. the fritzbox devices are known to do this. Asuswrt merlin has 2 firewall settings, one is the basic firewall which should be on, but there is a second option for anti ddos which when I used to use asuswrt as my router device I had that option disabled.
Basic firewall on, ddos off. I need to do some head-scratching re other kinds of packet inspection. I don't do Qos, or parental stuff, which might I suppose do that.
Also aside from the packet loss the latency variance increases at 10pm and then stays increased for the rest of the graph including overnight, I suspect there is possible malware on the network if the other stuff is ruled out, and we assume not seeding torrents or something.
Worrying. I don't do torrenting, or things like that. Some iPlayer/Netflix watching a couple of times a week.
Devices include: amazon firestick and tablet (off when not in use), 2 Raspbian pi's (always on: dslstats, and printer server), two linux systems (usually off), 2 mac systems (one always on, one hardly ever), wife's equipment eg. phone (evenings only). So, usually, on = mac, router, 2 pi's. The mac (and all other desktops, not too sure about the pi's) have basic firewalling, but no anti-virus stuff (at one time used free Sophos on the mac).
I close all unnecessary ports on anything, if I can, and am quite conscientious, I think, about good passwords and changing them.
So. I guess you'd advise (??)
1. Update router firmware.
2. Reinstall Sophos on the mac, at least to scan it.
3. Check out the firewalling on the pi's.
4. Anything else?
I appreciate your thoughts, and definitely have things to do. However, I am not entirely persuaded this is a malware problem.
I suppose I could ask a local friend to ping me at my static IP for a few days, and keep track of what packet-loss is detected from a non-TBB source.
Plusnet seem to change my gateway several times a week. I have some suspicions that their provision for customers in Scotlandshire is
not as good as AAISP's might be. I can't get a straight answer from them about they route static ips, or what disadvantages that might bring.
Many thanks.
-
The problem is I am only speculating. I am confident in saying something isnt right but thats about it. I agree with you that the ES is way too low to be causing what we seeing so it isnt a line problem, its not an isp problem, as other plusnet graphs are not like that.
If you can add a monitor on another service as well that would be good, so there is a second service pinging the line thats not tbb.
If it is malware related updating the router firmware is unlikely to do anything, it would most likely be on a device thats always on, if none of your devices are always on (except the router) then I am probably wrong about the malware.
-
The fact that there is no significant (if any) increase in the maximum latency (yellow) indicates to me that it is nothing to do with anything running at your end.
Like you, I'd be more suspicious of
their provision for customers in Scotlandshire
their capacity to your exchange, or the capacity to your cabinet.
-
The problem is I am only speculating. I am confident in saying something isnt right but thats about it. I agree with you that the ES is way too low to be causing what we seeing so it isnt a line problem, its not an isp problem, as other plusnet graphs are not like that.
No, I'm sincerely grateful for what you've brought to my attention. My pi's (that are always on) have completely ACCEPT iptables. One (the printer server, and ad-blocker) has ports 22, 53, 80, 139, 445, 5900 open (to the lan). Aaargh!
Interestingly (to me) the packet loss spikes currently seem to occur (more or less) every 2 hours.
Having kept an eye on the traffic on my lan, I was appalled at the number of things that "phone home". Like realVNC, dropbox, amazon gadgetry, apple, google etc. , generally to somewhere in america. I stamped out a few, but it seems to be a lifetime's work.
@jelv, there's a lot in what you say too. I've had red stalactites on TBB BBQM with Plusnet forever (though not as luridly as now). The yellow is I suspect down to something hostile in my neighbourhood. (There used to be a cannabis farm near the DSLAM. Maybe it's back.) I'd love a lower, more reliable ping, but to be honest, that's just the obsessive in me. I'm not a gamer, and the throughput is enough. I'm reconciled (grumpily) to internet treacle (and to it's getting progressively worse...).
Still, maybe this interchange might after all be quite relevant to the ISPs forum.
-
The latency has gone back to normal just before 6pm.
Maybe your fiddling fixed that side of things at least :)
The amount of things exploitable is alarming, someone hacked printers not long ago to promote a youtube channel.
-
Worry not about open ports on the LAN.
Worry about the WAN.
https://www.grc.com/shieldsup
Just remember this is ONLY monitoring packet loss between you and the TBB servers.
It does not necessarily indicate a problem.
Run some traceroutes during periods of packet loss showing.
Ping various different IP's and see if any packet loss is shown.
Some routers with DOS protection enabled will randomly drop ICMP packets.
I had a very similar BQM to you for about 3 months.
I can say with absolute certainty my connection was fine.
Replacing my old Asus router with a slightly better model (still an Asus) made the reported packet loss disappear.
-
Thanks pals.
We're (hopefully) done, I think.
Attached is what (just vefore) 6 o'clock (pm) looked like on my FEC page.
Maybe a graph of (certain kinds of) error correction in action.
-
Plusnet seem to change my gateway several times a week. I have some suspicions that their provision for customers in Scotlandshire is not as good as AAISP's might be.
I note you said you are on a static IP, therefore there are supposedly several gateways in London you can connect to. Dynamic IP's have a more static routing as the IP's are dished out by a local pool near to the bRAS before the MSILs
Interestingly, I've just done a tracert and the details for these have changed fairly recently as there used to be a description of the gateway name when you did a tracert. (eg pcn-bng04.plus.net)
Now it just gives me the IP address which if I look up just simply says it belongs to BT, but further digging shows that it was formerly the IP for pcn-ir01.plusnet. (Colindale PoP).
https://whatismyipaddress.com/ip/84.93.253.64
-
Thanks pals.
We're (hopefully) done, I think.
Attached is what (just vefore) 6 o'clock (pm) looked like on my FEC page.
Maybe a graph of (certain kinds of) error correction in action.
ahh yeah I forgot, FEC on g.inp does cause jitter, so that probably is the explanation for the latency thing I picked up on, and probably explains why jelv said its normal as well.
-
Are there any graphs of the retransmissions rather than FEC? It would make much more sense if it were the retransmissions causing the increased latency, not FEC.
-
Are there any graphs of the retransmissions rather than FEC? It would make much more sense if it were the retransmissions causing the increased latency, not FEC.
Well, the CRC is handy (attached, I hope), though not very dramatic. That corrects 1-bit errors doesn't it?
Is there some graph in dslstats that directly shows retransmissions?
-
CRCs are detected errors that have not been corrected.
-
G.INP tab, rtx_tx
rtx_uc and LEFTERS would also be interesting.
DslStats doesn't save any of these.
The G.INP tab only appears if enabled on the line.
-
G.INP tab, rtx_tx
rtx_uc and LEFTERS would also be interesting.
DslStats doesn't save any of these.
The G.INP tab only appears if enabled on the line.
I certainly have a G.INP tab. Unfortunately, dslstats has a gap from 20:30->08.50 (for the first time ever!),
making the graphs a bit rubbish. I aim to post them when they fill up again.
In the meantime, here is the "Data Summary"
Downstream Upstream
General
rtx_tx 132012362 0
rtx_c 72139756 0
rtx_uc 14228 0
LEFTRS 1467760 0
minEFTR 79999 0
errFreeBits 3436881935 0
Bearer 0
RxQueue 60 0
TxQueue 20 0
G.INP Framing 18 0
G.INP Lookback 20 0
RRC Bits 0 24
Interleave depth 16 1
INP 48.00 0.00
INPRein 0.00 0.00
Delay 0 0
Bearer 1
Interleave depth 3 0
INP 4.00 0.00
INPRein 4.00 0.00
Delay 3 0
Other data, copied from the bottom of the dslstats window is:
Modem uptime 155days... up speed : 18999 up SNRM : 8.0dB interleave D/U : 16/1
DSL uptime 155days ... down speed: 79999 down SNRM : 3.3dB ES D/U : 0/0.20
-
Concerning retransmissions, my graphs have filled up again.
rtx_tx is attached
LEFTRS isn't because it seems to be almost identical. Likewise rtx_c. (Can this be right??)
rtx_uc isn't because it's just 0 along the x axis!
-
I forgot to mention that clicking the camera/snapshot button on rtx-tx,
rtx-c and LEFTERS all return the same rtx-tx graph.
I don't know if rtx_uc works right as my graph is empty.
rtx_uc is correct.
I just learnt something new and that you can rewind in the graph and take a manual snapshot.
Scrolling back to a single rtx_uc a couple days ago it captures the correct graph.
Auto snapshots don't work on any of the G.INP tabs.
You would need to screenshot the actual DslStats app unfortunately.
edit: corrected info
-
I forgot to mention that clicking the camera/snapshot button on rtx-tx, rtx-c and LEFTERS all return the same rtx-tx graph.
I don't know if rtx_uc works right as my graph is empty.
For reference, have checked this on my RPi's/DSLStats by taking manually snapshots and comparing:
rtx-tx and rtx-c (snapshots v live DSLStats) do appear to be very similar if not identical, surely that's the objective of re-transmission (G.Inp), all or most of the rtx-tx errors corrected, under perfect conditions graphs identical, should always be difficult to tell the difference.
LEFTRS snapshot definately appears incorrect, looks like rtx-tx & rtx-c DSLStats displays as you have said.
rtx-uc snapshot would appear to produce the correct image by comparison to live DSLStats display.
-
Looking again rtx_tx and rtx_c do display the correct manually taken snapshots.
They differ slightly, as they should.
The LEFTERS manual snapshot definitely shows rtx_tx.
I never look at the OHFErr and RSCorr (bearer 1) G.INP tabs so no idea if their manual snapshots are correct.
Looking back at previous Telnet Data captures I have never had any of either, always 0
edited my post above.
-
Many thanks for the warnings about the camera icon.
Looking carefully through my dslstats pages (over realVNC, and via the webserver), it seems to be in a right pickle.
Some pages are stuck at 22 Jan, some are up to date.
My RSCorr (bearer 1) looks like it contains something (about 3 a minute), but it's 2 days old.
I'll have to investigate my configuration properly. Maybe restart dslstats, even reboot the pi.
Solid information, if any, will be some time ...
-
I just learnt something new and that you can rewind in the graph and take a manual snapshot.
Scrolling back to a single rtx_uc a couple days ago it captures the correct graph.
Not aware of this feature either, good spot, thanks for sharing, just hold left mouse button and scroll.
Noted that graph returns to current time at the next auto sample.
-
I recently upgraded my wireless router to a beefier device, and this has coincided with the packet loss disappearing from my BQM.
(I've retired an ASUS RT-N66U, generally considered obsolete, running RT-N66U_3.0.0.4_374.43_2-38L3j9527.zip.
Great wifi though. I now have another ASUS, a RT-AC86U, running Merlin's RT-AC86U_384.10_2.zip.)
My hunch is that the packet loss was in large part because I've a pi 3b+ running dslstats connected via the router by ethernet to
my Zyxel VMG1312-B10a modem, basically hammering the poor little router every minute or so.
(I have changed my mind about this more than once.) Another factors are the enormous amount of network rubbish that
some devices in my house generate. However, assiduously turning the bloody things off diminished the packet loss
by only a wee bit. Yes, I've actually got no convincing proof of my suspicions, but my BQM looks red-free now.
-
Is the pi storing the dslstats data locally or uploading to another device on the LAN (or elsewhere)?
-
Another factors are the enormous amount of network rubbish that some devices in my house generate.
Is the "network rubbish" LAN or WAN traffic? If it's LAN and the devices are wired get a cheap unmanaged Ethernet switch (https://www.amazon.co.uk/dp/B00011PFWG/), plug that in to the router and all wired devices in to the switch - that will take the traffic away from the router.
-
Is the pi storing the dslstats data locally or uploading to another device on the LAN (or elsewhere)?
Locally, on a usb stick.
-
Is the "network rubbish" LAN or WAN traffic? If it's LAN and the devices are wired get a cheap unmanaged Ethernet switch (https://www.amazon.co.uk/dp/B00011PFWG/), plug that in to the router and all wired devices in to the switch - that will take the traffic away from the router.
Thank you for that very sensible and elightening advice. All I can say is "doh!", that hadn't occurred to me.
I've ordered that switch, and will give it a try with my old router (which I like much more than my new, expensive, crazily-feature-larded one).