Kitz Forum
Broadband Related => FTTC and FTTP Issues => Topic started by: leonroy on October 26, 2019, 12:17:27 PM
-
Zen/BT just upgraded my property from VDSL to G.Fast (4-5 days ago). They fitted a new G.Fast faceplate and Huawei MT992 modem.
The BT engineer showed me his diagnostic device which shows the line syncing at the maximum possible speed of 400/100. Service is capped at 330/50 so no chance of getting those speeds alas!
In real world use I'm getting 290 Mbps down and 48 Mbps up. Things are stable with 0.0% packet loss until I try downloading something large and max out the line's bandwidth at which point my router shows packet loss of up to 20%.
Before:
(https://i.imgur.com/VRJcIxv.png)
Just started the download:
(https://i.imgur.com/kQhkg24.png)
After:
(https://i.imgur.com/6tEUYIS.png)
I wasn't seeing a single issue with the VDSL line. Any suggestions on what could be the problem?
-
What router are you using?
—
I have a problem understanding the definition of ‘packet loss’ in this situation. How can a router know that packets sent by it have been lost? Similar problem with packets in the other direction.
If your speed is capped by a rate limiter, this could mean very high packet loss, because the sender could be sending too fast causing packets to get dropped so as to enforce the rate limit.
-
What router are you using?
—
I have a problem understanding the definition of ‘packet loss’ in this situation. How can a router know that packets sent by it have been lost? Similar problem with packets in the other direction.
If your speed is capped by a rate limiter, this could mean very high packet loss, because the sender could be sending too fast causing packets to get dropped so as to enforce the rate limit.
I'm using a pfSense router. It uses 'Gateway Monitoring' (basically pings the ISP gateway) to verify connectivity. Here's a screen of where that's configured:
(https://i.imgur.com/LQH6Svo.png)
Running a ping to 8.8.8.8 from a local machine on the network during heavy traffic does show request timeouts:
Request timeout for icmp_seq 223
64 bytes from 8.8.8.8: icmp_seq=224 ttl=54 time=4.747 ms
64 bytes from 8.8.8.8: icmp_seq=225 ttl=54 time=15.432 ms
64 bytes from 8.8.8.8: icmp_seq=226 ttl=54 time=17.072 ms
64 bytes from 8.8.8.8: icmp_seq=227 ttl=54 time=18.167 ms
Request timeout for icmp_seq 228
64 bytes from 8.8.8.8: icmp_seq=229 ttl=54 time=16.615 ms
64 bytes from 8.8.8.8: icmp_seq=230 ttl=54 time=17.131 ms
64 bytes from 8.8.8.8: icmp_seq=231 ttl=54 time=17.275 ms
64 bytes from 8.8.8.8: icmp_seq=232 ttl=54 time=17.632 ms
Request timeout for icmp_seq 233
64 bytes from 8.8.8.8: icmp_seq=234 ttl=54 time=17.636 ms
64 bytes from 8.8.8.8: icmp_seq=235 ttl=54 time=15.776 ms
64 bytes from 8.8.8.8: icmp_seq=236 ttl=54 time=17.386 ms
64 bytes from 8.8.8.8: icmp_seq=237 ttl=54 time=10.089 ms
64 bytes from 8.8.8.8: icmp_seq=238 ttl=54 time=17.735 ms
64 bytes from 8.8.8.8: icmp_seq=239 ttl=54 time=11.701 ms
Request timeout for icmp_seq 240
64 bytes from 8.8.8.8: icmp_seq=241 ttl=54 time=17.605 ms
64 bytes from 8.8.8.8: icmp_seq=242 ttl=54 time=7.494 ms
64 bytes from 8.8.8.8: icmp_seq=243 ttl=54 time=16.695 ms
64 bytes from 8.8.8.8: icmp_seq=244 ttl=54 time=17.661 ms
64 bytes from 8.8.8.8: icmp_seq=245 ttl=54 time=4.683 ms
64 bytes from 8.8.8.8: icmp_seq=246 ttl=54 time=17.482 ms
Also setup BQM. Ignore the first two red peaks since those were caused by power cycling the modem. The current one is under heavy downloads:
(https://i.imgur.com/99agnYb.png)
-
in advanced - network settings, near the bottom there is 3 boxes related to offloading, which one's are ticked?
Also do you know what nic hardware you using?
-
in advanced - network settings, near the bottom there is 3 boxes related to offloading, which one's are ticked?
(https://i.imgur.com/O5aqrvV.png)
Also do you know what nic hardware you using?
Good question, they're onboard Intel I354 Gigabit NICs. Board is Supermicro A1SRi-2758F.
-
try unticking the TSO and LRO boxes, and reboot after to see if any impact.
pfsense dev's insist there is no benefits to these on routing, but I think they assume a non NAT scenario (such as datacentre usage). I dont know if it will help at all, but some on reddit did report by allowing all offloading it fixed packet loss during download activity.
-
Many thanks for that suggestion, alas did not seem to make a difference.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/0aa3fe08471c17484857ae279b40fb745de68ce3-28-10-2019.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/0aa3fe08471c17484857ae279b40fb745de68ce3-28-10-2019)
Is there a way I can determine the cause behind this problem using line stats?
Alternatively is it worth contacting my ISP? My only fear is they might throttle the line or something further.
-
You need to try another router to rule out the pfsense box as the culprit.
It's not uncommon for some hardware running pfsense to struggle at much higher throughput rates.
If/when you can confirm the same issue on another router (did Zen send you 1?) then contact the ISP.
-
The logical course of action to confirm pfsense is the cause is to temporarily use another router. The behaviour did actually worsen with the changes, so I expect pfsense is the culprit here.
-
Heheh, I asked Zen to keep the free router since I had a pfSense router (waste not and all that!). Guess I better tell them I'll need them to send one after all :D
-
I'd be surprised if it is your router, I looked up that board, an octa core Atom, whilst no Xeon - it's still pretty powerful and should be more than capable of handling 330Mbps.
I get could 1Gbps out of an older Atom CPU.
On the console of your pfSense machine you can run iperf - have you tried this to prove the throughput on your local LAN?
-
@dee.jay - this is weird...
So I have two sites, both with pfSense and connected via 1GbE. They've often run file transfers across without ever an issue before. Just ran iperf between them. I also connected an iperf client to the affected gateway to further load it. I also started a ping out to 8.8.8.8 and to the gateway to verify LAN and WAN connectivity.
Interestingly I see dropped packets on both LAN and WAN:
LAN
64 bytes from 10.1.0.1: icmp_seq=5 ttl=64 time=0.646 ms
64 bytes from 10.1.0.1: icmp_seq=6 ttl=64 time=0.613 ms
64 bytes from 10.1.0.1: icmp_seq=7 ttl=64 time=1.183 ms
Request timeout for icmp_seq 8
64 bytes from 10.1.0.1: icmp_seq=8 ttl=64 time=1488.186 ms
64 bytes from 10.1.0.1: icmp_seq=9 ttl=64 time=487.726 ms
64 bytes from 10.1.0.1: icmp_seq=10 ttl=64 time=0.674 ms
WAN
64 bytes from 8.8.8.8: icmp_seq=41 ttl=54 time=4.419 ms
64 bytes from 8.8.8.8: icmp_seq=42 ttl=54 time=3.936 ms
64 bytes from 8.8.8.8: icmp_seq=43 ttl=54 time=6.613 ms
Request timeout for icmp_seq 47
Request timeout for icmp_seq 48
64 bytes from 8.8.8.8: icmp_seq=44 ttl=54 time=5207.083 ms
64 bytes from 8.8.8.8: icmp_seq=45 ttl=54 time=4207.624 ms
64 bytes from 8.8.8.8: icmp_seq=46 ttl=54 time=3207.687 ms
To reiterate I'm not sending any traffic on WAN at all.
After running the iperf test for a few minutes both LAN and WAN have gone down on the pfSense box.
(https://i.imgur.com/13vyr1f.png)
Guess the issue is definitely a pfSense one...
-
Yeah, there is definitely something up with your hardware - the throughput 600MBps shows it's capable of actually pushing that much.
Is it running bare metal / as a VM?
Are the NIC's on board?
-
dee jay I have seen pfsense do this before, it happened on my old pfsense unit. So thats why I think its a possibility.
Ironically the best I have seen pfsense perform (and opnsense) is on VM's as virtual nic drivers have been very well developed and tested.
-
Maybe it's time for a new pfSense install.
I run a virtual instance, it's been faultless.
-
Swapped over to the Zen provided Fritz!Box 7530 and seeing the same issue with dropped packets on saturating the line.
https://en.avm.de/products/fritzbox/fritzbox-7530/
Similar dropped packets whilst trying to ping outside:
64 bytes from 8.8.8.8: icmp_seq=504 ttl=54 time=4.535 ms
64 bytes from 8.8.8.8: icmp_seq=505 ttl=54 time=4.577 ms
64 bytes from 8.8.8.8: icmp_seq=506 ttl=54 time=15.606 ms
64 bytes from 8.8.8.8: icmp_seq=507 ttl=54 time=16.302 ms
64 bytes from 8.8.8.8: icmp_seq=508 ttl=54 time=4.061 ms
Request timeout for icmp_seq 509
64 bytes from 8.8.8.8: icmp_seq=510 ttl=54 time=17.556 ms
Request timeout for icmp_seq 511
64 bytes from 8.8.8.8: icmp_seq=512 ttl=54 time=17.713 ms
64 bytes from 8.8.8.8: icmp_seq=513 ttl=54 time=4.612 ms
64 bytes from 8.8.8.8: icmp_seq=514 ttl=54 time=17.352 ms
64 bytes from 8.8.8.8: icmp_seq=515 ttl=54 time=16.380 ms
Pinging the router (unlike pfSense) is fine though:
64 bytes from 10.1.0.2: icmp_seq=81 ttl=64 time=0.593 ms
64 bytes from 10.1.0.2: icmp_seq=82 ttl=64 time=0.673 ms
64 bytes from 10.1.0.2: icmp_seq=83 ttl=64 time=0.441 ms
64 bytes from 10.1.0.2: icmp_seq=84 ttl=64 time=0.576 ms
BQM graph (last bit) indicates:
(https://i.imgur.com/uouRURn.png)
It's supposed to handle speeds up to 300Mbps so wondering what on earth could be the cause. Unless the Fritz!Box is similarly afflicted?
-
I would doubt highly it would be affected the same way. Looks like you have ruled out your original hardware, because the issue is still present.
-
Even tho you have not solved it, for the reasons dee jay said this is progress as you have now determined it is present after changing the CPE.
So this could be perhaps related to the way either zen or openreach (or BTw if BTw backhaul) queue traffic.
A way round that would be to shape below your maximum throughput so the bottleneck is moved to your network and then have higher control over the queuing of packets, the problem is this is harder to do on ingress. ISP's like aaisp give you control over what they rate limit the traffic at to mitigate downstream packet loss issues.
What type of service is the download been carried out on? Things like steam are horrible as they effectively DDOS the end user's line on a download with how high threaded the download is.
-
There's a setting the Fritz!Box router to reduce the line speed:
(https://i.imgur.com/cMKHI0I.png)
I've dropped it from 338/51 to 300/45. Same issue though, BQM is reporting dropped packets.
ISP is Zen and the package is Zen Fibre 4.
Don't suppose it could be the Huawei MT992 modem?
-
I saw this the other day and thought surely if the link is under extreme high load then there is the possibility of dropped packets, but I don't know enough about settings on that particular router etc to be able to contribute anything useful.
Jelv & Weaver are actually correct... the ISP is supposed to rate limit to match the IP profile or line rate, but the only ISP's where we have seen evidence of them doing this is with AAISP and Plusnet.
It's several years ago now since I read the docs, but from memory basically it is both to stop bottle necks on the BTw backhaul and also to help reduce the possibility of dropped {ATM} packets at the DSLAM. eg if the download is coming down the backhaul at say 1Gbps (or multi Gbps) and suddenly is restricted to say xDSL 10Mbps DSL then you can end up with a bottleneck and dropped packets at that point. I don't know if this also applies to G.fast, but certainly it was documented for FTTC and ADSL. I'd need to do some research and double check if its something to do with ATM or not before I start chipping in my 2p.
The takeup rate for g.fast is quite low... if he is sharing a BTw backhaul with only a couple of g.fast connections then he possibly has a nice big fat pipe practically to himself right up to the OLT. BTw only police the amount of bandwidth traffic passing through the ISP host link and not the speed - which is why they recommend the ISP rate limits before downstream traffic hits their backhauls.
I dont know how the IP profile for g.fast works... or even if there is one but presumably there must be.
----
ETA can't find the documentation right now.. searching brings me to here
https://kitz.co.uk/adsl/IPprofile.htm#why_IP_bRAS_profile :lol:
-
Thanks, appreciate the feedback from all here.
Just an update on my own findings. I kept a note of the IP of the Zen gateway so was able to tell when it changes. Some Zen gateways seem to be less reliable than others, right now it's working well.
My test to create packet drops might be partly responsible; I was downloading a bunch of little files across many SSL connections. This test reliably causes 9-16% packet loss and lost pings to 8.8.8.8.
However when I download a single big file like an ISO from a single host I get zero packet loss and zero lost pings.
Both tests saturate the pipe.
Some more discussion here:
https://forum.netgate.com/topic/147820/acceptable-packet-loss-on-wan/14
Will report back if I notice anything new.
-
yeah thats no surprise, the little files test will have a higher packets per second, and its pps thats really the measure of a network appliance.
-
Had another thought which I cant see mentioned. Buffer bloat?
-
Had another thought which I cant see mentioned. Buffer bloat?
Can do subnet calculations in my head (just about ;D) but first I've heard of buffer bloat, live and learn.
(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fwww.dslreports.com%2Fspeedtest%2F56309497.png&hash=67c3e2c711e4d697c390962cf900ec75ae6083b0)
Full results:
https://www.dslreports.com/speedtest/56309497
No packet loss reported in pfSense on the WAN side during the test.