Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: High packet loss and occasional disconnection under load  (Read 4178 times)

leonroy

  • Member
  • **
  • Posts: 22
High packet loss and occasional disconnection under load
« on: October 26, 2019, 12:17:27 PM »

Zen/BT just upgraded my property from VDSL to G.Fast (4-5 days ago). They fitted a new G.Fast faceplate and Huawei MT992 modem.

The BT engineer showed me his diagnostic device which shows the line syncing at the maximum possible speed of 400/100. Service is capped at 330/50 so no chance of getting those speeds alas!

In real world use I'm getting 290 Mbps down and 48 Mbps up. Things are stable with 0.0% packet loss until I try downloading something large and max out the line's bandwidth at which point my router shows packet loss of up to 20%.

Before:


Just started the download:


After:


I wasn't seeing a single issue with the VDSL line. Any suggestions on what could be the problem?
Logged

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 9904
  • Retd s/w dev; A&A; 3x7km lines; Firebrick; IPv6
Re: High packet loss and occasional disconnection under load
« Reply #1 on: October 26, 2019, 12:56:50 PM »

What router are you using?


I have a problem understanding the definition of ‘packet loss’ in this situation. How can a router know that packets sent by it have been lost? Similar problem with packets in the other direction.

If your speed is capped by a rate limiter, this could mean very high packet loss, because the sender could be sending too fast causing packets to get dropped so as to enforce the rate limit.
« Last Edit: October 26, 2019, 01:03:27 PM by Weaver »
Logged

leonroy

  • Member
  • **
  • Posts: 22
Re: High packet loss and occasional disconnection under load
« Reply #2 on: October 26, 2019, 02:30:20 PM »

What router are you using?


I have a problem understanding the definition of ‘packet loss’ in this situation. How can a router know that packets sent by it have been lost? Similar problem with packets in the other direction.

If your speed is capped by a rate limiter, this could mean very high packet loss, because the sender could be sending too fast causing packets to get dropped so as to enforce the rate limit.

I'm using a pfSense router. It uses 'Gateway Monitoring' (basically pings the ISP gateway) to verify connectivity. Here's a screen of where that's configured:



Running a ping to 8.8.8.8 from a local machine on the network during heavy traffic does show request timeouts:
 
Code: [Select]
Request timeout for icmp_seq 223
64 bytes from 8.8.8.8: icmp_seq=224 ttl=54 time=4.747 ms
64 bytes from 8.8.8.8: icmp_seq=225 ttl=54 time=15.432 ms
64 bytes from 8.8.8.8: icmp_seq=226 ttl=54 time=17.072 ms
64 bytes from 8.8.8.8: icmp_seq=227 ttl=54 time=18.167 ms
Request timeout for icmp_seq 228
64 bytes from 8.8.8.8: icmp_seq=229 ttl=54 time=16.615 ms
64 bytes from 8.8.8.8: icmp_seq=230 ttl=54 time=17.131 ms
64 bytes from 8.8.8.8: icmp_seq=231 ttl=54 time=17.275 ms
64 bytes from 8.8.8.8: icmp_seq=232 ttl=54 time=17.632 ms
Request timeout for icmp_seq 233
64 bytes from 8.8.8.8: icmp_seq=234 ttl=54 time=17.636 ms
64 bytes from 8.8.8.8: icmp_seq=235 ttl=54 time=15.776 ms
64 bytes from 8.8.8.8: icmp_seq=236 ttl=54 time=17.386 ms
64 bytes from 8.8.8.8: icmp_seq=237 ttl=54 time=10.089 ms
64 bytes from 8.8.8.8: icmp_seq=238 ttl=54 time=17.735 ms
64 bytes from 8.8.8.8: icmp_seq=239 ttl=54 time=11.701 ms
Request timeout for icmp_seq 240
64 bytes from 8.8.8.8: icmp_seq=241 ttl=54 time=17.605 ms
64 bytes from 8.8.8.8: icmp_seq=242 ttl=54 time=7.494 ms
64 bytes from 8.8.8.8: icmp_seq=243 ttl=54 time=16.695 ms
64 bytes from 8.8.8.8: icmp_seq=244 ttl=54 time=17.661 ms
64 bytes from 8.8.8.8: icmp_seq=245 ttl=54 time=4.683 ms
64 bytes from 8.8.8.8: icmp_seq=246 ttl=54 time=17.482 ms

Also setup BQM. Ignore the first two red peaks since those were caused by power cycling the modem. The current one is under heavy downloads:
Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 6443
Re: High packet loss and occasional disconnection under load
« Reply #3 on: October 27, 2019, 10:49:54 AM »

in advanced - network settings, near the bottom there is 3 boxes related to offloading, which one's are ticked?

Also do you know what nic hardware you using?
Logged
AAISP - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

leonroy

  • Member
  • **
  • Posts: 22
Re: High packet loss and occasional disconnection under load
« Reply #4 on: October 27, 2019, 08:08:26 PM »

in advanced - network settings, near the bottom there is 3 boxes related to offloading, which one's are ticked?



Quote
Also do you know what nic hardware you using?

Good question, they're onboard Intel I354 Gigabit NICs. Board is Supermicro A1SRi-2758F.
Logged

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 6443
Re: High packet loss and occasional disconnection under load
« Reply #5 on: October 27, 2019, 09:52:07 PM »

try unticking the TSO and LRO boxes, and reboot after to see if any impact.

pfsense dev's insist there is no benefits to these on routing, but I think they assume a non NAT scenario (such as datacentre usage).  I dont know if it will help at all, but some on reddit did report by allowing all offloading it fixed packet loss during download activity. 
Logged
AAISP - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

leonroy

  • Member
  • **
  • Posts: 22
Re: High packet loss and occasional disconnection under load
« Reply #6 on: October 28, 2019, 09:23:27 AM »

Many thanks for that suggestion, alas did not seem to make a difference.



Is there a way I can determine the cause behind this problem using line stats?
Alternatively is it worth contacting my ISP? My only fear is they might throttle the line or something further.
Logged

j0hn

  • Kitizen
  • ****
  • Posts: 3449
Re: High packet loss and occasional disconnection under load
« Reply #7 on: October 28, 2019, 11:35:45 AM »

You need to try another router to rule out the pfsense box as the culprit.

It's not uncommon for some hardware running pfsense to struggle at much higher throughput rates.

If/when you can confirm the same issue on another router (did Zen send you 1?) then contact the ISP.
Logged
BT FTTP 160/30 - BQM - speed test

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 6443
Re: High packet loss and occasional disconnection under load
« Reply #8 on: October 28, 2019, 12:14:09 PM »

The logical course of action to confirm pfsense is the cause is to temporarily use another router.  The behaviour did actually worsen with the changes, so I expect pfsense is the culprit here.
« Last Edit: October 28, 2019, 04:12:49 PM by Chrysalis »
Logged
AAISP - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

leonroy

  • Member
  • **
  • Posts: 22
Re: High packet loss and occasional disconnection under load
« Reply #9 on: October 28, 2019, 12:25:59 PM »

Heheh, I asked Zen to keep the free router since I had a pfSense router (waste not and all that!). Guess I better tell them I'll need them to send one after all  :D
Logged

dee.jay

  • Reg Member
  • ***
  • Posts: 483
Re: High packet loss and occasional disconnection under load
« Reply #10 on: October 28, 2019, 01:12:32 PM »

I'd be surprised if it is your router, I looked up that board, an octa core Atom, whilst no Xeon - it's still pretty powerful and should be more than capable of handling 330Mbps.

I get could 1Gbps out of an older Atom CPU.

On the console of your pfSense machine you can run iperf - have you tried this to prove the throughput on your local LAN?
Logged
Sky + AAISP FTTC ~ 130/34 combined @ 3dB HG612's
Routed by pfSense on VMware ESX 6.7 on Ryzen 3 3200

leonroy

  • Member
  • **
  • Posts: 22
Re: High packet loss and occasional disconnection under load
« Reply #11 on: October 28, 2019, 02:41:57 PM »

@dee.jay - this is weird...

So I have two sites, both with pfSense and connected via 1GbE. They've often run file transfers across without ever an issue before. Just ran iperf between them. I also connected an iperf client to the affected gateway to further load it. I also started a ping out to 8.8.8.8 and to the gateway to verify LAN and WAN connectivity.

Interestingly I see dropped packets on both LAN and WAN:

LAN
Code: [Select]
64 bytes from 10.1.0.1: icmp_seq=5 ttl=64 time=0.646 ms
64 bytes from 10.1.0.1: icmp_seq=6 ttl=64 time=0.613 ms
64 bytes from 10.1.0.1: icmp_seq=7 ttl=64 time=1.183 ms
Request timeout for icmp_seq 8
64 bytes from 10.1.0.1: icmp_seq=8 ttl=64 time=1488.186 ms
64 bytes from 10.1.0.1: icmp_seq=9 ttl=64 time=487.726 ms
64 bytes from 10.1.0.1: icmp_seq=10 ttl=64 time=0.674 ms

WAN
Code: [Select]
64 bytes from 8.8.8.8: icmp_seq=41 ttl=54 time=4.419 ms
64 bytes from 8.8.8.8: icmp_seq=42 ttl=54 time=3.936 ms
64 bytes from 8.8.8.8: icmp_seq=43 ttl=54 time=6.613 ms
Request timeout for icmp_seq 47
Request timeout for icmp_seq 48
64 bytes from 8.8.8.8: icmp_seq=44 ttl=54 time=5207.083 ms
64 bytes from 8.8.8.8: icmp_seq=45 ttl=54 time=4207.624 ms
64 bytes from 8.8.8.8: icmp_seq=46 ttl=54 time=3207.687 ms

To reiterate I'm not sending any traffic on WAN at all.

After running the iperf test for a few minutes both LAN and WAN have gone down on the pfSense box.



Guess the issue is definitely a pfSense one...
« Last Edit: October 28, 2019, 02:49:17 PM by leonroy »
Logged

dee.jay

  • Reg Member
  • ***
  • Posts: 483
Re: High packet loss and occasional disconnection under load
« Reply #12 on: October 28, 2019, 03:09:56 PM »

Yeah, there is definitely something up with your hardware - the throughput 600MBps shows it's capable of actually pushing that much.

Is it running bare metal / as a VM?

Are the NIC's on board?
Logged
Sky + AAISP FTTC ~ 130/34 combined @ 3dB HG612's
Routed by pfSense on VMware ESX 6.7 on Ryzen 3 3200

Chrysalis

  • Content Team
  • Addicted Kitizen
  • *
  • Posts: 6443
Re: High packet loss and occasional disconnection under load
« Reply #13 on: October 28, 2019, 04:16:11 PM »

dee jay I have seen pfsense do this before, it happened on my old pfsense unit.  So thats why I think its a possibility.

Ironically the best I have seen pfsense perform (and opnsense) is on VM's as virtual nic drivers have been very well developed and tested.
Logged
AAISP - Billion 8800NL bridge & PFSense BOX running PFSense 2.4 - ECI Cab - LINE STATISTICS CLICK HERE

dee.jay

  • Reg Member
  • ***
  • Posts: 483
Re: High packet loss and occasional disconnection under load
« Reply #14 on: October 28, 2019, 04:18:30 PM »

Maybe it's time for a new pfSense install.

I run a virtual instance, it's been faultless.
Logged
Sky + AAISP FTTC ~ 130/34 combined @ 3dB HG612's
Routed by pfSense on VMware ESX 6.7 on Ryzen 3 3200
Pages: [1] 2
 

anything