Kitz Forum
Broadband Related => FTTC and FTTP Issues => Topic started by: Ronski on April 05, 2020, 10:09:58 AM
-
Since January we seem to be having periods of high packet loss, this is noticeable as websites occasionally don't load, and my daughters complain the wi-fi has gone off. When we get the issues if I look in pFsense then it shows a very high RTT, lots of loss and states the connection is down or has latency (see attached). It's not completely down, as the modems not dropped the connection, but the pings are very high, unless very bad speed tests still run and record good speeds.
The test immediately before this one actually recorded a high ping and 0/0Mbps speeds, but didn't get saved to my results
(https://www.speedtest.net/result/9239036671.png) (https://www.speedtest.net/result/9239036671)
21/01/2020
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/331ba3f917b35a9c72e17463c86f7388ebf2860f-21-01-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/331ba3f917b35a9c72e17463c86f7388ebf2860f-21-01-2020)
04/04/2020
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/bb7eb48e95fa628a6e87e5ffc8db9f08d3c010c0-04-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/bb7eb48e95fa628a6e87e5ffc8db9f08d3c010c0-04-04-2020)
Pinging bbc.co.uk [151.101.128.81] with 32 bytes of data:
Reply from 151.101.128.81: bytes=32 time=427ms TTL=58
Reply from 151.101.128.81: bytes=32 time=13ms TTL=58
Reply from 151.101.128.81: bytes=32 time=440ms TTL=58
Reply from 151.101.128.81: bytes=32 time=493ms TTL=58
I have asked another user who's just down the road from me and on Virgin, and he's not had any complaints from his family, who've been hammering the connection recently, he's going to set up a ping monitor to see what the connections like.
In another thread d2d4j suggested running a pathping, so here's what I get
pathping bbc.co.uk
Tracing route to bbc.co.uk [151.101.64.81]
over a maximum of 30 hops:
0 MY-PC [192.168.0.11]
1 pfSense.changed.co.uk [192.168.0.1]
2 10.213.48.1
3 brig-core-2a-xe-830-0.network.virginmedia.net [80.3.65.181]
4 * * *
Computing statistics for 75 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 MY-PC [192.168.0.11]
0/ 100 = 0% |
1 0ms 0/ 100 = 0% 0/ 100 = 0% pfSense.changed.co.uk [192.168.0.1]
9/ 100 = 9% |
2 365ms 98/ 100 = 98% 89/ 100 = 89% 10.213.48.1
0/ 100 = 0% |
3 186ms 9/ 100 = 9% 0/ 100 = 0% brig-core-2a-xe-830-0.network.virginmedia.net [80.3.65.181]
Trace complete.
That was run when things were working OK but my ping was a little high, but numbers 2 & 3 don't look good, and suggest the problem is on Virgins network??
And this
tracert bbc.co.uk
Tracing route to bbc.co.uk [151.101.192.81]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms pfSense.changed.co.uk [192.168.0.1]
2 592 ms 536 ms 587 ms 10.213.48.1
3 577 ms 549 ms 492 ms 80.3.65.221
4 * * * Request timed out.
5 24 ms 19 ms 19 ms eislou2-ic-3-ae0-0.network.virginmedia.net [94.174.238.226]
6 16 ms 25 ms 16 ms 157.52.127.6
7 16 ms 15 ms 17 ms 151.101.192.81
Trace complete.
I watched over two hours of Netflix last night & Friday night, didn't have any issues, so I'm guessing it's buffered enough to keep playing with out problems.
Is this something on the Virgin network, or an issue on my network given the other user I've asked doesn't appear to have any issues?
-
Hi ronski
Many thanks
Yes 2 and 3 don’t look good but could be edge dropping pings if busy servicing other things
I would shutdown your pfsense, and vm modem, wait 2 minutes (with power lead unplugged from power socket completely), and then turn on the vpn modem and pfsense
Test - is this any better
Many thanks
John
-
Hi
Sorry I would change pfsense dns servers to 1.1.1.1 and 8.8.4.4
Many thanks
John
-
Thanks John, currently using 9.9.9.9 and 149.112.112.112
I will try your suggestions now.
-
Unfortunately neither made any difference, currently RTT and RTTsd are fluctuating into the 200 to 400ms range.
-
Hi ronski
Many thanks
Could I ask if you unplugged power lead as well (there is a reason for this - allows stacks to fully reset to zero)
Is your first hop (10.) still the same as hops past that ping fine from here
If so, I would report to VM as it’s only vm who can fix
Many thanks
John
-
I did initially forget to unplug the power leads, so did it again, this time unplugging the power leads.
Not sure what you mean by "Is your first hop (10.) still the same as hops past that ping fine from here"
Looks like I'll have to put the modem back to router mode, and run it like that for a while, as I know full well Virgin will ask me to do that. Was hoping not to have to deal with their support again, will use the forum.
-
Realised what you meant
tracert bbc.co.uk
Tracing route to bbc.co.uk [151.101.128.81]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms pfSense.changed.co.uk [192.168.0.1]
2 10 ms 9 ms 13 ms 10.213.48.1
3 21 ms 11 ms 13 ms brig-core-2a-xe-833-0.network.virginmedia.net [80.3.65.193]
4 * * * Request timed out.
5 13 ms 13 ms 15 ms eislou2-ic-2-ae0-0.network.virginmedia.net [62.254.84.62]
6 16 ms 14 ms 15 ms 157.52.127.8
7 14 ms 15 ms 16 ms 151.101.128.81
Need to try it really when things are being problematic
-
Hi ronski
Many thanks and sorry, should have been clearer sorry
The pings looks fine on the last test
Have you changed modem settings. If not, I would leave and periodically test as it appears intermittent
I am thinking the 10. Is your gateway so if so, it should affect all users using that gateway, which hopefully vm should be aware and fix
If checking with neighbours, could you ask if they use same first hop 10.
Many thanks
John
-
The neighbours will use the same router. If on a different cabinet a slim chance the immediate neighbours will use a different port but VM kit is pretty geographic in separation much as Openreach kit is even if you may not be able to see that on traceroute.
-
So currently back to fiddling with the broadband. This morning I switched the modem back to router mode, and therefore took pfSense out of the equation
This is in modem mode and pfSense
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/daec6c8c80f89afed2940c37cf97c65d15a57394-11-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/daec6c8c80f89afed2940c37cf97c65d15a57394-11-04-2020)
This is in router mode, no pfSense, and so far it looks much better.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/613f39fdf3a42ae4326358816aa77c894a909763-11-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/613f39fdf3a42ae4326358816aa77c894a909763-11-04-2020)
My IP address has changed, but everything else seems the same:
pathping bbc.co.uk
Tracing route to bbc.co.uk [151.101.128.81]
over a maximum of 30 hops:
0 MY-PC [192.168.0.11]
1 192.168.0.1
2 10.213.48.1
3 brig-core-2b-xe-832-0.network.virginmedia.net [80.3.65.221]
4 * * *
Computing statistics for 75 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 X99-PC [192.168.0.11]
0/ 100 = 0% |
1 1ms 0/ 100 = 0% 0/ 100 = 0% 192.168.0.1
0/ 100 = 0% |
2 8ms 98/ 100 = 98% 98/ 100 = 98% 10.213.48.1
0/ 100 = 0% |
3 10ms 0/ 100 = 0% 0/ 100 = 0% brig-core-2b-xe-832-0.network.virginmedia.net [80.3.65.221]
Trace complete.
C:\tracert bbc.co.uk
Tracing route to bbc.co.uk [151.101.128.81]
over a maximum of 30 hops:
1 1 ms 1 ms 2 ms 192.168.0.1
2 9 ms 9 ms 7 ms 10.213.48.1
3 11 ms 11 ms 11 ms brig-core-2b-xe-832-0.network.virginmedia.net [80.3.65.221]
4 * * * Request timed out.
5 23 ms 21 ms 23 ms eislou2-ic-3-ae0-0.network.virginmedia.net [94.174.238.226]
6 18 ms 17 ms 17 ms 157.52.127.6
7 18 ms 19 ms 18 ms 151.101.128.81
Trace complete.
I'll see what it looks like tomorrow morning.
-
So 24 hours later, and I still have a very good ping graph, and no complaints about the broadband.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/e6d4e39c8603b0b238ec71c042c900b68cebaf7e-12-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/e6d4e39c8603b0b238ec71c042c900b68cebaf7e-12-04-2020)
So where do I go from here in tracking down the issue?
As I'm not having issues in router mode is it definitely something in my setup causing it, or could it still be something on Virgins network (different routing perhaps)?
Edited to add, this is Friday's graph when running in modem mode and pfSense as the router
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/ac787807ebfc72353d58d3bc4c41bc12929d30fd-10-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/ac787807ebfc72353d58d3bc4c41bc12929d30fd-10-04-2020)
-
Hi
Have you looked in the pfSense various system logs to see if there are any errors reported or perhaps something constantly restarting (DNS Resolver perhaps). Have you recently updated to pfSense 2.4.5 which is causing problems for some along these lines.
Regards
Phil
-
Hi Phil,
No I haven't looked in any logs, given VMs history it was expected to be them at fault and I deliberately hadn't updated to 2.4.5, IIRC I'm on v 2.4.4
I'll switch back to modem mode and pfSense and have a look at the logs, at some point today or tomorrow.
Thanks
-
Hi
Have you looked in the pfSense various system logs to see if there are any errors reported or perhaps something constantly restarting (DNS Resolver perhaps). Have you recently updated to pfSense 2.4.5 which is causing problems for some along these lines.
Regards
Phil
I had DNS resolver crash just today, but that's been an ongoing thing with pfSense sometimes, not specific to 2.4.5. If anything, 2.4.5 was a huge improvement on older versions, its so much faster.
What does tend to trip up pfSense is packet loss and high pings in general, as it will mark the gateway as down and restart the firewall. Its why I found hooking up 4G to pfSense unusable, as when one link goes down, the whole thing goes down as the firewall reloads.
It may be you need to fiddle with the gateway settings so its more tolerant of high latency. System, Routing, Edit your WAN gateway, Advanced settings, Latency/Packet loss thresholds. Also using a different Monitor IP might help, if the default one is not responding to ICMP well. But this is assuming you are seeing it reporting high latency/packet loss.
This is where DNS Resolver is also prone to fail, it occasionally does not restart correctly with the firewall. Heck, even the firewall itself sometimes glitches as I find my load balancing rules are missed and I get stuck with routing out only one ISP. I did kinda assume maybe it was my unique configuration, but it does suggest its something that can happen.
Its not a huge problem in general, but I can see how having just the wrong scenario it could be.
-
I had DNS resolver crash just today, but that's been an ongoing thing with pfSense sometimes, not specific to 2.4.5. If anything, 2.4.5 was a huge improvement on older versions, its so much faster.
There is an ongoing issue (https://forum.netgate.com/topic/115482/frequent-unbound-restarts) with the DNS Resolver (unbound) when 'Register DHCP leases in the DNS Resolver' is checked, as each renewal of a lease causes the DNS Resolver to restart, and during the time it is restarting there is some extra latency. In my case when I switched to IPv6 there was suddenly a lot more DHCP traffic noise that was triggering the resolver to restart every few seconds that simply didn't happen on IPv4 only.
The Gateway issue is a valid one and so it will be worth Ronski checking the Gateway system logs for issues there.
Regards
Phil
-
There is an ongoing issue (https://forum.netgate.com/topic/115482/frequent-unbound-restarts) with the DNS Resolver (unbound) when 'Register DHCP leases in the DNS Resolver' is checked, as each renewal of a lease causes the DNS Resolver to restart, and during the time it is restarting there is some extra latency. In my case when I switched to IPv6 there was suddenly a lot more DHCP traffic noise that was triggering the resolver to restart every few seconds that simply didn't happen on IPv4 only.
Yes, its always been recommended to not use that option and assign any hosts you want to resolve a static IP.
I think its a pretty bad idea to allow clients to pick their own resolvable hostname anyway.
IPv6 really adds a curve ball, because not all clients let you assign them static addresses and its one reason I'm keeping it turned off an the LAN until its absolutely essential and hopefully all clients behave sensibly by then.
-
I think its a pretty bad idea to allow clients to pick their own resolvable hostname anyway.
Why ;D If you don't trust the client to pick a suitable hostname then what are they doing on your network fall-stop?
IPv6 really adds a curve ball, because not all clients let you assign them static addresses and its one reason I'm keeping it turned off an the LAN until its absolutely essential and hopefully all clients behave sensibly by then.
Why do you need everything to have a static IP address, certainly little reason for a static IPv6 unless you are serving a website or similar. Just give them a static IPv4 if you need some sort of host name resolution internally and let them do their own thing with an IPv6 address and move with the times, IPv6 has only been out for 20 years already :lol:
Regards
Phil
-
Thank you both for your replies.
I've had a look at the logs but there nothing that really jumps out at me, probably because I don't really know what I'm looking at.
One thing I have noticed is that whenever I wake up my office PC it causes a bought of latency - I tested this by running a ping from Fing on my phone, it was fine until my PC woke up.
This corresponds with the logs below at 16:28 and 17:50
The Gateway often does report latency, or that it's down - I've disabled gateway monitoring to see if that has any effect. I thought I had previously told it not to monitor 8.8.4.4, but it seems I probably didn't apply the changes or something.
Gateways
Apr 14 17:50:50 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Clear latency 181769us stddev 200885us loss 16%
Apr 14 17:50:20 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Alarm latency 251851us stddev 213510us loss 21%
Apr 14 16:28:21 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Clear latency 93324us stddev 148846us loss 12%
Apr 14 16:27:32 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Alarm latency 123820us stddev 149520us loss 21%
Apr 14 12:53:33 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Clear latency 205517us stddev 243997us loss 15%
Apr 14 12:53:01 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Alarm latency 274140us stddev 246650us loss 21%
Apr 14 12:23:36 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Clear latency 287337us stddev 249855us loss 16%
Apr 14 12:23:15 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Alarm latency 340286us stddev 228988us loss 21%
Apr 14 09:33:14 dpinger VIRGINMEDIA_DHCP 8.8.4.4: Clear latency 249639us stddev 219601us loss 19%
It also had the same effect when starting a different PC, but no entries in the above log.
'Register DHCP leases in the DNS Resolver' was checked so I have unchecked this for the moment as well.
It doesn't look like DNS rolver restarts are my problem, as there's nothing list after 10:22 this morning.
DNS Resolver Log Entries
Apr 13 10:22:34 unbound 87991:0 notice: Restart of unbound 1.9.1.
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 3: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 2: requestlist max 2 avg 1.33333 exceeded 0 jostled 0
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 2: 3 queries, 0 answers from cache, 3 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 1: 1 queries, 0 answers from cache, 1 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 0: requestlist max 19 avg 13.82 exceeded 0 jostled 0
Apr 13 10:22:34 unbound 87991:0 info: server stats for thread 0: 100 queries, 0 answers from cache, 100 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Apr 13 10:22:34 unbound 87991:0 info: service stopped (unbound 1.9.1).
Apr 13 10:22:34 unbound 87991:0 info: start of service (unbound 1.9.1).
Apr 13 10:22:34 unbound 87991:0 notice: init module 1: iterator
Apr 13 10:22:34 unbound 87991:0 notice: init module 0: validator
The pppoe0 interface errors look a little odd, as that interface is disabled so why is it trying to get an IP address?
Routing Log Entries
Apr 14 18:58:00 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 18:43:29 miniupnpd 46112 upnpevents_processfds: 0x801c15100, remove subscriber uuid:d55e1653-7e7f-11ea-9d08-004243ad0314 after an ERROR cb: http://192.168.0.11:2869/upnp/eventing/rvptmhmbsp
Apr 14 18:43:29 miniupnpd 46112 upnp_event_recv: recv(): Connection reset by peer
Apr 14 18:43:29 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 16:36:18 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 16:27:19 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 12:06:12 miniupnpd 46112 SoapMethod: Unknown: GetPortMappingNumberOfEntries urn:schemas-upnp-org:service:WANIPConnection:1
Apr 14 12:06:12 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 04:57:19 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 04:47:16 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 14 04:31:15 miniupnpd 46112 upnpevents_processfds: 0x801c15100, remove subscriber uuid:c74182b5-7e08-11ea-9d08-004243ad0314 after an ERROR cb: http://192.168.0.11:2869/upnp/eventing/ckkmjuoyes
Apr 14 04:31:15 miniupnpd 46112 upnp_event_recv: recv(): Connection reset by peer
Apr 14 04:31:15 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 13 20:45:42 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 13 20:36:42 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 13 20:18:12 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 13 20:08:41 miniupnpd 46112 Failed to get ip address for interface pppoe0
Apr 13 20:00:11 miniupnpd 46112 upnpevents_processfds: 0x801c15080, remove subscriber uuid:61ec7033-7dc1-11ea-9d08-004243ad0314 after an ERROR cb: http://192.168.0.11:2869/upnp/eventing/rqpxpugbah
I'll leave it at that for tonight and see if anything improves, or if anyone has anything more specific let me know, although I probably won't get chance to look until tomorrow night.
-
Hi
You could try using pfTop and see exactly what your PC is doing, although if connected over Wi-Fi and you ping from Wi-Fi it would show some latency anyway but shouldn't be enough to trigger an alarm on the Gateways.
So Diagnostics -> pfTop add to filter expression src <your computer IPv4> then turn off your PC, wait for pfTop to show the connections closing, then boot up your PC and see if anything starts immediately pulling down or pushing up loads of traffic. You can see the destination IP and can usually Google those or use a WhoIs tool online and find who they belong to and see if anything looks suspicious.
Regards
Phil
-
Thanks Phil, nothing looks suspicious, I also installed Glasswire the other day, and nothing looks odd there either, and if it was this PC causing it I would also have had problems when the using the SH3 as a router.
Also this PC is asleep most of the time, or should be - looking in the logs it's certainly been off all day.
After the changes I made yesterday the ping graph does look better, but nowhere near as good as when using just the SH3
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/7982cdf32cd8598e16131e037115b96ecca0ee50-15-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/7982cdf32cd8598e16131e037115b96ecca0ee50-15-04-2020)
-
Hi Ronski
I would test with upnp disabled (fully) to see if it makes any difference (note your log shows a soap failure for upnp) and restart pfsense (i prefer a full restart when problem solving)
Also, what is pathping bbc.co.uk showing (before and after upnp disabled and pfsense restart)
You may be interested to note the following on VM forums, for exactly the same issue from what I can read and if so, it reads as though it is a SH3 firmware issue. Note the date 10 Feb 2020 - but you SH3 may have recently been firmware updated
Perhaps try to check if a newer formware is available past the 10 feb 2020
Many thanks
John
Re: High Ping Spikes
a month ago
The Puma 6 issue affected latency in both Modem and Router mode, it was modem mode all those years ago when I first noticed it coming from a SH2AC. Also, the SH3 has Puma 6, the Hub4 and Hitron CGNV4 uses Puma 7, which incidentally also sucks.
When reading the 'Gaming' portion of these forums it would appear as though many people are reporting serious latency issues since the 10th Feb firmware update; one particular person managed to get a SH3 replaced with one using an older firmware and it worked fine, until it took another firmware update.
PFSense can run on a potato, and it is incredibly unlikely that all of us using pfsense, opnsense and the like are suddenly experiencing simultaneous hardware/performance issues.
Phil
https://community.virginmedia.com/t5/Speed/High-Ping-Spikes/td-p/4170642/page/3
[Moderated edited to insert [quote][/quote] tags around the above quotation.]
-
Thanks John, certainly sounds very similar - my issues appeared to start 21 January - see first post. I still have a Zyxel router, so I'll switch pfSense out and try that at some point - probably the weekend.
-
So currently back to fiddling with the broadband. This morning I switched the modem back to router mode, and therefore took pfSense out of the equation
This is in modem mode and pfSense
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/daec6c8c80f89afed2940c37cf97c65d15a57394-11-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/daec6c8c80f89afed2940c37cf97c65d15a57394-11-04-2020)
This is in router mode, no pfSense, and so far it looks much better.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/613f39fdf3a42ae4326358816aa77c894a909763-11-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/613f39fdf3a42ae4326358816aa77c894a909763-11-04-2020)
My IP address has changed, but everything else seems the same:
pathping bbc.co.uk
Tracing route to bbc.co.uk [151.101.128.81]
over a maximum of 30 hops:
0 MY-PC [192.168.0.11]
1 192.168.0.1
2 10.213.48.1
3 brig-core-2b-xe-832-0.network.virginmedia.net [80.3.65.221]
4 * * *
Computing statistics for 75 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 X99-PC [192.168.0.11]
0/ 100 = 0% |
1 1ms 0/ 100 = 0% 0/ 100 = 0% 192.168.0.1
0/ 100 = 0% |
2 8ms 98/ 100 = 98% 98/ 100 = 98% 10.213.48.1
0/ 100 = 0% |
3 10ms 0/ 100 = 0% 0/ 100 = 0% brig-core-2b-xe-832-0.network.virginmedia.net [80.3.65.221]
Trace complete.
C:\tracert bbc.co.uk
Tracing route to bbc.co.uk [151.101.128.81]
over a maximum of 30 hops:
1 1 ms 1 ms 2 ms 192.168.0.1
2 9 ms 9 ms 7 ms 10.213.48.1
3 11 ms 11 ms 11 ms brig-core-2b-xe-832-0.network.virginmedia.net [80.3.65.221]
4 * * * Request timed out.
5 23 ms 21 ms 23 ms eislou2-ic-3-ae0-0.network.virginmedia.net [94.174.238.226]
6 18 ms 17 ms 17 ms 157.52.127.6
7 18 ms 19 ms 18 ms 151.101.128.81
Trace complete.
I'll see what it looks like tomorrow morning.
is it one of the intel puma devices? I remember they had an issue in bridge mode with latency.
Also if possible try another router behind it in bridge mode to see if pfsense is the culprit.
-
The Puma issue was fixed, or worked around some time ago.
I have planned on trying another router at the weekend.
-
I'm using the SH3 in modem mode with no such issues so it isn't necessarily the SH3.
-
Hi Ronski
If you have QoS enabled in pfSense you could set up pings from thinkbroadband to a real-time queue, this could help rule out pings being delayed due to stuff happening on your network or inside pfSense, another alternative is to push the pings to another computer to respond. Other things you may have already tried but different network cables, even if the same ones appear to work connected to different equipment, always worth trying other ones to rule them out. Other options perhaps is to try and reset pfSense configuration (back up first so you can easily restore), and see if issues still exist with a default and lightly configured pfSense.
Regards
Phil
-
Or just remove Pfsense from the loop.
You get issues in modem mode with pfsense
You get no issues in router mode
You need to try modem mode without pfsense, that's the next logical step.
Swap it for another router and see if behaviour continues.
-
Thanks for the replies.
I used a different network cable between pfSense and the switch yesterday (which made no difference), the one between pfSense and the hub is OK as I used that when running the hub in router mode.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/0632470353c018918b821eff461651a739bcea70-18-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/0632470353c018918b821eff461651a739bcea70-18-04-2020)
I have now got my old VMG8324-B10A out the loft and after a reset managed to get it working, so we shall see how that goes over the next 24 hours.
-
Remember to enable ICMP on the management page as it looks off on your BQM
-
Thanks John, the one above is for pfSense/SH3 Modem mode, Zyxel/SH3 Modem mode has a different IP so had to set up a new monitor. I did remember I had to enable ICMP on the Zyxel but it took a bit of Googling to find out how to - as shown by the width of the red line below.
Early hours, but things are certainly looking better, which means it looks like it is something to do with pfSense.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/4e2944fef4310373b3d2d88f9a8be93daeb233b7-18-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/4e2944fef4310373b3d2d88f9a8be93daeb233b7-18-04-2020)
-
Hi
This is my chart on pfSense but not using VM. So it isn't an issue as such with pfSense, and so pfSense shouldn't be adding any 'colour' to the chart.
Regards
Phil
-
So after 24 hours on the Zyxel VMG8234-B10A and SH3 in modem mode we have the following
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/ef222480a7f0a069eb577a474500665a8b30c832-19-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/ef222480a7f0a069eb577a474500665a8b30c832-19-04-2020)
This is Friday using pfSense and SH3
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/56b2538aacc92e19fd650f2eff359f73341a51a1-17-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/56b2538aacc92e19fd650f2eff359f73341a51a1-17-04-2020)
This is the previous Sunday with the SH3 in router mode
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/e6d4e39c8603b0b238ec71c042c900b68cebaf7e-12-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/e6d4e39c8603b0b238ec71c042c900b68cebaf7e-12-04-2020)
Last night I watched 2 hours of Netflix (7 - 9pm), the same on the other two charts, strange how last night had no extra latency at that time, also odd why I've constantly got so much yellow on all three setups.
Given the above I've come to the conclusion it's something to do with pfSense, and the minor adjustments I made after Easter did improve things a little.
Still at least they all look better than this one from late January, every day looks pretty much like below right through until just after Easter when I made changes as documented earlier in the thread.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/2370f794e5047c1deeece870ecb68d2f6fffb17f-25-01-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/2370f794e5047c1deeece870ecb68d2f6fffb17f-25-01-2020)
I'm going to switch back to pfSense and do some more fiddling.
-
Do you have periods of high activity on your network? pfsense has had reports where it handles high network loads badly, but if your connection's mostly idle its a head scratcher, and I am very curious if you find the problem, on what it is.
Possible places to look.
Set powerD to hiadaptive or maximum.
disable hardware TCP segment offload and large receive offload. If it uses a realtek nic also disable checksum offload.
-
also odd why I've constantly got so much yellow on all three setups.
I've never seen the Virgin hub do anything else to be honest.
At least you know the yellow peaks are down to Pfsense.
-
Thanks John, perhaps the yellow low level latency is just a VM thing.
Back on pfSense now.
I've always run pfSense with the whole world pretty much blocked, everywhere except the UK is blocked, unless I'm out of the country then I allow the country I'm visiting as well.
Now pfBlockerNG states:
"It's also not recommended to block the 'world', instead consider rules to 'Permit' traffic from selected Countries only. Also consider protecting just the specific open WAN ports and it's just as important to protect the outbound LAN traffic."
The above makes total sense, but I could never work out how to just permit the UK rather than block everything else, so have always blocked the world and temporarily allowed any countries I'm visiting, which is not often these days.
I've been using the above approach ever since I've been using pfSense, so I doubt its that causing the issue, and the system certainly doesn't seem stressed with CPU usage around 2-5%
-
Ronski if you operate the firewall on a default deny basis (which you really should be doing on your WAN, and is also the default configuration), then you simply create allow rules based on the UK geoip. The rest is then blocked by the default deny rule.
-
As far as I know the firewall is on a default deny basis, but I do have open ports for access to my server, VPN and a small website.
The logic says I should have a rule that says if the IP is outside the UK IP addresses then drop the packets, but I don't know how do this, and googling just turns up to many results. This is one reason why I like Draytek routers, much easier to find and clearer examples that I can understand.
-
So basically have the default deny.
In pfblockerng for the geoip list, save it as an alias (alias native).
On the allow rules, for VPN and web server make an allow rule, with the alias as the allowed source. (modify existing rule if you want just adding the alias)
To use the alias, select single host or alias as address type, start typing it out and you should be able to click on it as it will popup.
Here is an example from the destination box, but works same way in source box.
-
Thanks Chrysalis.
So I changed GeoIP to List Action - Alias Native for all the country groups and save it.
A Cron update will be needed to update it.
Then I need to modify the rule as per attached.
But, a couple of questions.
1. In GeoIP I should select just the countries I want to ALLOW
2. In the rule how to I allow multiple countries which are in different groups say UK and America as there only seems the option to enter one alias list?
Thanks for your help - I just don't dabble often enough to get to know my way around.
-
only bother making lists for the countries you need also, will reduce load on your unit.
so yes only the countries you want to allow in geoip.
if you want to allow multiple regions, I dont think you can use multiple alias per rule in the gui, so would need multiple allow rules.
-
Thanks Chrysalis, I've made those changes and most of the world appears to be blocked.
But when I select France in my VPN, which gives me an IP address of 84.17.42.21 I can still connect to my website, do the GeoIP lists need updating or is that automatic? There is two other options under France in the VPN and both those are blocked.
-
Hi ronski
Sorry that ip shows as own in France but used in uk - datacom I think
Many thanks
John
-
Do you have periods of high activity on your network? pfsense has had reports where it handles high network loads badly, but if your connection's mostly idle its a head scratcher, and I am very curious if you find the problem, on what it is.
Possible places to look.
Set powerD to hiadaptive or maximum.
disable hardware TCP segment offload and large receive offload. If it uses a realtek nic also disable checksum offload.
I missed this reply earlier.
Powerd was set to adaptive, now changed to hiadaptive
Hardware TCP segment offload and large receive offload are already disabled
They are Intel Nic's
-
Hi ronski
Sorry that ip shows as own in France but used in uk - datacom I think
Many thanks
John
Thanks John, main purpose is to limit who can try and hack my open ports, so has long as most are blocked I'm happy.
-
Ronski is your geoip database up to date? maxmind now needs a registration code to fetch updates. Latest version of pfblockerng has option to input your private code in there.
-
In that case no it's not, thanks again.
I'll look into it tonight.
-
I've updated a couple of packages, and added a 'Installed Packages' widget to my dashboard, so in future hopefully I'll notice when updates are available :-[.
I've created and applied the Maxmind licence, so presumably the database will update automatically - saw mention of the first Thursday of the month, but presumably as I've just added the licence it will update before then.
-
It seems to be behaving at the moment, thanks to everyone that helped.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/c78789d402c591117c4b7cb69b421b7348f14a38-21-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/c78789d402c591117c4b7cb69b421b7348f14a38-21-04-2020)
Someone on the VM forums (see bottom of page 4 (https://community.virginmedia.com/t5/Speed/High-Ping-Spikes/td-p/4170642/page/4))who also had the same issues is now OK, but apart from trying other routers they didn't make any changes and there ping graph now looks very much like mine, just a bit more low level yellow.
-
Wow - can't believe your issues are identical to the ones I've been facing for the past couple of months.
The only difference with my setup has been I was running pfsense virtualised (although now switched to bare metal for testing).
In particular, I thought I was going mad when I noticed whenever I woke a pc, the virgin connection would ping spike and I'd start getting packet loss. I tried all sorts of permutations to try and isolate the problem (I run quite a complex vlan setup at home), ending up with a very basic install to try and get to the bottom of the problem.
Only yesterday I noticed there are no ping spikes when running the SH3 in router mode, which then got me thinking it may be pfsense and/or a problem with virtualization and pfsense. So I've switched to bare metal pfsense now, and will report back findings. I suspect that factory resetting the hub yday may have had a positive impact? Is it possible they've tweaked the firmware quietly to resolve this issue?
-
I was having issues again yesterday, except I noticed Chrome would load webpages instantly but Firefox would sit there trying to update multiple pages. I tried various things with Firefox and then reinstalled it and it seems to have fixed it. This is certainly different than my previous problem, but it appeared the same until I checked how Chrome responded.
-
Was there anything specific you changed to resolve? I think I saw a post on another forum about disabling DNS pre-fetch - but that is not something I have enabled. I also do not register DHCP leases in DNS Resolver.
I'm thinking of trying a different firewall application (OPNSense maybe?) - but today at least, my BQM seems to be a lot better than it has been over the past few months.
-
I'm not convinced it was one particular setting, or just a result of fiddling, but it has improved. See the link below for the changes I made.
https://community.virginmedia.com/t5/Speed/High-Ping-Spikes/m-p/4226259#M225820
PS Welcome to the forums.
-
Thanks - that's the post I remember seeing. The only option I had not tried was tweaking powerd, so I'll do that now. (EDIT - looks like that is disabled anyway).
As you can see, I've had a couple of spikes this morning (ignore anything before 9pm on this graph - I was trying all kinds of things). The spikes have not been crippling, though, unlike previously (not sure about the one at 1am):
-
I still get the odd spike like you, but it's not noticeable in normal use.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/32bd39e7ba0ca7f9a5b94791232c7004b78406ee-16-05-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/32bd39e7ba0ca7f9a5b94791232c7004b78406ee-16-05-2020)
The graph was much cleaner (see this post (https://forum.kitz.co.uk/index.php/topic,24600.msg414594.html#msg414594)) when using the SH3 in router mode, or the SH3 in modem mode with a Zyxel router, so there's something going on between pfSense and the SH3, but I can't work out what it is.
Our area is currently having FTTP deployed, so I may change to that if prices are good when live, or simply change to a SH4 when they become standard replacements.
-
I'm going to try OPNsense in the next few days - just been trying to replicate all of my firewall rules and setup.
My home network is very much dependent on my router, so I can't easily just switch stuff out. Hopefully, OPNsense doesnt exhibit the same issues (it's also based on FreeBSD).
-
Let me know how you get on, thanks.
-
So it was all looking good over the weekend (only a couple of spikes - and did not cripple my internet). However, today as soon as I jumped on a Zoom call, the spikes / packet loss are back.
During the week, I use Zoom a lot for work, so looks like that is exacerbating it. I'm going to have to try Router mode for a while whilst using Zoom.
-
I've made 2 changes:
Disable Gateway Monitoring Action
Disable "Block bogon networks" on WAN interface
Seems to be positive so far - but it's early days.
This bug report may be relevant: https://redmine.pfsense.org/issues/10414
-
Watching this with interest. Also have SH3 (modem mode) with pfSense (metal). I had no end of troubles when I updated to 2.4.5 so went back to 2.4.4-p3
Things are better but still experience times of packetloss. Not sure if it is pfSense or SH3.
**LIVE**
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/bca28640c8778643e920a9e637f6a17288a67312.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/bca28640c8778643e920a9e637f6a17288a67312)
9th April 2020 (when I first setup BQM)
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/b46cb446df75b624a04b32abe8098e937381032f-09-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/b46cb446df75b624a04b32abe8098e937381032f-09-04-2020)
-
My issues predated 2.4.5, I've since updated from 2.4.4 to 2.4.5.
-
And has that made any difference?
I've finally gotten OPNsense setup correctly. I'm going to try that tomorrow morning. Also just preparing a 2.4.5 install - will try that as well.
Disabling Bogon's and Gateway Monitoring helps somewhat - in that when the packet loss occurs, it subsides much quicker (which ties in with the link I sent about the firewall rules resetting). However, the underlying packet loss issue is still there.
I'd like to try router mode during a work day - I suspect it too will have some issues. The problem is my network setup at home uses the same subnet as the Virgin router, and it's not possible to change the virgin config!! Reconfiguring my own subnet would be very time consuming (but will bite the bullet if neither OPNsense nor 2.5 solve the issue).
Cant believe how much time Ive wasted on this crap.
-
No it didn't make any difference.
-
Thanks for the feedback.
I did manage to get 2.5 working briefly, but it's sooo flaky (as expected of a beta), that I couldn't live with it.
I'm running opnsense now so will feedback whether the issue reoccurs.
-
Used OPNsense all of today and hit the issue only once. Certainly positive, but will run for a few more days before reverting to pfsense to test that again (in case Virgin have fixed something on their side).
-
Glad I haven't seen this, my appliances CPU runs at unholy temperatures at 100% usage, but that only happens briefly when fiddling in the UI.
-
I've been testing this for a number of days now.
I should say I've had a virgin engineer round too who added a noise suppressor and also replaced a section of cable which had multiple joins with one new length. This has no doubt improved things somewhat.
Ran opnsense for a couple of days and had one, maybe 2, instances of packet loss. Switched back to pfsense and things have improved, but still getting far too many spikes.
Disabling the bogon network check means I no longer get a complete loss of connectivity. Only a spike in ping and packet loss for about 15 seconds or so.
No conclusions yet, but a picture paints a thousand words.
Graph from yday (24th May) where I switch back to opnsense at 3.30pm:
This graph from today shows my switch back to opnsense yday at 3.30pm:
Clearly opnsense seems to be far, far better.
-
Interesting, clearly something's amiss with pfsense then.
-
Interesting, clearly something's amiss with pfsense then.
From what I've read, it seems to be a bug in the FreeBSD version they are using so not a quick fix.
-
Could you point me to where you've seen this?
I'm running OPNsense 20.1 (based on FreeBSD 11.2) which is fine.
PfSense 2.4.5 is based on FreeBSD 11.3. Is this the version affected?
If so, rolling back to 2.4.4.-p3 may resolve, as it is based on FreeBSD 11.2.
https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html
-
Could you point me to where you've seen this?
I'm running OPNsense 20.1 (based on FreeBSD 11.2) which is fine.
PfSense 2.4.5 is based on FreeBSD 11.3. Is this the version affected?
If so, rolling back to 2.4.4.-p3 may resolve, as it is based on FreeBSD 11.2.
https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html
As mentioned on the previous page https://redmine.pfsense.org/issues/10414 although looks like they have a temporary fix now.
-
Thank you. I missed the disable smp workaround. I tried the bogon / firewall entries, and it did not resolve the issue for me.
Looks like 2.4.5-p1 is days away from release which should resolve this.
-
Be interesting if it resolves mine, as my issue existed prior to installing 2.4.5
-
I tried disabling smp and also reducing my vCPU count to 1.
I ran the tests in the last post here: https://redmine.pfsense.org/issues/10414, and I dont see a CPU spike. Which confirms that the issue they describe has indeed been solved. But...
Within one hour, I got a ping spike / packet loss and gateway disconnect. So I dont think this is the same issue.
I've now reverted back to OPNSense.
Will try once more when 2.4.5-p1 is released, but not hopeful.
-
Within one hour, I got a ping spike / packet loss and gateway disconnect. So I dont think this is the same issue.
The thing with pfSense is, if it sees a big enough ping spike/packet loss it will disconnect the gateway assuming its gone down. This behaviour can be tweaked in System -> Routing -> Click the edit icon next to the default gateway and expand the Advanced options. You're looking for Latency thresholds and Packet Loss thresholds.
This is why I do not have gateway monitoring action enabled at all for my VPN Clients nor the LTE connection, as these things regularly have packet loss and latency spikes, causing the firewall to restart.
-
Yes, already disabled gateway monitoring action. And also disabled monitoring bogon networks (which exacerbates the disconnect).
However, switching back to pfsense, didn't take long to see the packet loss / ping spike.
-
This is driving me mad.
I said yesterday I saw a spike within an hour of using pfsense (with SMP disabled). So I switched back to OPNsense - which I'd been happily running for 3 days with no spikes and no packet loss - and guess what? I saw a spike/packet loss/gateway disconnect within a couple of hours (!).
Coincidence? Perhaps. Maybe some kind of strange cutover issue between the 2 virtual machines (I run them both on an esxi server).
So Im repeating my pfsense test today. This time, I rebooted my esxi box, so it should be a clean run. Will update later.
-
Hi
I've been using pfSense 2.4.5 on metal with 4 cores enabled and not needed any workarounds, so I guess lucky in that respect, but if affected then p1 release is due soon and fixes it. I've no packet loss or latency issues on the BQM graph or gateway drops, but then I'm also not with Virgin media, which on the charts I've seen for them, always look poor and jagged.
As your troubling shooting shows sometimes these things are hard to pin down.
If it is only spikes on pfSense and the dropping of the gateway is resolved are those spikes causing a problem? Remember that the monitoring is only testing pings, which could just be getting a lower priority on pfSense internally in its code than maybe OPNSense. You could try turning on QoS and putting pings into real-time traffic queue to see if that results in a better chart. Also if pfSense is only running on 1 core for the work around, that might be enough to affect the ping responses as well.
Regards
Phil
-
Hi - thanks for the response.
I think this does seem to be very much a Virgin Media specific issue, from what I've been reading.
I do have QoS already turned on to prioritise pings - but you are right, single core performance could be causing an issue.
However, it is quite obvious and specific when this specific issue occurs. You suddenly get a very high ping spike + packet loss and it always lasts a specific amount of time (around 10 seconds).
-
I didn't think I had any issues, but considering the terrible experience I had with trialing Stadia yesterday I'm starting to wonder.
I have seen the odd latency spike, but it doesn't seem frequent enough to be this issue, or is it?
-
Hello!
I just wanted to chime in here and say that I too (since around early Feb) have had issue with pfsense/opnsense and Virgin Media. It was at the point where I've pretty much turned every single damn feature off to try and resolve it, naturally none of this worked.
My symptoms are(were) pretty much exactly as described in this thread.
An example of my shoddy connection can be found here (on a good day).
https://www.thinkbroadband.com/broadband/monitoring/quality/share/3c940bc31113361aa9a14f8d553109a043597235-21-05-2020
And this on a generally bad day, when I'm using it the most.
https://www.thinkbroadband.com/broadband/monitoring/quality/share/127ebfdc885ffbd30e9e7796d467fe34f4412d8d-14-05-2020
Now this is today
https://www.thinkbroadband.com/broadband/monitoring/quality/share/26214d30be6ee15756204e05efee1113f9bbfdcd-29-05-2020
And yesterday
https://www.thinkbroadband.com/broadband/monitoring/quality/share/aaf09c1a9076591555eda9f3d0fb8537910521d6-28-05-2020
So the fix, at least for me was that by default unbound operates in resolver mode, great, that's what we want! Turns out that it completely borks my virgin connection every time I want to use it, and I think this seems to pair with a firmware update that we took around the time this started happening.
So, for the last two days I've been using forwarding mode (with prefetch off) to 1.1.1.1 and OpenDNS and it's been faultless, the only spikes in the graphs are where I've been downloading at my full 350mbps for extended periods for which I expect latency to increase.
Hope this helps some of you guys with Virgin Connections.
Edit: Fixed the images, in my haste I didn't notice they were all the same!
-
Check your post mate. All the images are the same:
https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/7afe1defa532e5c5ee7b6984c474df360d2340ee.png
-
Yes - your pics are all the same. Pls check.
Thanks for the tip - let me try and do the same with unbound.
-
I would like to try this as well, not completely clear on what I need to do though.
Is it just a case of disabling DNS Resolver and enabling DNS forwarder?
Anything else that needs doing?
-
Yeah, just put it in forwarding mode with some decent (non virgin) DNS servers. I also turned off prefetch (opnsense terminology).
I’ve attached a couple of images from opnsense with the appropriate settings on them.
-
Yeah, just put it in forwarding mode with some decent (non virgin) DNS servers. I also turned off prefetch (opnsense terminology).
I’ve attached a couple of images from opnsense with the appropriate settings on them.
Actually it depends, you can use dnsmasq or unbound for forwarding. I can imagine dnsmasq being less problematic as unbound does have its quirks sometimes, it seems to take longer to restart when the firewall reloads for example.
Any particular reason you turned off prefetch? It should make things faster I believe?
-
Actually it depends, you can use dnsmasq or unbound for forwarding. I can imagine dnsmasq being less problematic as unbound does have its quirks sometimes, it seems to take longer to restart when the firewall reloads for example.
Any particular reason you turned off prefetch? It should make things faster I believe?
I’d received a modest improvement in previous testing with my connection quality by turning prefetch off, so it is less about improving DNS performance and more about preventing my virgin connection from constantly falling over.
In an ideal world I’d have it in resolver mode with prefetch enabled but in that world virgin media wouldn’t be my ISP, unfortunately I've got more chance of winning the lottery than getting FTTP in my area.
-
@Kerman19 - Thank you for the tip. I think it may well have resolved the issue. I'll know for sure during the week when I use my connection constantly for Zoom calls.
Easy to switch in pfsense.
1. Go to Services -> DNS Resolver
2. Tick "Enable Forwarding Mode"
3. Optionally enable "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers"
-
Thanks, I've now done that, we#ll see what happens.
There is also a DNS Forwarder service, I thought it was that which was being referred to.
-
@Kerman19 - Thank you for the tip. I think it may well have resolved the issue. I'll know for sure during the week when I use my connection constantly for Zoom calls.
Easy to switch in pfsense.
1. Go to Services -> DNS Resolver
2. Tick "Enable Forwarding Mode"
3. Optionally enable "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers"
I have my fingers crossed for you!
-
It was absolutely fine today. Thanks again for the tip!!
-
Thanks from me also, only a couple of spikes over the last 24 hours
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/032b949079af0152f52ffda4c2a440d73f6b4959-01-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/032b949079af0152f52ffda4c2a440d73f6b4959-01-06-2020)
Compared to Saturday's
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/229f1dbe9c8e3c26371678cef3c34f22e9176c53-30-05-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/229f1dbe9c8e3c26371678cef3c34f22e9176c53-30-05-2020)
-
I dont know what the cause of the issue Ronski discovered is.
But what I will say is this.
A while back (several years now), FreeBSD developers decided to modify PF the firewall imported from OpenBSD to better scale to extra cpu cores, at the time it was considered a great thing to increase PPS performance caps, but then because this was done, no one wanted to merge in future PF updates from OpenBSD as because of this patch, the code was no too different. This standoff situation lasted for multiple years with more and more outstanding bugs in PF/ALTQ left unresolved, then another discussion happened, and it was decided to still not update PF *sigh* but at least the developers are now starting to maintain PF and do FreeBSD bug fixing for it. OpenBSD developers maintain that the version of PF in FreeBSD is now considered an outdated buggy mess.
I have been using FreeBSD since the 4.x days, a long long time. I migrated my servers to PF not long after it was ported over, as I considered it a large step forward from ipfw. However in all the years I have been using PF it has become apparent there is a lot of bugs, many of these are either very minor or can be worked around, some cannot be worked around and are just there, ipv6 has a fair amount of buggy behaviour in FreeBSD PF, one of which PF will not pass on fragmented packets, and when running cloudflare's fragment tester tool, this will be evident if you are behind a PF firewall.
For as long as opnsense pfsense are based on FreeBSD they effectively have to adopt the buggy PF, opnsense has a couple of hardenedbsd developers now on their team, who have been porting over openbsd security features, I would absolutely love for opnsense to move to OpenBSD, and then they would have the proper fixed modern PF, and I think that move alone would drag over a big part of pfsense's userbase to them.
As for bugs that only affect pfsense but not opnsense, a possible explanation is that pfsense do patch parts of the kernel with custom networking code, this kernel source code is no longer publically available (a reason why many devs jumped ship to opnsense), so it is possible these patches have introduced even more bugs than base FreeBSD.
It is on my to do list to migrate to opnsense at home, I no longer use pfsense in datacentres, they all migrated to opnsense a long time ago.
There is also little niggly things that are not bugs but have been implented in opnsense but not pfsense.
So e.g. in opnsense I can block outbound dns requests not directed to my LAN ipv6 dns server, and actively reroute them to the LAN dns server, the same way that I do on ipv4, so basically I enforce all of my LAN to use my firewall DNS server on both stacks. On pfsense this is only possible on ipv4, because even though its in pf command line, I think one one of the lead dev's massively aganst NAT66, NAT46 etc. and even though this is not technically NAT, it is seen as a NAT type feature so the rdr feature is ipv4 only on pfsense.
-
ipv6 has a fair amount of buggy behaviour in FreeBSD PF, one of which PF will not pass on fragmented packets
I thought with IPv6 fragmented packets were a thing of the past and so support is not required by routers? I've been using IPv6 with over 65% of all traffic leaving or arriving from the WAN is IPv6 and no issues here.
It is on my to do list to migrate to opnsense at home, I no longer use pfsense in datacentres, they all migrated to opnsense a long time ago.
I keep wanting to give OpnSense a go and will when I get chance, I like the fact it is all open source.
... dev's massively aganst NAT66, NAT46 etc. and even though this is not technically NAT, it is seen as a NAT type feature so the rdr feature is ipv4 only on pfsense.
I can understand why they are against it as the whole point of IPv6 was to remove the need for this sort of thing and it can break things. At the end of the day they have to decide how much development work they spend on features and if hardly anyone is going to use it or really needs it, priority just gets pushed down and down.
As for the Virgin media chart and all the yellow and blues and spikes, that is down to VM and their network, I don't think I've ever seen one look much different. My BQM chart is below, on pfSense 2.4.5, the only difference, it isn't Virgin Media.
Regards
Phil
-
I dont know what the cause of the issue Ronski discovered is.
But what I will say is this.
A while back (several years now), FreeBSD developers decided to modify PF the firewall imported from OpenBSD to better scale to extra cpu cores, at the time it was considered a great thing to increase PPS performance caps, but then because this was done, no one wanted to merge in future PF updates from OpenBSD as because of this patch, the code was no too different. This standoff situation lasted for multiple years with more and more outstanding bugs in PF/ALTQ left unresolved, then another discussion happened, and it was decided to still not update PF *sigh* but at least the developers are now starting to maintain PF and do FreeBSD bug fixing for it. OpenBSD developers maintain that the version of PF in FreeBSD is now considered an outdated buggy mess.
I have been using FreeBSD since the 4.x days, a long long time. I migrated my servers to PF not long after it was ported over, as I considered it a large step forward from ipfw. However in all the years I have been using PF it has become apparent there is a lot of bugs, many of these are either very minor or can be worked around, some cannot be worked around and are just there, ipv6 has a fair amount of buggy behaviour in FreeBSD PF, one of which PF will not pass on fragmented packets, and when running cloudflare's fragment tester tool, this will be evident if you are behind a PF firewall.
For as long as opnsense pfsense are based on FreeBSD they effectively have to adopt the buggy PF, opnsense has a couple of hardenedbsd developers now on their team, who have been porting over openbsd security features, I would absolutely love for opnsense to move to OpenBSD, and then they would have the proper fixed modern PF, and I think that move alone would drag over a big part of pfsense's userbase to them.
As for bugs that only affect pfsense but not opnsense, a possible explanation is that pfsense do patch parts of the kernel with custom networking code, this kernel source code is no longer publically available (a reason why many devs jumped ship to opnsense), so it is possible these patches have introduced even more bugs than base FreeBSD.
It is on my to do list to migrate to opnsense at home, I no longer use pfsense in datacentres, they all migrated to opnsense a long time ago.
There is also little niggly things that are not bugs but have been implented in opnsense but not pfsense.
So e.g. in opnsense I can block outbound dns requests not directed to my LAN ipv6 dns server, and actively reroute them to the LAN dns server, the same way that I do on ipv4, so basically I enforce all of my LAN to use my firewall DNS server on both stacks. On pfsense this is only possible on ipv4, because even though its in pf command line, I think one one of the lead dev's massively aganst NAT66, NAT46 etc. and even though this is not technically NAT, it is seen as a NAT type feature so the rdr feature is ipv4 only on pfsense.
I feel like some of that story is missing as you do not mention if OpenBSD went multi-core on PF eventually and how did they address it differently to FreeBSD?
-
You right its not the complete story, if it was it would have been a much much longer post.
But as I understand it the core performance of the latest PF is now in magnitudes faster which to some extent cancels out the effect of the multi threading addition to the FreeBSD code. I don't know if they actually added their own core scaling as well as I haven't been following every change. But the PF situation has been one of the biggest political issues on FreeBSD in recent years.
I thought with IPv6 fragmented packets were a thing of the past and so support is not required by routers? I've been using IPv6 with over 65% of all traffic leaving or arriving from the WAN is IPv6 and no issues here.
The PF bug in question probably only affects one use case scenario I can think off, and its a very uncommon scenario, which is dnssec. Which is why you haven't noticed any issues. As I said a lot of these bugs are trivial things that wont break a connection by themselves.
In terms of the fragmentation, basically routers themselves are not supposed to fragment ipv6 packets, however if they receive packets that are already fragmented then its a question if they should forward those packets as normal. PF in FreeBSD silently drops them with no way to change the behaviour other than editing the source code, PF in OpenBSD passes them on, Iptables passes them on. Also I am not 100% sure of all the technical info on this issue, I think its with only certain fragments, but regardless it is an issue that the vast majority of people wont notice.
I can understand why they are against it as the whole point of IPv6 was to remove the need for this sort of thing and it can break things. At the end of the day they have to decide how much development work they spend on features and if hardly anyone is going to use it or really needs it, priority just gets pushed down and down.
PF already supports it, so the code is there, its basically just the UI frontend, NAT on ipv6 is has split the industry, some people want it in, some are strongly against it. There is people who say NAT was a hack to allow ipv4 to gain a decade before ip exhaustion, and that was its only purpose so it should be nowhere near IPv6, there is others who say they have designed their networks around private address space and want to continue doing so in the future. Personally I see "some" merit in NAT, and I think the choice should be down to the administrator of the network, but it's probably only useful for corporate networks and edge cases. There is however certainly a case for allowing redirect to be used in the firewall.
What was interesting though is I found a blog for one of the guys who has worked with the address space for many years, and he really opposes ipv6 NAT, but the article on his blog was an ipv6 NAT how to for iptables, so how did this come about? Basically he ordered a service from a datacentre provider and it was supplied with a ipv6 /128, a single ipv6 address. He ended up finding out there is situations where NAT might be desired, his other option he stated was tunneling, but NAT had the lower performance impact so he used it, and he published the how to. I think it is really silly to give a customer just one ipv6 address, but sadly it does happen, there is providers who operate in that way.
-
You right its not the complete story, if it was it would have been a much much longer post.
But as I understand it the core performance of the latest PF is now in magnitudes faster which to some extent cancels out the effect of the multi threading addition to the FreeBSD code. I don't know if they actually added their own core scaling as well as I haven't been following every change. But the PF situation has been one of the biggest political issues on FreeBSD in recent years.
I'm dubious as pfSense have supposedly been trying to work towards able to route at 10Gig speeds and that's with the multi-threaded model.
-
As for the Virgin media chart and all the yellow and blues and spikes, that is down to VM and their network, I don't think I've ever seen one look much different. My BQM chart is below, on pfSense 2.4.5, the only difference, it isn't Virgin Media
Phil you are missing the point completely, the problem I and many others were having is clearly related to some interaction between pfSense and the SH3 which manifested itself after a firmware update on it back in January/February.
Yes the base yellow in the good graphs is down to Virgins network, but that does not affect the operation of the connection.
This is my connection after the SH3 firmware update, unfortunately I can't seem to access the chart for when it actually changed, but it literally went from like the second graph to the first. This was having a bad affect on our connectively, with my daughters regularly complaining, sometimes the graphs were much worse.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/a7d35111c45b2eca3be8be18404eb1a396a55367-22-02-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/a7d35111c45b2eca3be8be18404eb1a396a55367-22-02-2020)
This is the SH3 in router mode - see the difference?
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/e6d4e39c8603b0b238ec71c042c900b68cebaf7e-12-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/e6d4e39c8603b0b238ec71c042c900b68cebaf7e-12-04-2020)
This is the SH3 in modem mode and using my old Zyxel as a router, again you see the difference from the top graph?
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/ef222480a7f0a069eb577a474500665a8b30c832-19-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/ef222480a7f0a069eb577a474500665a8b30c832-19-04-2020)
Now we are back to SH3 in modem mode, and pfSense with lots of various adjustments, see we still have the peaks all the way to the top of the graph, although clearly a lot better than the first graph.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/b103cfc7d110ec34134818144bb4216bf0455ee5-30-04-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/b103cfc7d110ec34134818144bb4216bf0455ee5-30-04-2020)
Now we have a graph where I changed to DNS Forwarding mode as suggested by another user who was having exactly the same problems as me.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/771793e697bd47d4563518d337dac1b252bdef39-02-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/771793e697bd47d4563518d337dac1b252bdef39-02-06-2020)
I know of at least three other people that have changed to DNS Forwarding mode and it's cured the problem, two here and one over on the Virgin forums
This is the user on VM forums, a graph showing when they were having the problem - ignore the large red chunk.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/4116f9c6684d0dfdede6c7f938f378d3c6f01eae-12-03-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/4116f9c6684d0dfdede6c7f938f378d3c6f01eae-12-03-2020)
And this is now after changing to DNS Forwarding mode.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/83dd3f789d32d97909545e6222f115a428d8923e-03-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/83dd3f789d32d97909545e6222f115a428d8923e-03-06-2020)
-
Hi
Apologies I was reading it as the yellow being the issue still, I'd forgotten about the charts before that and the more service affecting issues.
Regards
Phil
-
Easy done, no problem.
-
Interesting Ronski, if you dont mind lets try this experiment.
The difference between DNS forwarder and DNS resolver, is by default DNS resolver will send uncached lookups direct to authoritive DNS servers, whilst DNS forwarder will send lookups to whatever DNS you have configured as your upstream DNS.
However the unbound DNS resolver can be configured quite easily to act in forwarder mode and I am very curious if when this is in forwarder mode it also stops the excessive spikes, if it does, then we know when your unit is sending lookups directly to authoritive servers it is for some reason causing delays to inbound ping's which given the hardware spec of your unit is very weird, but we would at least narrow it down to that.
So to do this, you would switch back to DNS resolver.
Then in services menu select DNS resolver.
On that page is a tick box for "Enable Forwarding Mode"
Tick that, hit save, and hit apply.
Then check later if it has the same effect. If it has the same effect, I would run in this mode as unbound is newer and has better features than the forwarder service, but it also is useful for diagnosing this problem.
Also on the advanced section of DNS Resolver you can change the logging verbosity, as well which might be useful in showing errors that might be occurring but that will flood your log, so my request is just to switch it to forwarder mode.
Just remembered, also make sure "Register DHCP leases in the DNS Resolver" is unticked. Really that should be unticked by default in my opinion.
When it is ticked, every time a device registers on your DHCP, then unbound will restart to add the Hostname, which is silly, so that box should be unticked, a unbound restart could potentially cause spikes for sure. Especially when using large DNS lists, which I believe you do with pfblockerng right?, so this one may well be your problem. Because with DNS forwarder service on instead of DNS resolver, your filter lists wont be working anymore (they will still work when using DNS resolver in forwarder mode).
-
Hi Chrysalis,
I am using DNS Resolver in forwarding mode, with "Register DHCP leases in the DNS Resolver" unticked.
I thought at first they meant to use the DNS forwarder service, but someone explained in a post on the previous page how to set it up.
-
Chrysalis -
Same as Ronski. I am using unbound in forwarder mode. This has also resolved the problem for me.
FYI:
1. I have UNticked "Register DHCP leases in the DNS Resolver".
2. I have ticked "Register DHCP static mappings in the DNS Resolver".
3. I have also ticked "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers".
-
Chrysalis, just to clarify do you mean to use DNS Forwarder (dnsmasq), rather than DNS Resolver (unbound) in pfsense?
-
Underzone, so the dns forwarder service is dnsmasq.
Unbound is the dns resolver service, but in the dns resolver service settings if you enable forwarding, you will still be using unbound aka dns resolver service, but just in forwarding mode instead of resolver mode, I hope that makes sense.
What this changes is only what happens when it has to go out to the internet to make a lookup. Local caching, and internal dns lists work the same regardless if its in forwarding or resolving mode. I recommend forwarder mode anyway because a busy dns server like cloudflare, google or isp dns, will have way more cache hits due to the amount of traffic they get.
-
I elaborated a bit further on the Virgin Media forums regarding heavy use of UDP on these modems would cause them to fall over either way. I suspect that is why Resolver mode hurts them more than forwarder mode. Incidentally I can force these modems to fall over when I hit them with a decent amount of UDP load; for example loading up a ton of saved tabs in firefox (forcing a lot of DNS requests) from a previous day and setting up a couple of OpenVPN tunnels.
Incidentally I have a 65mbps FTTC connection and a BT FTTP 300/50 connection that I can use as well. When I use my opnsense box in it's current configuration running similar tests to those above I see absolutely zero movement on the latency, the Virgin connection is truly a crock of poo.
Phil
-
Thanks for the info, I use Firefox and anyways have a stack of tabs open, had loads of problems with it recently, now I know why.
-
This is the user on VM forums, a graph showing when they were having the problem - ignore the large red chunk.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/4116f9c6684d0dfdede6c7f938f378d3c6f01eae-12-03-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/4116f9c6684d0dfdede6c7f938f378d3c6f01eae-12-03-2020)
And this is now after changing to DNS Forwarding mode.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/83dd3f789d32d97909545e6222f115a428d8923e-03-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/83dd3f789d32d97909545e6222f115a428d8923e-03-06-2020)
That's me. The first graph above shows the SH3 going offline briefly around 3:30 am, which is when it did the firmware upgrade (I checked the Hub logs the next day). The three weeks prior to that red line (which was when I had the SuperHub replaced) are like the graph to the left of around 3:30am. Everything after that is like the graph to the right.
The first thing I did was turn off the 'add DHCP leases to DNS' entry in the DNS resolver in pfsense. That improved the situtation to this:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/8d040172d488316ca39408961e7b8ea0a1cda7c0-28-05-2020.png)
That was a dramatic improvement, but then a few days ago I ticked the box on the DNS resolver in pfsense to make it operate in 'forwarding' mode. I'm now getting graphs more like this:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/1bbf27677feef3c46472e84fde74b2ae331a33c7-05-06-2020.png)
Still room for improvement, but again, much better.
I should add that the same version of pfsense was running perfectly for around 6 months before the firmware upgrades to the Super Hub 3 started rolling out in around January of this year.
To conclude, there seems to be some sort of interaction between the recent firmware update and the 'resolver' mode of the DNS resolver in pfsense. I'd love to get to the bottom of it if possible.
Andy
-
An additional way to drop the DNS traffic load a little, is on the advanced tab of "dns resolver" settings, enable "serve expired", that is a brilliant feature. I take pride in that I got that added to pfsense gui. This will make DNS lookups a lot more responsive overall. Especially for commonly used services in your household, things like twitter, facebook etc.
-
Andy, welcome to forums.
Chrysalis, I'll take a look at that option over the weekend.
-
Hi
PfSense release 2.4.5p1 is now out, this fixes issues with latency spikes on multi-core CPUs introduced on 2.4.5.
Regards
Phil
-
PfSense release 2.4.5p1 is now out, this fixes issues with latency spikes on multi-core CPUs introduced on 2.4.5.
I'm still on 2.4.4-RELEASE-p3, so unless that also had the issues not sure this is relevant in my case?
Andy
-
Hi Andy
I'm still on 2.4.4-RELEASE-p3, so unless that also had the issues not sure this is relevant in my case?
We spoke about issues with 2.4.5 in this thread, so it was just an update for Ronski and anyone else interested.
Regards
Phil
-
We spoke about issues with 2.4.5 in this thread, so it was just an update for Ronski and anyone else interested.
Ok. I'm seeing the same latency issues on my setup (very similar to his) which seeems to have been dramatically improved by turning off registering DHCP hosts in the DNS resolver, and turning on DNS forwarding in the resolver.
I'd prefer to go back to my original setup if possible, but I'm loathe to risk a pfsense upgrade at the moment!
Andy
-
I upgraded to 2.4.5-p1 yesterday and I still see latency spikes via BQM.
I've enabled DNS resolver forwarding mode to 1.1.1.1 and 1.0.0.1
Will monitor my BQM again. If other FTTP had equally good speeds i'd drop VM instantly. All my alternatives are FTTC with super turd speeds. Aaargg!
**LIVE**
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/bca28640c8778643e920a9e637f6a17288a67312.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/bca28640c8778643e920a9e637f6a17288a67312)
-
Hi
PfSense release 2.4.5p1 is now out, this fixes issues with latency spikes on multi-core CPUs introduced on 2.4.5.
Regards
Phil
Thank you. Updated and re-enabled SMP and multiple cpus.
-
I also had the issues prior to 2.4.5, but will install the update.
-
I'd be interested if those installing the update are able to revert the settings back to true DNS Resolver with DHCP hosts, and still be able to get decent latency plots.
Andy
-
I'm willing to try that, I'll update later and test it.
-
I upgraded to 2.4.5-p1 yesterday and I still see latency spikes via BQM.
That's a perfect graph for VM, you're not going to get better than that, scroll up the page to see a bad graph.
-
I'd be interested if those installing the update are able to revert the settings back to true DNS Resolver with DHCP hosts, and still be able to get decent latency plots.
Andy
Installed this morning, only uncheck, saved and applied "Enable Forwarding Mode" and a brief period of really bad pings then it cleared, I've now enabled forwarding mode, will test it again overnight tonight.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/30e47951eddf41b93b8936b97e900acc69ad5ba5-11-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/30e47951eddf41b93b8936b97e900acc69ad5ba5-11-06-2020)
-
I'd be interested if those installing the update are able to revert the settings back to true DNS Resolver with DHCP hosts, and still be able to get decent latency plots.
Andy
You mean assign DNS to dynamic DHCP clients? (or whatever its called) I believe that's inherently always going to be broken as Unbound has to be restarted every time a new host is added/removed, which takes down DNS for a few seconds.
-
I'd be interested if those installing the update are able to revert the settings back to true DNS Resolver with DHCP hosts, and still be able to get decent latency plots.
Andy
I dont suggest it, some devices phones etc. can be quite often updating dhcp, and you risk dns resolution outages if you try and do a lookup during a dns service restart.
If dhcp hostnames are important to you then setup dhcp static mapping for those devices.
-
With Forwarding mode off I still latency spikes, so I've turned forwarding back on
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/e7c4c0e07d30c41b6fa6608b68c94d7c3e234589-12-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/e7c4c0e07d30c41b6fa6608b68c94d7c3e234589-12-06-2020)
-
With Forwarding mode off I still latency spikes, so I've turned forwarding back on
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/e7c4c0e07d30c41b6fa6608b68c94d7c3e234589-12-06-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/e7c4c0e07d30c41b6fa6608b68c94d7c3e234589-12-06-2020)
Do you actually notice them in use though or just on the graph? They don't really look frequent enough to indicate a problem IMO.
-
No, but then I don't notice anything detrimental having forwarding mode on either.
-
We have to remember, ping is only a rough guide. You can have huge ping latency but real-world performance be fine.
-
I'm well aware of that ;) It certainly wasn't fine when I was having all the problems, it was that bad my daughters were complaining, it's only the tweaks we've made that have improved things. And that final tweak eliminates the last signs of it, with no detrimental effects.
-
I think the only benefits of direct dns mode (forwarding off) is if for privacy concerns you just dont want to use a MITM dns server for fear of logging, or for commercial use as for that direct has more benefits. DNS lookups are not just a simple query for one A record, you have to also resolve NS, and other records, some times queries to glue servers as well, so the performance hit on direct mode can be quite significant with much more dns traffic.
-
There is also the security of getting your results unfiltered from the source, especially domains using DNSSEC.
Even though they added the UI for DNS over TLS, the Netgate staff on the forum always go on about DNS Resolution being the correct way to do DNS.
I can see their logic though, theres always a lot of novices on the forum trying to use Google, Cloudflare and QUAD9, which can give unpredictable results if you randomly get different records back. Full resolution means you aren't trusting someone to not modify the result.
-
well i trust them to serve proper dns response, the trust issue is down to logging, tracking, and out of the box configuration will have you using isp supplied dns forwarding, ip results been different? services like google will have many public facing ips, and using a different dns resolver ip could easily have a different one been picked.
it is normal behaviour for ip records to change frequently on some services.
but yeah if you have trust issues thats a reason to go direct.
-
well i trust them to serve proper dns response, the trust issue is down to logging, tracking, and out of the box configuration will have you using isp supplied dns forwarding, ip results been different? services like google will have many public facing ips, and using a different dns resolver ip could easily have a different one been picked.
it is normal behaviour for ip records to change frequently on some services.
but yeah if you have trust issues thats a reason to go direct.
I don't think that is what they mean, its that services like QUAD9 do filtering, like OpenDNS. So sometimes you might get a filtered result, sometimes the real one, creating an inconsistent experience.
-
I dont know what they supposedly mean given I cannot find any posts of them telling people to not use forwarding, but if you are using a forwarder that filters, then unless you mixing it with another resolver that doesnt filter then it will be consistent. The developers wrote their own dns over tls guide for cloudflare, they wouldnt do that if they didnt support such a configuration.
If someone misconfigures their unit that doesnt make a feature bad or good, thats just simply operator error.
-
I never said it was bad, I said that there is less scope for user error and for DNSSEC a guarantee you are getting unfiltered DNS rather than trusting the upstream supplier to be honest about how they are managing DNS.
What the forums mods pointed out was if you are using DoT to avoid your ISP logging your DNS, you are still trusting the upstream resolver you are using to not log those requests. Their logic being that if you are getting all DNS from the root servers, this is less likely to be an issue as any logging is spread across all those servers so no central place to retrieve all your history from.
I'm honestly not sure which side I'm on in this argument. I kinda liked Cloudflare as my domains are hosted there, but have been using full resolution for a while now. I do think perhaps I've had a few less resolution errors since doing so but its obviously impossible to really test.
-
Well yeah its the same for VPNs really dont trust your isp, so use a VPN, but then of course the VPN provider can log etc.
I have moved between different positions, at one point had DNSSEC resolution enabled as well, I have settled on using cloudflare DNS currently. Going direct DNS doesnt protect you from MITM attacks either, thats why DNSSEC exists but the amount of domains DNSSEC enabled is a pittance and that has its own issues.
Cloudflare dont do trackable logging, unless you think they lying, but one thing I am sure on is I dont think direct DNS resolution is worth it for consumer use, DNS is so performance sensitive.
Ultimately in security there is a bunch of what if's and then there is a bunch of practical situations, if you end up getting your DNS records manipulated, unless its part of a feature like malware/parental filtering on the upstream provider, then its more then likely going to be a compromise on your own network.
Also one thing to consider as well, if your DNS queries are not encrypted, its trivial for an isp to intercept your lookups, try doing DNS encryption direct.
If you are really paranoid, which it sounds like you are, then you could use dnscrypt or something similar to your own private resolver hosted outside of the UK. Which is what I used to do until I realised it really messes up some geo services (they may check DNS server country origin) include getting ip banned from amazon video. After I switched to a mainstream forwarder service I realised I just wasnt that bothered enough anymore.
I think most people prefer a 3rd party service to log instead of their own isp as their isp already will have records with their personal details for billing purposes, so its fairly logical to conclude the further away the logging is from your isp the better.
-
Good job I'm not paranoid ;)
-
Sorry to open an old post but I think I may have this same issue for many months.
Attached is a typical day on my BQM.
I used to have Virgin SH3 Modem mode -> pfSense -> LAN, but every hour or so we used to lose the internet for 60 seconds, very frustrating!
I then switched to Virgin SH3 Router mode -> pfSense -> LAN, no full drop outs anymore but still get these spikes on BQM. If I check my neighbours BQM it is a perfect graph.
If I have understood this thread correctly... having lots of LAN traffic can overload the DNS and cause SH3 some issues and hence when BQM is pinging it, it is then slow to respond and hence you see the spikes. This also can affect outgoing LAN->WAN traffic. By changing to DNS Forwarder, pfSense is caching DNS entries and therefore less load on VM DNS, one step further is not to use VM DNS, then problem goes away?
Is this still the correct way to go now on 2.4.5 p1
Or have I completely missed the point?
I was on 2.4.4 p3, this evening i upgraded to 2.4.5 p1. Then I found this thread while googling.
-
Hi
pfSense caches all DNS requests in DNS resolver mode, so will a PC and most other devices after the first request so there simply isn't that much DNS traffic on a typical network and very few packets.
If you go into pfSense, select pfTop in Diagnostics, and under the filter expression type 'port 53' to view all DNS traffic going in and out, you will see how much traffic there is related to DNS, on my network with lots of devices there can be lots of requests but they are just a few packets and over and done with in a split second before the firewall times them out after 30 seconds or so, and fewer still tend to go out to the wider Internet, with a lot being resolved from the pfSense DNS cache. Compared to all the other traffic going in and out, DNS is just a very tiny fraction of data, and to the router, VM modem and their network, a DNS request is no different to anything else.
So I'm not sure why the forwarder mode would make any difference, however there is a bug with pfSense when the DNS Resolver is enabled in that it keeps reloading if the option 'Register DHCP static mappings in the DNS Resolver' is checked (under Servers - DNS Resolver) this can cause some down time when DNS isn't being resolved and makes pfSense a bit busy for a bit especially where you have other packages installed that also restart because that.
Your chart looks typical really for Virgin Media really, was your neighbours chart also from Virgin Media?
-
Hi,
That chart is not typical of VM.
I'd try making the changes above, enabling forwarder mode in the resolver, and turning off the registration of DHCP addresses. This made a huge difference for me.
Andy
-
Hi,
That chart is not typical of VM.
All the ones I've seen have loads of yellow and blue, I've yet to see one mostly green :-)
There must be something seriously wrong with VMs network if it is going wonky because of the small amount of traffic from DNS requests to authoritative servers.
Regards
Phil
-
@exdirectory your graph doesn't look to bad compared to how mine was at the start of this thread, shame I can't still see them. Anyway the advice in this thread certainly vastly improved my graphs and experience.
This is what mine looks like on a good day without much use.
https://www.thinkbroadband.com/broadband/monitoring/quality/share/7a9583a419a7a17c5a125902d2ec2a7bc4c67481-05-10-2020
And here with some use.
https://www.thinkbroadband.com/broadband/monitoring/quality/share/8c6b0e7cea7c5b1f06a8e28219c8cfe09cd3ca24-07-10-2020
@PhillipD
It isn't so much as Virgins network it was something to do with the combination of pfSense and the Super Hub in modem mode, mine was horrendous and it also affected browsing. Switch the hub to router mode and it was perfect, well as perfect as Virgin can be. I even used a ZyXEL router and the hub in modem mode and it was perfect, switch back to pfSense and it went bad, the changes suggested in this thread vastly improved it. It seemed to be related to a hub firmware update in January /February can't remember the details.
-
I have two different neighbours graphs attached...
-
IMO, they are both quite a lot better than mine.
Note that I am the only one running pfSense, they are just using SH3 as normal.
So my next step then is to "enabling forwarder mode in the resolver, and turning off the registration of DHCP addresses"
Should I go back to modem mode first though?
As I have wifi off on router mode, what do I benefit from being in modem mode, I am only running pfSense in vanilla with DHCP server enabled.
Should I change from using VM DNS servers or is that not relevant?
Also, my pfTop port 53 graph attached in case anyone spots anything odd there
-
pfTop
-
I rest my case, nothing good about those either :) Latency ups and down on Virgin are all part and parcel of their network. Yes their charts might be better than yours, but all of them aren't exactly what you would want to see. A snapshot of mine is at the post here https://forum.kitz.co.uk/index.php/topic,24600.msg414550.html#msg414550.
Nothing on pfTop that is out of the ordinary that I can see, just standard DNS lookups using a tiny tiny amount of data as expected. DNS is about as light weight as you can possibly get for internet traffic that does anything meaningful, just a bit above that of a ping. If you do pfTop again and this time filter by 'proto icmp' you will probably see more pings constantly from Thinkbroadband and other things on your network than you do DNS requests going out.
I'm not sure what exactly the issue is but it is odd that DNS is causing a particular issue as 99% of the time the network isn't making DNS requests, they are just extremely fleeting and they are just ordinary data packets, so something fishing going on with their network. What happens to the graph if you disconnect everything so it is just pfSense connected to the WAN, where there will be no DNS requests being made, are you still seeing the issue?
It would be interesting to see if the same problem happens with a PC on the network doing DNS Resolution and not pfSense and is the same latency issues observed, that way it rules out pfSense, and rules in something Virgin is doing when it detects DNS packets going straight out to authoritative servers, i.e. maybe the modem is inspecting DNS packets going out direct to the internet to record what the requests are, perhaps to keep a record of what sites are being visited to supply the data should they get a court request to. There is more to this than meets the eye I think.
Edit: Okay perhaps it is something related to checks they do, maybe they deliberately throttle DNS packets going to the wider internet on the assumption they are or could be malicious or staging a Denial of Service attack, maybe their throttling when it turns just works by slowing down small packets so ends up affecting pings as well, with popular DNS caching servers like Google etc being whitelisted.
https://www.virginmedia.com/help/open-dns-resolver-vulnerability-alert
Regards
Phil
-
Your chart looks typical really for Virgin Media really, was your neighbours chart also from Virgin Media?
The vast amount of yellow is typical, but the recurring very high yellow peaks are not.
They are caused by pfsense running in resolver mode, nothing more.
Switching a Virgin Hub to router mode immediately removes the high peaks.
Switching pfsense to forwarding mode does the same.
Going by this thread and a thread on the Virgin forums it's something that seems to effect all Virgin lines running pfsense is resolver mode but it only seems to effect Virgin lines.
The difference between my Virgin (FTTP/RFOG) line and my OpenReach (FTTP/GPON) line is laughable. The BQM's are night and day.
20ms base latency and about 10ms constant jitter on Virgin.
13.8ms base latency on OpenReach with zero jitter. A tiny amount of sporadic yellow during high usage.
-
What happens to the graph if you disconnect everything so it is just pfSense connected to the WAN, where there will be no DNS requests being made, are you still seeing the issue?
Good test, will try that overnight and share the BQM.
I will change one thing at a time, but what about these?
1, Should I go back to modem mode first though?
2, Should I change from using VM DNS servers or is that not relevant?
-
Those BQMs all look like the Virgin network serving you is quite heavily loaded.
Not critical but notable.
-
They are caused by pfsense running in resolver mode, nothing more.
But why? What is the difference between sending a DNS request to a DNS forwarder as opposed to sending a DNS request to an authoritative server? How can that cause higher latency and only on Virgin? DNS requests leave pfSense periodically, it's tiny amounts of data, they last mere milliseconds, how can that have an effect on latency on incoming pings that also happen periodically a few times a second that also last milliseconds?
Something else is going on here, and a good test would be to run a DNS resolver elsewhere on the network to see if that triggers the same latency issues to rule out or in pfSense involvement, or perhaps someone has already.
The difference between my Virgin (FTTP/RFOG) line and my OpenReach (FTTP/GPON) line is laughable. The BQM's are night and day.
20ms base latency and about 10ms constant jitter on Virgin.
13.8ms base latency on OpenReach with zero jitter. A tiny amount of sporadic yellow during high usage.
Is this due to being oversubscribed or just typical of DOCIS?
Edit: I just ran DNS Benchmark from GRC.com which makes hundreds and thousands of DNS requests to dozens of DNS servers including pfSense DNS resolver to see which is fastest. With all that going on I had pings running to bbc.co.uk and thinkbroadband.com and they didn't change in anyway. I see nothing on the BQM chart that indicates this test was run. The benchmark showed that the pfSense resolver was the fasted by some margin, hardly a surprise, and worked 100% without error.
So DNS itself is transparent to the BQM, no matter how hard you try to flood the network with requests, so something else is going on here, it's not the DNS requests themselves causing a problem, but something they are triggering.
Regards
Phil
-
Something else is going on here,
Indeed. Not necessarily pfsense at fault, that's not what i was suggesting.
It only happens with pfsense and Virgin lines using the Hub in modem mode.
Is this due to being oversubscribed or just typical of DOCIS?
Just typical of DOCSIS.
I was 1st on our very large Virgin cabinet in a brand new build area.
The BQM was full of yellow (every minute of every day) and the line suffered poor jitter from the moment it went live till i ceased the service a couple weeks ago
-
How's this for VM BQM?
-
Mine and Ronski's back 2 back...
-
How's this for VM BQM?
About as good as it gets tbh.
You'll be hard pushed to find a "cleaner" BQM on Virgin.
-
Mine and Ronski's back 2 back...
Can you disconnect everything from the network leaving just pfSense and the SH connected, and see what the results are? It could just be normal use and a bit of congestion as you have very few red dropped packets, when I had my issues I had loads of red. Have you made the changes suggested in this thread?
My area is all new fibre to the home, my connection was installed in April 2018 as soon as it went live, and they are still installing so utilisation might not be too bad and the network is probably built to handle it better.
-
About as good as it gets tbh.
You'll be hard pushed to find a "cleaner" BQM on Virgin.
That's good then, let's hope it stays that way :fingers:
-
Have you made the changes suggested in this thread?
Ronski, am taking this a step at a time just so I can see where I get to.
Previously I had modem mode enabled and I would lose the internet for approx 1 min at least once an hour. After switching to router mode I did not lose internet anymore but as per my graph I am sure it could be better!
Steps taken so far...
1, Factory reset on pfSense
2, Factory reset on SH3
3, Switch SH3 to modem mode
4, Set on "All IPv6 traffic will be blocked by the firewall unless this box is checked" - I don't use ipv6
5, Added icmp ping rule on WAN
6, Tested ping from external
7, Reconfigured BQM
I will post my next BQM in 24 hours and will put side to side to ones above.
Only diff to before is I am on 2.4.5 p1 instead of 2.4.4 p3
Will be interesting to see if I get drop outs tomorrow when working from home.
In reality, I am aiming for my neighbours graphs - yours is pretty perfect - did you actually use the internet that day!?
-
In reality, I am aiming for my neighbours graphs - yours is pretty perfect - did you actually use the internet that day!?
You caught me out, I didn't, but my daughter and her boyfriend are there so it will be getting some use, but to be honest I've no idea how much, actually just looked through quite a few different dates and they all look pretty similar, even on nights I would have been streaming Netflix.
-
Please see earlier post regarding network load. The average drops to the same level as the minimum off-peak and rises during peak periods indicating a load component.
Download speeds may drop somewhat during peak times. As VM use better schedulers and queuing than they used to even noticeable download contention will only minimally affect latency.
-
Hi
4, Set on "All IPv6 traffic will be blocked by the firewall unless this box is checked" - I don't use ipv6
Virgin Media don't support IPv6 and their network also has some odd issues when customers use a tunnel to get IPv6 https://www.ispreview.co.uk/index.php/2020/08/virgin-media-uk-move-to-fix-20mbps-speed-cap-on-ipv6-tunnels.html so definitely worth making sure nothing that resembles IPv6 packets gets on to the WAN.
Regards
Phil
-
Virgin Media don't support IPv6 and their network also has some odd issues when customers use a tunnel to get IPv6 https://www.ispreview.co.uk/index.php/2020/08/virgin-media-uk-move-to-fix-20mbps-speed-cap-on-ipv6-tunnels.html so definitely worth making sure nothing that resembles IPv6 packets gets on to the WAN.
Phil, Should i be also disabling either of these or is that one advanced setting enough?
-
Hi
IPv6 configuration types should be set to none as there is no IPv6 on VM, but I suspect those settings are ignored anyway if you don't have IPv6 enabled.
Regards
Phil
-
My overnight graph is not looking too bad, however I just had a video call and it was probably worse than before, but I have a couple more today so will see how they go.
One thing that is bothering me, before when I had major drop outs, I saw that I would lose pings to the SH3 cable modem. I am now seeing that again...
-
One thing that is bothering me, before when I had major drop outs, I saw that I would lose pings to the SH3 cable modem. I am now seeing that again...
Found something on the VM forum, this cable modem ping issues relates to the SH3, sometimes the 192.168.100.1 is going through the VM gateway when it should not be. Anyway, seems unrelated to this thread so I will ignore that.
My 24 hours is nearly up and I will post the BQM. Suffice to say my video calls today were fine after the first one, no dropouts.
-
So this is interesting (to me anyway), I would have thought that only changing from router to modem mode, there should be no difference. But clearly the two graphs are quite different.
I have been using the internet all day working from home, 3 longish video calls.
If it was not for the red elements I would call it a day and suggest this might be the best I can get.
Next step is to try the DNS change, I will read back through the thread to find out what I need to do.
-
Now about to make one change...
Services -> DNS Resolver -> Enable Forwarding Mode = Ticked
As a note, my DNS in general setup is configured as 8.8.8.8.
Rest is factory default, so with reference to other settings mentioned in this thread they are untouched...
System -> General Setup ->
Allow DNS server list to be overridden by DHCP/PPP on WAN = Ticked
Services -> DNS Resolver ->
Use SSL/TLS for outgoing DNS Queries to Forwarding Servers = Unticked
Register DHCP leases in the DNS Resolver = Unticked
Register DHCP static mappings in the DNS Resolver = Unticked
Will see what next 24 hours brings.
-
From what I've seen, Netgate generally recommend using Unbound with full resolution so you get a pure DNS solution.
DNS on your end would have no impact on the BQM though as that is pinging directly to your IP.
If your upstream was maxing out, I'd expect some dropped ping packets. Although I wouldn't expect a video call to do that, I don't really trust VM to not be heavily contended upstream.
-
I have had to change my DNS servers to 1.1.1.1 and 1.0.0.1.
Just using 8.8.8.8 I could not access some sites, such as thinkbroadband.com
-
DNS on your end would have no impact on the BQM though as that is pinging directly to your IP.
Alex, from my somewhat limited understanding that is the point of this thread. It does not make sense but there is some problem between pfSense, VM and SH3 in modem mode that for some reason causes weird stuff to happen on BQM.
I am now following Ronskis solution for 24 hours to see if my BQM improves.
Since changing only from router to modem mode+pfSense factory already my BQM has changed (see previous post) which already seems strange and unexplainable, by me anyway.
-
From what I've seen, Netgate generally recommend using Unbound with full resolution so you get a pure DNS solution.
Alex, are you suggesting I need to turn off this setting to get pure a DNS solution?
System -> General Setup -> DNS Server Settings -> Allow DNS server list to be overridden by DHCP/PPP on WAN
-
Alex, are you suggesting I need to turn off this setting to get pure a DNS solution?
System -> General Setup -> DNS Server Settings -> Allow DNS server list to be overridden by DHCP/PPP on WAN
If you're going to define your own DNS servers or use the resolver then yes I'd have that turned off to avoid it adding or overriding with the ISP defined ones.
-
If you're going to define your own DNS servers or use the resolver then yes I'd have that turned off to avoid it adding or overriding with the ISP defined ones.
Cheers. Now turned off. My DNS settings look like this now.
Am assuming I can leave to None as single WAN and am not using TLS for DNS.
-
Cheers. Now turned off. My DNS settings look like this now.
Am assuming I can leave to None as single WAN and am not using TLS for DNS.
Well you CAN use TLS on those servers, its slower but prevents your ISP from snooping on your requests. Or alternatively don't forward at all, do full resolution (what DNS Resolver does by default).
-
So this is interesting (to me anyway), I would have thought that only changing from router to modem mode, there should be no difference. But clearly the two graphs are quite different.
I have been using the internet all day working from home, 3 longish video calls.
If it was not for the red elements I would call it a day and suggest this might be the best I can get.
Next step is to try the DNS change, I will read back through the thread to find out what I need to do.
I never seem to have two days looking the same on my BQMs so you will see some differences, the point being is are they showing a similar trend and I would say yes on yours, which are typical Virgin Media, however you are getting packet loss on the more recent. DNS doesn't cause packet loss, too much time is spent worrying about DNS, it's important, but it simply uses a tiny part of the connection. If DNS isn't working you will find you can't get to web pages at all and will get a DNS type of error reported by the web browser, if it works, it is used just once when you visit a web page or use a service, then plays no further part in the connection.
Using the built in DNS Resolver is by far the best option and using something like DNS Benchmark from GRC.com shows it is 100% reliable on pfSense and faster than any public ones, plus it is more private as it is your own DNS server hidden locally on your own network.
Your packet loss is likely being caused external to pfSense and showing an issue on the Virgin network or their modem, it might be fine another day. Virgin Media has never been known for it's good looking BQM charts, all you can hope for is that things seem to work okay and try and ignore the charts ::)
Regards
Phil
-
Phil, you need to accept that there was/is something going on with the SH3 in modem mode and pfSense DNS combination, I and others on VM forum's proved this at the time. The difference between running the SH3 in router mode and in modem mode with pfSense was night and day and the same goes for using a different router, only when using SH3 in modem mode with pfSense was the problem present, and it affected browsing.
There shouldn't be, but there is/was, until I made the changes detailed in this thread my browsing experience was poor and my BQM was appalling even by VM standards.
My graphs were much worse than exdirectory's, so I've no idea if they are experiencing the same issue or not, but the problem did exist for me and other users.
-
My graphs were much worse than exdirectory's, so I've no idea if they are experiencing the same issue or not, but the problem did exist for me and other users.
Ronski, my experiences back in May time were much worse on modem mode. Two things have changed that may have improved this as yesterday I had no visible outages to me unlike in May which was hourly.
1, Recently I shortened the cable and re-did the coax cable to VM SH3, I was seeing low power on one channel, this seems to have sorted this and I no longer get pre RS errors.
2, I upgraded to 2.4.5
Here is my May graph. So light and day from graph yesterday but now I gained packet loss!
-
Hi
Phil, you need to accept that there was/is something going on with the SH3 in modem mode and pfSense DNS combination, I and others on VM forum's proved this at the time. The difference between running the SH3 in router mode and in modem mode with pfSense was night and day and the same goes for using a different router, only when using SH3 in modem mode with pfSense was the problem present, and it affected browsing.
There shouldn't be, but there is/was, until I made the changes detailed in this thread my browsing experience was poor and my BQM was appalling even by VM standards.
My graphs were much worse than exdirectory's, so I've no idea if they are experiencing the same issue or not, but the problem did exist for me and other users.
I do accept there is a problem but don't agree with this being a fix. Why is the DNS Resolver seemingly causing a problem when all it is doing is going out to authoritative servers (and not all the time once they are cached) with DNS requests? Has someone run a package capture to see what is happening or what the differences are between the two modes? What if you run a DNS Resolver on your network that is separate to pfSense (like that in Windows Server or an open source option), does that still trigger the problem?
Turning off something as fundamental as a DNS Resolver on your network and switching to a forwarder in order to have a working internet connection is not a fix, just indicative of a serious issue somewhere on VMs network or with the modem.
What next? Will the next version of pfSense trigger the same issue with your only option to turn off something else, then something else, then something else until you might as well not be using pfSense at all and the only option is to use the VM supplied kit?
It doesn't really worry me as I'm unlikely to ever be a VM customer, but I like pfSense and can't believe it is the cause of the issue and just like a puzzle and getting a proper answer ::)
Regards
Phil
-
You seemed to imply that DNS won't be causing the problems because the packets are tiny, you didn't say a work around was the wrong way to fix it.
I agree it's a work around, but work it does and my families browsing experience is so much better for it. Unfortunately I am not technically minded enough nor have the time to learn, and when I do I soon forget, so I'm not willing to nail down the cause of it, if the cause was PfSense there would be a slim chance of getting it fixed, if it's the hubs firmware which is more likely there is zero chance of getting it fixed. It seemed to coincide with a hub firmware update.
It's also far beyond my capability and I required the help of many people on here and VM forum's to sort out the work around. I find pfSense rather difficult to get on with, and information hard to find, there's just too much of it, I had a much better experience setting up a Draytek router at work, so if pfSense became too much of an issue whether it's own fault or not I'd switch to a Draytek device, it does pretty much everything I need. At the moment I have to use VM to get a decent speed, that will hopefully change soon as our area is getting FTTP, once live I'll see what options I have.
-
Ronski, my experiences back in May time were much worse on modem mode. Two things have changed that may have improved this as yesterday I had no visible outages to me unlike in May which was hourly.
1, Recently I shortened the cable and re-did the coax cable to VM SH3, I was seeing low power on one channel, this seems to have sorted this and I no longer get pre RS errors.
2, I upgraded to 2.4.5
Here is my May graph. So light and day from graph yesterday but now I gained packet loss!
I don't see any red indicating packet loss, just a busy connection. I occasionally get some red packet loss, but not very often. It may well be worth checking devices on the network, a few months ago my browsing experience went bad, thought the problem had returned, turned out to be an Amazon Fire TV that was having a melt down, no idea what it was doing but it was extremely hot, turned it off and everything went back to normal.
-
Philip you asked what the difference is, its more than you think.
A public resolver like google dns or cloudflare will have lots and lots of traffic, meaning all widely used hostnames will rarely be served to you cold, they will almost always be from their cache.
If you are requesting a uncached hostname the following may happen.
You have to determine the authoritative name server for the domain. Thats the first request.
Then you have to resolve the ip of the nameserver hostname e.g. ns1.superduperhosting.com, sometimes this might need a root dns server lookup for glue record.
Then maybe you finally at this point you can send a dns request to the authoritive server for the ip of the hostname you looking up, typically these are not on cdn's so might be high latency.
All that just for one hostname, many websites have several to resolve.
Now days many main stream servers use very low TTL values so results dont stay cached for long, on a home router, most of the time, you wont get a cached record as it will expire before you click again, this is mitigated either by enforcing high min TTL's or using a feature called 'serve expired' an option you can see inside pfSense.
If you using something like cloudflare dns, the vast majority of the time you just request the hostname and you served from its cache, and they will have a local dns node to serve your requests from so low latency as well.
The best combination for a home user is dns resolver with forwarding enabled, and have serve expired on. Also due to bad behaviour on pfSense turn off the dynamic DHPC registration. Also be aware if you using pfblockerng, the frequency of its updates affect when the DNS resolver is reloaded.
I hope this helps you understand the pros and cons more of direct dns lookups vs using forwarders.
Also in regards to Alex comment most of netgate's recommendations are more suited to enterprise users. The pre tuning for things like super high mbuf's and the direct dns resolution aimed at a different type of usage.
-
Hi
Philip you asked what the difference is, its more than you think.
A public resolver like google dns or cloudflare will have lots and lots of traffic, meaning all widely used hostnames will rarely be served to you cold, they will almost always be from their cache.
I know how it works, but typically for most of us we will visit similar websites during our day. pfSense will also cache all the results, it will also cache lookups to find the various authoritative servers/name servers. Regardless if pfSense needs to do a few lookups more initially for a new address, we are still talking about a handful of packets, and the data transferred is nothing, and the processing is nothing.
As evidence to this as I posted further up, running a DNS benchmark that is making thousands of DNS lookups against the pfSense resolver for random addresses I certainly would never have visited and so would not be cached locally, the benchmark shows the fastest DNS server is my pfSense, with Google etc coming further down the list, even though Google shows a small percentage of cached results, less than you might expect on the random list of addresses the benchmark picks, pfSense is by far the fastest, responds 100% of the time, and was 100% error free.
So whilst that benchmark is running, hammering pfSense DNS Resolver, also hammering a dozen or more other DNS servers to benchmark against all at the same time, pfSense shows nothing to indicate extra loading, a continuous ping to the BBC shows every ping still taking 8ms, so no change or jitter in latency, and the BQM chart shows nothing different during the period of the test. Ergo, DNS is nothing, it places very little demand on a home network, uses next to no CPU cycles, and the data it uses is insignificant.
A DNS Resolver in pfSense is not a resource hog and is not the cause of slowing things down, that is ludicrous to suggest and shows a lack of appreciation to what DNS is technically, that is tiny packets of data and the network is dealing with more traffic processing the BQM chart pings from Thinkbroadband than it does dealing with DNS resolution! pfSense works fine across the planet using its own DNS Resolver, but there is something odd happening when the resolver is enabled on Virgin.
Regards
Phil
Edit, evidence is better than assumptions. I've attached the DNS Result from another test that did as many DNS lookups in 5 minutes than most people will do in a lifetime! pfSense shows CPU around 5% (normal idle amount) so loading unchanged and even the CPU didn't feel the need to throttle up from it's minimum 600MHz (APU2 motherboard). The test result is pfSense is the fastest. Feel free to download and do the same test from GRC.com and then peer review if you like.
-
I don't see any red indicating packet loss, just a busy connection. I occasionally get some red packet loss, but not very often.
Ronski, see yesterday, lots of red.
-
The benchmark is not really representative of real world usage, also the DNS resolver will still cache just the same in forwarder mode, that's not an exclusivity to resolver mode. So really you trying to state, that multiple internet requests are faster going to authoritive servers and root servers vs a single cached result from a DNS cacher.
I have said my 5 pennies worth though, ultimately that's why it can be configured, for everyone to run how they wish to.
-
Ronski, see yesterday, lots of red.
See how it goes, apart from the red the graph looks pretty good. Yesterday looking through my results I saw at least one day with quite bit of red, but otherwise perfectly ok, I'm on my phone & tablet at the moment so not quick and easy to find them.
-
Philip, I am aware of the tool and its outdated approach, I can assure you my knowledge of DNS queries is not based on assumptions, it is just how it works. For a proper test though you would ignore all of the results to the remote servers and simply benchmark your local resolver twice, once with forwarding enabled, and again with it disabled and take note of the result for that, otherwise you will not be doing apple to apple testing, the unbound resolver itself has its own optimisations and settings which can affect performance vs sending queries direct from the OS resolver which is what those remote tests are doing.
A proper test would be something like this. This test was also favourable as I skipped the overhead of finding out the authoritive nameserver, I just assumed it was already known and it was still 3x slower.
root@PFSENSE unbound # dig @ns1.imagis.ro floro.ro. | grep "Query time:"
;; Query time: 31 msec
root@PFSENSE unbound # dig @ns1.imagis.ro floro.ro. | grep "Query time:"
;; Query time: 27 msec
root@PFSENSE unbound # dig @1.1.1.1 floro.ro. | grep "Query time:"
;; Query time: 9 msec
root@PFSENSE unbound # dig @1.1.1.1 floro.ro. | grep "Query time:"
;; Query time: 9 msec
I will leave it at that though, the good thing about pfSense it offers that flexibility, so each person can use as they wish.
My post was really just intended to tell you why it's considered easier on the resolver.
-
The benchmark is not really representative of real world usage
:lol: :lol: :lol: :lol: God talk about egg on face, just why not admit when you are wrong and faced with the evidence. People can read through your politicians reply :P :P :P :P
So what is not "real world" about the test? It takes a random domain name, such as you might find doing a google, and converts it to an IP address. Just because you are shown to be wrong with your statement please don't try and belittle the time someone has spent testing the concept.
Okay you are right in one respect it isn't real world, simply because a normal home network would not make hundreds of DNS requests a second, but the fact the test does and shows pfsense is still faster under such unusual loads, where you said it wasn't faster, just disproves your statement all the more.
The default for pfSense is to use the DNS Resolver for good reason, from their help pages https://docs.netgate.com/pfsense/en/latest/services/dns/resolver.html
DNS Resolver
The DNS Resolver in pfSense® utilizes unbound, which is a validating, recursive, caching DNS resolver that supports DNSSEC and a wide variety of options. The DNS Resolver is enabled by default in current versions of pfSense.
By default, the DNS Resolver queries the root DNS servers directly and does not use DNS servers configured under System > General Setup or those obtained automatically from a dynamic WAN. This behavior may be changed, however, using the DNS Query Forwarding option. By contacting the roots directly by default, it eliminates many issues typically encountered by users with incorrect local DNS configurations, and the DNS results are more trustworthy and verifiable with Domain Name System Security Extensions (DNSSEC).
This is why people are trying to use it and this thread exists, as it is recommended and better practice, and is why people spend the time learning and using pfSense because it provides these extra options over and above bog standard consumer gear. Better to try and fix the problem than work around it by turning it off.
Phil
-
Philip, I am aware of the tool and its outdated approach, I can assure you my knowledge of DNS queries is not based on assumptions, it is just how it works.
Well you can't get more real world than this (pfSense has the tools to benchmark DNS lookups).
bbc.co.uk - result in 1 ms, nearest to me ISP DNS servers coming in at 8-9ms.
Regards
Phil
-
I think 1ms is clearly a locally cached result Philip, and local caching works in both modes, anyway I am not going to enter a childish game of me is better than you, you asked a question I provided an answer.
-
I think 1ms is clearly a locally cached result Philip, and local caching works in both modes, anyway I am not going to enter a childish game of me is better than you, you asked a question I provided an answer.
That's a real world result, so one minute you argue the test isn't real world and discredit it, so when I show a real world result you want tp discredit it as well? Yes a cached result, or are you saying Google or other DNS servers wouldn't have that cached? This real world test is an apples for apples comparison, what you wanted. The fastest those other DNS servers can get a cached result back to me is 9ms, pfSense is less than 1ms and saves outgoing traffic as well, win win. Any small hit taken on the first non-cached lookup of a domain is paid back many times over serving from the cached result later.
The other benefit is pfSense DNS Resolver honours TTL, that often isn't the case given the number of times working on websites and adding DNS entries other people wait ages to get the newest IP, whereas I never have that issue, and if I do, I can clear the pfSense cache, good luck asking Google to do that.
You started of being childish with the I'm going to belittle this person, as you often do. Be humble and sometimes admit perhaps you are wrong and that someone might actually have a valid point. All you've done is continue to obfuscate some valid testing and results then go on to rubbish someone else's bench mark tool, you can't help yourself.
Regards
Phil
-
Philip both google and cloudflare honour TTL values, even as low as a few seconds. The original point made was in reference to the number of lookups been carried out. In addition its been a long time since i seen any UK ISP's DNS not honour DNS records.
Unbound the resolver on pfSense will still cache records locally in forwarder mode, so any reference to caching performance is the same in both configuration's, if you are in forwarder mode, it does not disable the cache so if something is cached at 1ms, it will be cached in both modes, the difference is in uncached performance, unless you are using serve expired or increasing the min ttl overriding the domain's expiry, in a typical home family environment there will be a high miss ratio. In this situation both configurations will send out the queries over the internet, in the forwarder mode the query will be sent to a DNS cacher, such as isp DNS or google DNS, those due to their high levels of traffic will very likely have the record in their cache for popular domains, it is effectively a level 2 cache in this respect.
If I came across as saying this would be faster in every single use case, then I apologise for been misleading, as I never intended to say that, my intention was to say in a low traffic environment the miss ratios will be higher on the local resolver, and as a result more queries get sent upstream. Its the queries that get sent upstream where the behaviour is different. If you have a forwarder configured, you effectively have a level 2 cache, a second shot of hitting a cache so to speak, the hit ratios on large public resolver's will be high due to the sheer amount of traffic they have.
This explanation is not intended to say it will always be faster, which might be why you think I posted it, but it is to explain why I posted that advice.
I have contributed code to the development of Unbound (the DNS resolver that pfSense uses), and to pfSense itself in its DNS resolver implementation.
The issue I have with your posts is that you seem to have decided to go on a personal tirade against me, when all I did was reply to try to answer a question you asked, Alex disagrees with my advice which is fine, but he hasn't acted in the same way as yourself.
Bear in mind this is a topic on a specific VM problem, the guys in here are trying to resolve an issue with latency spikes whilst using pfSense and it got discovered this was in relation to the behaviour of the resolver. This may make no sense as to why the change of configuration has such an impact on that, but it does.
-
So yesterday showed no real improvment over the previous day in terms of switching on DNS Forwarding, not for me anyway.
Could be that 2.4.5 p1 already fixed something, and/or VM fixed something since May, or maybe me messing with the coax cables fixed a problem as I no longer see preRSErrors.
BQM below. So this maybe the best I can get.
I am going to try three more things before I give up though...
1, Fully turning off ipv6 and leaving that for a day,
2, Try adding QOS to the incoming ping to see if that reduces packet loss,
3, Checking logs to see which devices on my LAN might be hammering (incorrectly) the WAN line
I need to learn how to do these first though.
-
The issue I have with your posts is that you seem to have decided to go on a personal tirade against me, when all I did was reply to try to answer a question you asked, Alex disagrees with my advice which is fine, but he hasn't acted in the same way as yourself.
I'm actually using TLS forwarding to Cloudflare personally, I was merely pointing out that Netgate themselves seem to recommend using full resolution as its the only method guaranteed to give unfiltered, unadulterated results.
Resolver = Pure DNS, slowest on cache misses
Forwarding over TLS = Security from ISP snooping, also somewhat slow
Plain Forwarder = Fastest result
-
Back home now after a very nice week of social distancing on the Isle of Wight.
I've had a look and found a TBB ping graph from 1st of April, just to show how bad my connection was.
-
Ronski,
Nice!! Mine was pretty bad during March, I do not have a record though.
I have posted on VM forum, someone said "There should be no "red fringing" on your BQM, that's packet loss." and went on to say I need an engineer.
I went to try and set up traffic shaping for my incoming ping for QOS. But to be honest I got lost in pfSense.
Have you got a record of how you did it?
-
I would suggest they don't totally know what they are talking about, TBB monitor works by pinging our IP addresses, if a router anywhere on the network is overly busy those pings are pretty much the lowest of the low and could well get dropped, resulting in red.
I occasionally get red dropped packets, these two are the worse one's I could find although I have seen worse - it is VM after all, but others have smaller red bits.
https://www.thinkbroadband.com/broadband/monitoring/quality/share/918c4f2e85166f199554fec205e338f4321e09d6-25-09-2020
https://www.thinkbroadband.com/broadband/monitoring/quality/share/7863a2ea038bd11a0de2b077249b23f0d5c157cf-26-08-2020
If you do decide to pursue getting a engineer out good luck on that one, it won't go well, telephone support hasn't a clue what modem mode is and they are obsessed with wi-fi, been there done that and got the grey hairs to prove it :wall: :wall: :wall:
I've got no idea if I setup QOS for pings, I doubt it, but if I did it will be in the build thread (https://forum.kitz.co.uk/index.php/topic,18987.0.html).
-
I would suggest they don't totally know what they are talking about, TBB monitor works by pinging our IP addresses, if a router anywhere on the network is overly busy those pings are pretty much the lowest of the low and could well get dropped, resulting in red.
I occasionally get red dropped packets, these two are the worse one's I could find although I have seen worse - it is VM after all, but others have smaller red bits.
Yeah I'd have to say they're talking rubbish, you'll pretty much ALWAYS get packet loss if your line is maxed out as like you say, ping traffic is the first to get dropped/delayed. Not to mention that we don't know exactly how long BQM actually waits for a response before declaring it lost. Then theres backhaul (not to mention backhoe) problems, temporary work, etc. Its unrealistic to expect to NEVER get packet loss and unless you have QoS prioritising ping packets (not necessarily advised but I do), even your own router likely drops/delays them under load.
I probably don't go a week without seeing some packet loss over my VPN connections on Plusnet or Zen.
-
unless you have QoS prioritising ping packets (not necessarily advised but I do)
Alex, I would like to just try this for a day or two and see what happens and if I lose the packet loss. Any chance I could get some tips on how to traffic shape, i started the Multi- LAN/WAN wizard in pfSense but there are so many options. I assume my setup would be simple but googling ICMP struggled to find a useful set of instructions to follow.
Comments re BQM TTL are interesting. What I could do instead is create my own BQM using a cloud server to ping my connection then with a longer TTL.
-
Oh crikey, its been ages since I set it up and I'm never 100% sure its all working correctly. As I recall you can pretty much stick to the defaults, the important part is figuring out what figure to set for the bandwidth limit to reduce buffer bloat.
I actually just checked and it looks like I DON'T have it prioritised at the moment on the Zen line but do on Plusnet which is interesting as BQM does indicate less packet loss on the Plusnet line.
Once its setup you just add a Firewall rule to classify ICMP echo request from the WAN into one of the high priority queues (display advanced at the bottom of the page, last option is Queue).
-
You can't QoS incoming pings only outgoing responses to them.
-
You're getting ping spikes under load because of bufferbloat.
-
You can't QoS incoming pings only outgoing responses to them.
Good point, although outgoing does at least stop them buffering in the router.
I've never understood how incoming QoS is supposed to do anything at all, seeing as techically it needs to be done on the fatter pipe BEFORE reaching your smaller one (the whole point of line profiles). Yet I've seen it make a huge difference.
-
Do you have a superhub 3/4? Its the chipset in modem mode that's the problem, simply can't handle processing UDP packets. There is no resolution you are just stuck, maybe call up VM and ask for a Superhub 2 which isn't affected. To test put the SH in router mode and double NAT yourself, you will see the high latency/packet loss spikes don't happen then.
Refs:
https://community.virginmedia.com/t5/Speed/UDP-issues-on-SuperHub3-collective-thread/td-p/4382720?lightbox-message-images-4399795=157280i8A7E8A92741C3322
https://www.reddit.com/r/VirginMedia/comments/ihh30v/a_serious_udp_issue_on_sh34/
-
I have pfSense and hub 3 in modem mode, currently no issues, even watched Iplayer last night just after nine. Probably doomed it now........
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/0035ffa204892ee88336fbe8fe58cd662d95d4a4-20-10-2020.png) (https://www.thinkbroadband.com/broadband/monitoring/quality/share/0035ffa204892ee88336fbe8fe58cd662d95d4a4-20-10-2020)
-
Still looking good Ronski, a massive change from your ECI days having all that bandwidth.
-
Do you have a superhub 3/4? Its the chipset in modem mode that's the problem, simply can't handle processing UDP packets.
I'm confused how a modem can fail to handle UDP in bridge mode (where its merely acting as a switch) but handle them in NAT mode? If anything, you'd expect the reverse.
Some pretty shoddy coding I guess.
-
I'm confused how a modem can fail to handle UDP in bridge mode (where its merely acting as a switch) but handle them in NAT mode? If anything, you'd expect the reverse.
Some pretty shoddy coding I guess.
I know right, hard to believe but its true... Also from my reading it appears to be a hardware defect otherwise surely Intel would have corrected it. More unbelievable is the Puma 7 used in the SH4 still has the same issue.
https://www.theregister.com/2017/04/27/intel_puma6_chipset_trivial_to_dos/
https://www.techpowerup.com/232517/intel-atom-based-puma-6-modem-chipset-has-severe-latency-issues-many-cable-modems-affected?cp=2
-
I know right, hard to believe but its true... Also from my reading it appears to be a hardware defect otherwise surely Intel would have corrected it. More unbelievable is the Puma 7 used in the SH4 still has the same issue.
https://www.theregister.com/2017/04/27/intel_puma6_chipset_trivial_to_dos/
https://www.techpowerup.com/232517/intel-atom-based-puma-6-modem-chipset-has-severe-latency-issues-many-cable-modems-affected?cp=2
I knew one of the Puma chipsets had an issue but from what I heard it impacts EVERYTHING, not just bridge mode, and I could have sworn people said it WAS fixed in the newer SoC.
-
The Hub 4 / Puma 7 definitely has the same or similar issues.
By the time Intel finally admitted the issues with the Puma 5 and 6 chipset they also admitted the Puma 7 was affected.
https://www.ispreview.co.uk/index.php/2018/08/intel-coughs-to-puma-cpu-flaw-that-hit-virgin-media-hub-3-router.html
After nearly two years Intel has finally published an advisory and formal CVE entry for a flaw in their Puma 5, 6 and 7 chipsets that resulted in various broadband ISP routers, such as Virgin Media UK’s Hub 3.0 (ARRIS TG2492S/CE), suffering from a mix of latency spikes and a DDoS security vulnerability.
It was never fixed. It's a hardware problem that was fudged with firmware.
-
I see Virgin are offering free HUB 4 upgrades, coincidence? :lol:
https://www.ispreview.co.uk/index.php/2020/10/virgin-media-uk-offering-free-upgrades-to-hub-4-router-again.html
-
This thread has finally helped me to identify the source of packet loss on my VM connection after 3 months of investigation. However, my DNS Resolver was already configured in Forwarding Mode (using DNSSec and SSL/TLS), and I wasn't registering DNS mappings and leases in the DNS Resolver so I don't understand why I was seeing this issue.
I have had to Disable the DNS Forwarder under System/General Setup to eliminate packet loss, but I'd much prefer to have DNS Resolver active for all the reasons pointed out in this thread. Would someone who has this working on VM be able to post all their DNS Resolver configuration details to help me replicate their setup?
Thank you!
-
As I understand it DNSSEC is redundant when you are in forwarder mode, as DNSSEC is for the resolver which in this case is whoever you are forwarding to.
So using plain DNS Forwarder should work just as good, unless you specifically need functions in Unbound (the resolver) that DNSMasq (forwarder) does not support.
-
Thanks for the reply. I'm using Pfblocker so would prefer to use Dns Resolver/Unbound rather than Forwarder/DNSmasq. I might have a choice to make if I can't get the Resolver working though! I'll take a look at DNSsec.
Would appreciate resolver config details from someone who has this working so I can troubleshoot any differences.
-
Are you using pfBlockerNG-devel? Confusingly this is the supported version, they're just waiting on pfSense 2.5 to rename it back from -devel.
Also do you use the DNS blocking? I only use it for firewall rules not the DNS functionality, so no idea if it might cause issues there.
-
Hi, I'm using regular PFBlockerNG (as far as I can tell - I can't see any dev reference in the menus). I am blocking malicious domains.
I've switched DNS Resolver back on with DNSSec disabled as you suggested. I've also allowed DNS records with a TTL=0 under Advanced settings, which I think was mentioned earlier in the thread.
There's no immediate packet loss showing, but it can take a while to develop. I'll keep monitoring.
Would be good to understand the cause of this problem and what Resolver setting is implicated.
Thanks...
-
You have to remove PFBlockerNG in packages and install the PFBlockerNG-devel package. Its the only one that supports all blocklists due to needing a free subscription for MaxMind now.
-
Thanks Alex, I'm still monitoring the WAN link after the earlier changes. No packet loss (which is good) but I am seeing occasional latency spikes. Having re-read the thread, and looked at your comment above, I've disabled PFBlocker for the moment because I realise it forces Unbound to reload and may impact latency. I will now monitor again but the Virgin line behaviour seems to vary a fair bit so it's difficult to be definitive about the effect of the changes.
Do you know if the newer version of PFBlocker avoids a need to reload UnBound?
-
Do you know if the newer version of PFBlocker avoids a need to reload UnBound?
Good question, I'm not sure, I do think its supposed to open up some new options thanks to the Unbound Python support but I haven't looked into it as it works fine for my needs as it is.
Unbound restarting shouldn't really be relevant to the TBB monitor though, that would only be noticeable from actual use of the network. It would be the firewall reloading that would show packet loss from the outside as for a second or so ping packets will be rejected. It shouldn't happen often though, just however often you have it set to refresh the blocklists, which shouldn't be THAT often.
-
This pfBlockerNG-devel (ver:3.0.0_3) option is present (which I use):
Resolver Live Sync
Enable When enabled, updates to the DNS Resolver DNSBL database will be performed Live without reloading the Resolver.
This will allow for more frequent DNSBL Updates (ie: Hourly) without losing DNS Resolution.
This option is not required when DNSBL python blocking mode is enabled.
Note: A Force Reload will run a full Reload of Unbound
-
Ah yes, that's right, but its still the firewall restarting that may cause momentary loss of ping responses but I do not believe it impacts active connections. I've certainly never noticed anything but then my ping chart doesn't show those spikes either.
-
This pfblockerNG option is present (which I use):
Resolver Live Sync
Enable When enabled, updates to the DNS Resolver DNSBL database will be performed Live without reloading the Resolver.
This will allow for more frequent DNSBL Updates (ie: Hourly) without losing DNS Resolution.
This option is not required when DNSBL python blocking mode is enabled.
Note: A Force Reload will run a full Reload of Unbound
Thanks for this, I appreciate the help. I can't find this option in my PfBlockerNG install - is it only available in the Dev version?
More generally I've seen some sixeable latency spikes on the Monitor today with some minor packet loss (better than over Xmas, but worse than in recent days). Difficult to tell whether this is the Virgin Media link failing (again) to cope with lockdown, or some underlying issue with PFSense config. I'll continue to monitor in the next day or so to see if there is a pattern, and may disable DNS Resolver again to try to do a performance comparison. I'll report back.
-
Added: pfBlockerNG-devel (ver:3.0.0_3)
to my post...
-
The current version is 3.0.0_7.
-
Hi, I'm using regular PFBlockerNG (as far as I can tell - I can't see any dev reference in the menus). I am blocking malicious domains.
I've switched DNS Resolver back on with DNSSec disabled as you suggested. I've also allowed DNS records with a TTL=0 under Advanced settings, which I think was mentioned earlier in the thread.
There's no immediate packet loss showing, but it can take a while to develop. I'll keep monitoring.
Would be good to understand the cause of this problem and what Resolver setting is implicated.
Thanks...
Hi just to make sure I understand, what do you mean by allow records with a ttl of 0, I am guessing the serve expired option, which is good if you turned it on, but I just want to be sure thats what you meant.
-
Coming back to this, as I've noticed a few firmware upgrade on the SH3 since originally getting this issue.
I've just turned off DNS Forwarding mode in the DNS resolver, will monitor by BQM graphs to see if anything has improved.
Andy
-
Ok, made the change (turning off DNS forwarding mode in the DNS resolver) yesterday afternoon. Doesn't look to have been too bad since:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/8ef3347c0faf5b887bb325c98c1723b9dd3e52c2-15-01-2021.png)
Anyone else fancy making the same change to see if they've fixed the issue?
Andy
-
Just turned off forwarding mode.
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/1d6fcf2a6171c36e1ae1f255513a45d3adad717a.png)
-
Let's see if it stays like that! Your graph is nicer than mine, but I'm doing a lot of uploading at the moment.
Andy
-
Just noticed I forgot to click apply :no:
-
If it hasnt already been mentioned, I suggest either 1232 or auto for the EDNS buffer size. 1232 is aimed to be the new default for dns resolvers moving forward.
-
Just noticed I forgot to click apply :no:
How's it looking @ronski?
Here's my latest Broadband Monitor:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/6163c52046658e5b018b52822374c22073737189-19-01-2021.png)
Andy
-
Still looks OK, see the graph on the previous page, it's a live graph.
-
Looking good. A few blips like mine, but way better than it was.
My next step is to try turning 'Register DHCP clients in DNS' back on again.
Andy
-
Ok, made the change (turning off DNS forwarding mode in the DNS resolver) yesterday afternoon. Doesn't look to have been too bad since:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/8ef3347c0faf5b887bb325c98c1723b9dd3e52c2-15-01-2021.png)
Anyone else fancy making the same change to see if they've fixed the issue?
Andy
Andy, I tried this after my earlier posts without noticing any improvement. VM quality has been variable (ie appalling) since the New Year though and it's been difficult to tell whether it is my changes or VM that are making a difference. In the last 48 hours I've started to see dropouts on an OpenVPN connection that's been working happily for over a year though so I need to revisit these changes. I've disabled forwarder and will report back...
-
Looking good. A few blips like mine, but way better than it was.
My next step is to try turning 'Register DHCP clients in DNS' back on again.
Andy
Static mapping has always been fine but dhcp leases was never fixed. https://redmine.pfsense.org/issues/5413
-
@adhawkins I've not seen any difference by disabling DNS resolver. What has made a major difference is switching my DNS provider from Cloudflare to Quad 9. I'm using my DNS as the monitor IP for the Virgin link and this shows that peak RTT has more than halved from ~50ms to ~20ms when I look at the PFSense status monitor. I've attached the monitor graph and circled the point where the change was made.
I'm not certain, but think I can see an improvement in the BQM. The first BQM below shows the link before I made the change and the second one is after. For reference, I currently have the Virgin Hub in router mode:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/cf24cfe72105225651a64e9d9d64d632ba53ef95-28-01-2021.png)
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/1c6537dc5044094044122b765402b050bb883e86-29-01-2021.png)
So I'm assuming Virgin has routing problems with Cloudflare (which wasn't always the case), and these don't affect Quad9 (or Google from my quick look). What I'm not sure is why these routing issues would affect the BQM, and how much impact they have on general web use. Maybe some of the more knowledgeable folks on this forum can help with these questions?
Next step is to switch modem mode back on and see how the new configuration copes when broadband is subjected to load on weekdays.
(Edit: would also be good to understand which DNS folks are using when we have discussion about performance so we can compare like with like if Virgin is not routing sites correctly.)
[Moderator edited to fix the links to display the two images.]
-
I've gone back to DNS Forwarding mode. It seemed to me that although the problem was nowhere near as bad as it had been, my connection was slightly worse with the Forwarder turned off.
Andy
-
@adhawkins I've not seen any difference by disabling DNS resolver. What has made a major difference is switching my DNS provider from Cloudflare to Quad 9. I'm using my DNS as the monitor IP for the Virgin link and this shows that peak RTT has more than halved from ~50ms to ~20ms when I look at the PFSense status monitor. I've attached the monitor graph and circled the point where the change was made.
I'm not certain, but think I can see an improvement in the BQM. The first BQM below shows the link before I made the change and the second one is after. For reference, I currently have the Virgin Hub in router mode:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/cf24cfe72105225651a64e9d9d64d632ba53ef95-28-01-2021.png)
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/1c6537dc5044094044122b765402b050bb883e86-29-01-2021.png)
So I'm assuming Virgin has routing problems with Cloudflare (which wasn't always the case), and these don't affect Quad9 (or Google from my quick look). What I'm not sure is why these routing issues would affect the BQM, and how much impact they have on general web use. Maybe some of the more knowledgeable folks on this forum can help with these questions?
Next step is to switch modem mode back on and see how the new configuration copes when broadband is subjected to load on weekdays.
(Edit: would also be good to understand which DNS folks are using when we have discussion about performance so we can compare like with like if Virgin is not routing sites correctly.)
[Moderator edited to fix the links to display the two images.]
To my mind both of those are awful. I can't see much of an improvement between those two graphs.
Just to be clear, are you also using pfsense with the VM Hub in Modem mode?
Andy
-
I bet if you gave the circuit a good 'hair cut' things would improve nicely.
Set a limiter with a bandwidth set to 70% of line speed.
The difference will be immediately apparent on your BQM graph if it corrects things.
Limiter setup:
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/814
-
Since I enabled limiters I had an inconsistent problem with my Plusnet line blocking ping.
I had it working again but upgrading to 2.5.0 RC has broken it again. In fact, the pfSense box can't ping out of the Plusnet WAN at all for some reason, even though from the LAN I can.
-
Since I enabled limiters I had an inconsistent problem with my Plusnet line blocking ping.
I had it working again but upgrading to 2.5.0 RC has broken it again. In fact, the pfSense box can't ping out of the Plusnet WAN at all for some reason, even though from the LAN I can.
To fix that you need to add a floating rule (pass, quick match) as shown here:
(https://poseidon.feralhosting.com/frat3000/ping.png)
-
@adhawkins Yes, using hub in modem mode. I think the status monitor does show a clear improvement in ping time for Quad9. I agree though the BQM is shocking and I'm keen to sort it out if it's in my control, which I have concerns about.
@underzone Interesting suggestion that I've not seen before. I'm not too clear on how to do this from the link you posted although I can see Traffic Shaper offers these options. I'm happy to give this a try but before I subject the family to more internet disruption would you be able to expand on what you think is the issue and why limiting bandwidth will help? A small part of me resents limiting the bandwidth I'm paying VM for unless this is an issue on my side!
-
@underzone Interesting suggestion that I've not seen before. I'm not too clear on how to do this from the link you posted although I can see Traffic Shaper offers these options. I'm happy to give this a try but before I subject the family to more internet disruption would you be able to expand on what you think is the issue and why limiting bandwidth will help? A small part of me resents limiting the bandwidth I'm paying VM for unless this is an issue on my side!
Giving your connection a good hair-cut works for me. Plenty of others use it too.
It is simple really, don't push too much data down a line and overwhelm it.
Give it a good hard cut at first, like I said 70% of line sync, maybe even 60%.
Then increase it in steps so that you still get great results here (I now get A+ for everything):
http://www.dslreports.com/speedtest
PS. Your family wont even notice - no pfsense reboot is required ;)
-
FYI: My settings are shown from "pfsense, Diagnostics, Limiter Info" (I sync at at 19000/78000) as:
Limiters:
00001: 15.000 Mbit/s 0 ms burst 0
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002: 59.000 Mbit/s 0 ms burst 0
q131074 50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
sched 65538 type FIFO flags 0x0 0 buckets 0 active
Schedulers:
00001: 15.000 Mbit/s 0 ms burst 0
q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 50ms quantum 300 limit 2048 flows 1024 ECN
Children flowsets: 1
00002: 59.000 Mbit/s 0 ms burst 0
q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 50ms quantum 300 limit 2048 flows 1024 ECN
Children flowsets: 2
-
To fix that you need to add a floating rule (pass, quick match) as shown here:
(https://poseidon.feralhosting.com/frat3000/ping.png)
Yeah that's the fix but it randomly stopped working and now on 2.5.0 seems to permanently be broken again.
Traceroute is also broken if I tell it to use Plusnet as the source gateway.
Strange thing is I can see the Floating rule matches and the entries in the state table.
-
Just to come back on this one after the helpful suggestions made. Before I could make the changes to limiters suggested by @Underzone there was a big improvement in my BQM - you can see the step change in quality below and this has continued. Speaking to some VM engineers in the area they've had a lot of noise on the line and have been stripping out old cables - look like one of these changes has done the trick and the line is now much more stable. I have random reboots of my Hub 3.0 but that's another story ::) . I have @Underzone's config all ready to go on my PfSense in case I get a recurrence. Thanks for the input
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/07b0393867ec5e146013039f4bd5c70ff5668195-04-03-2021.png)
[Moderator edited to display the external image.]
-
Just to come back on this one after the helpful suggestions made. Before I could make the changes to limiters suggested by @Underzone there was a big improvement in my BQM - you can see the step change in quality below and this has continued. Speaking to some VM engineers in the area they've had a lot of noise on the line and have been stripping out old cables - look like one of these changes has done the trick and the line is now much more stable. I have random reboots of my Hub 3.0 but that's another story ::) . I have @Underzone's config all ready to go on my PfSense in case I get a recurrence. Thanks for the input
<a title="Broadband Ping" href="https://www.thinkbroadband.com/broadband/monitoring/quality/share/07b0393867ec5e146013039f4bd5c70ff5668195-04-03-2021"><img alt="My Broadband Ping - Virgin - PFSense interface" src="https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/07b0393867ec5e146013039f4bd5c70ff5668195-04-03-2021.png" /></a>
Are you sure the BQM is monitoring your WAN IP address? The graph seems to have an exceptionally low latency and "clean" looking for a VM connection.
-
As it's a dynamic IP address the BQM is monitoring the hostname of my PfSense, which has been the case for months. I've confirmed that the IP address has updated correctly in PfSense, with the same IP address as my DNS provider is displaying. So as far as I can tell the BQM is correct but am happy to have any issues pointed out.
I agree it looks strangely clean. The VM engineers told me there had been a lot of complaints about noise on the network which you can see on previous BQMs, and they were replacing older cable and components to resolve the issue. As long as it's stable I'll be happy....
-
I have never ever seen a BQM that clean, period. The latency looks way too low for consumer broadband.
I'd be curious to see a traceroute to pingbox1.thinkbroadband.com to see if its accurate.
-
Hi - traceroute is attached. Let me know what you think. Cheers, Andy
-
The TBB graph doesn't remotely compare to that traceroute which looks more what I'd expect. It should still look like the start of the graph at best if its monitoring the correct IP.
I mean look at mine:
1 vt1.cor1.lond2.ptn.zen.net.uk (51.148.72.23) 12.525 ms 12.203 ms 13.495 ms
2 lag-8.p2.ixn-lon.zen.net.uk (51.148.73.206) 13.293 ms 14.450 ms 12.605 ms
3 lag-2.p2.thn-lon.zen.net.uk (51.148.73.138) 12.521 ms 12.580 ms 12.741 ms
4 lag-2.br1.thn-lon.zen.net.uk (51.148.73.167) 12.489 ms 14.438 ms 13.373 ms
5 netconnex-gw.zen.net.uk (82.71.254.2) 12.227 ms 13.891 ms 12.457 ms
6 po11-13.bdr-rt3.thdo.ncuk.net (80.249.97.22) 12.751 ms 12.237 ms 11.858 ms
7 po4-31.core-rs4.thdo.ncuk.net (80.249.97.85) 13.149 ms 13.249 ms 13.760 ms
8 pingbox1.thinkbroadband.com (80.249.99.164) 12.402 ms 12.280 ms 11.836 ms
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/thumb/7ffe78679c754eb5e9b18a79537c9e8abd48ef15.png)
NOTE: Using my IPv6 graph as something has gone wrong on my router and the IPv4 graph isnt working.
-
I'm with the other 2 posters.
The 1st half of that BQM is a typical Virgin DOCSIS connection.
The 2nd half is not showing the replies from your home connection.
It's physically and technically impossible for your base latency to drop that much, unless the speed of light changed in the last couple days.
Even Virgin connections next door to the TBB BQM pingbox won't have latency that low.
-
Fair enough. I didn't mean to imply I was enjoying latency levels that broke the laws of physics, just that changes made by Virgin have improved the original cause of my problem which was latency spikes. I've attached the PfSense monitor results (which use pings to Quad9 DNS) for the line before the change on the BQM was made (circled), and afterwards. PfSense has averaged some of the data points which masks some of the effect, but the only latency spikes to the right of that change are the Hub reloading and I think the line quality is improved. So in my case I currently think that noise is the root cause of the issue (though this doesn't explain why a switch from CloudFlare to Quad9 improved performance so noise may not be the complete story). Based on helpful advice here I've got limiters set up and ready to go and have tweaked resolver settings. But for the moment I'm happier than I was. I'll probably create another BQM to see if that resolves concerns.
-
Hi all,
Re-awakening this topic, as I'm receiving a new Hub 4 today. Has anyone else had any success with pfsense connected to a Hub 4? Can I revert back to using the DNS server in non-forwarding mode?
Thanks for any input.
Andy
-
Hi Andy, no direct experience. But in my research I've not seen any reports that the SH4 resolves the underlying UDP issues that impact use of DNS Resolver in non-forwarding mode, so I'll be very interested to see what response you get. Out of interest, how did you manage to get hold of a SH4?
-
Hi,
Upgrading to Gigabit today. I think that's the only official way to get one at the moment.
Andy