Kitz Forum
Broadband Related => ISPs => Topic started by: Chrysalis on November 05, 2022, 05:32:42 PM
-
On AAISP network over a cable connection.
Tracing route to bbc.co.uk [151.101.0.81]
over a maximum of 30 hops:
1 17 ms 17 ms 19 ms l2tp.thn.aa.net.uk [90.155.53.19]
2 17 ms 19 ms 19 ms k-aimless.thn.aa.net.uk [90.155.53.101]
3 17 ms 18 ms 18 ms 195.66.225.91
4 19 ms 18 ms 18 ms 151.101.0.81
Trace complete.
Using windows L2TP client at the moment as pfsense is freaking out, I activate the tunnel and VM goes down alongside AAISP not coming up, but on windows it worked as expected.
-
That's odd, works fine here. The only problem I've had is if it goes down (as I'm using it over 5G) it sometimes fails to come back up.
-
Well this is what I did please let me know if this is wrong.
https://support.aa.net.uk/L2TP_Client:_pfSense
Followed the above guide, so new PPP, L2TP type, selected igb3 which for me is my VM interface, entered my auth details, and the aaisp L2TP IP.
I then went in the interface after to configure IPv6 as SLAAC as the above guide says and enabled it.
VM then went to 100% packet loss, errors to VM gateway, AAISP L2TP gateway stayed on pending, logs showed no responses to L2TP packets. Could only get back online by disabling the L2TP service, and cycling the VM DHCP.
However on windows it just came up as expected, its capped to 100mbit, so I am guessing the L2TP on broadband accounts doesnt have the new 200mbit cap.
My theory is that I am linking it to a physical interface, and maybe when it works its on a virtual PPPoE interface, but I am using IPoE.
-
Okay, here is what I have.
Worth noting, I had to set System, Routing, AAISP_SLAAC (the IPv6 one) to Disable Gateway Monitoring as the AAISP gateway did not respond to pings and while the documentation says "it should work anyway" I found it did not. This does not however impact IPv4.
-
Looks almost the same, you ticked the ipv4 parent box as the only difference, after I ordered my food shopping I will give it another go.
-
Its a no go, I tested it over EE (ue0) and it works fine like it does on your mobile network, but as soon as I try the VM interface it kills VM.
The error that appears for the VM connectivity is "arpresolve: can't allocate llinfo" and igb3 flips between active and no carrier state until I intervene by killing the L2TP.
-
Is working now Alex, seems spoofing mac on the host interface breaks l2tp on pfsense.
Also IPv6 works however the monitoring if you dont specify a monitor ip reports 100% loss (aaisp gateway doesnt respond to pings) and if selecting a manual ip to ping although it pings manually the SLAAC gateway gets stuck in a pending state which I am guessing is a pfsense bug, so I force it to the online state so it gets used and then everything works.
-
Is working now Alex, seems spoofing mac on the host interface breaks l2tp on pfsense.
Also IPv6 works however the monitoring if you dont specify a monitor ip reports 100% loss (aaisp gateway doesnt respond to pings) and if selecting a manual ip to ping although it pings manually the SLAAC gateway gets stuck in a pending state which I am guessing is a pfsense bug, so I force it to the online state so it gets used and then everything works.
The gateway monitoring issue isn't limited to AAISP, it happens on Zen too for me, stuck in pending right now even though I have confirmed manually the IP I have in monitoring is responding fine.
This is one of the reasons I just don't care about IPv6, its just not as reliable as IPv4, at least on pfSense using L2TP/PPP as the carrier layer. When I had the HE.NET GRE tunnel configured, that always seemed to work.
More interestingly I seem to have lost IPv6 routing over AAISP completely at the moment. If I try to force AAISP as the src IP, it just goes over Zen anyway. :-\
-
Out of interest, what are the use cases whereby you find it useful to have something like this L2TP service setup?
-
The gateway monitoring issue isn't limited to AAISP, it happens on Zen too for me, stuck in pending right now even though I have confirmed manually the IP I have in monitoring is responding fine.
This is one of the reasons I just don't care about IPv6, its just not as reliable as IPv4, at least on pfSense using L2TP/PPP as the carrier layer. When I had the HE.NET GRE tunnel configured, that always seemed to work.
More interestingly I seem to have lost IPv6 routing over AAISP completely at the moment. If I try to force AAISP as the src IP, it just goes over Zen anyway. :-\
Yep its a pfsense bug.
Go to system menu -> routing, select the gateway, click edit, and tick the option "disable gateway monitoring (assume its online)".
It will go to online state and work normally.
-
Out of interest, what are the use cases whereby you find it useful to have something like this L2TP service setup?
For me I have certain uses, and also use cases where I need static IP addressing, and also have use cases for IPv6 as well which VM has no IPv6 support at all.
-
For me I have certain uses, and also use cases where I need static IP addressing, and also have use cases for IPv6 as well which VM has no IPv6 support at all.
Interesting. I am intrigued by it. Especially as I see that for £10/per month domestic it comes with 2TB traffic. I have 10TB at the moment on FTTP don't fully need, but 1TB isn't enough, so I pay the extra £10. If the 2TB /month that comes with the L2TP service goes into the pot, I could have 3TB / month Inc the L2TP service, which would be more than enough data and offer the extra functionality.
Edit: actually I don't think broadband and L2TP can share quotas. Oh well.
-
Its built in if you have a broadband account and I assume shares the quota. Thats how I am using it now. But of course can buy it as standalone also.
https://support.aa.net.uk/Category:Incoming_L2TP
-
Its built in if you have a broadband account and I assume shares the quota. Thats how I am using it now. But of course can buy it as standalone also.
https://support.aa.net.uk/Category:Incoming_L2TP
Ah, gotcha! :) So is there a good configuration of a particular router that will only bring up the L2TP over eg: cellular if the main connection goes down?
-
Firebrick can do just that. Using ‘profiles’.
-
Yeah thats a big selling point of the firebricks, the automatic failover.
Pfsense can automatically switch connectivity but that is in the sense when all the connectivity is always online and it just switches its routing policy, to actually disable one connection (rather than just change the active gateway) and then auto enable another that I think is more specialised as Weaver said.
-
Yeah the big drawback with pfSense is if your backup connection goes wobbly, all connections go wobbly, its extremely frustrating.
It seems almost pointless having a backup if it actually causes downtime where not having one would have been more stable. You just end up having to force the gateway as always online and then if it HAS stopped working, should your main connection go down you have no backup.
-
I suppose the failover you want ideally is quite complex, too. Ideally I'd probably prefer for it to be AAISP (main line) -> AAISP (L2TP over alternative transit) -> Alternative transit (in case the reason for failure is actually an issue at AAISP, and not a transit issue). In the last case you might not have all services up (if there are things reliant on the static IP setup over L2TP). But you'd at least have basic connectivity.
-
Alex probably will be killing IPv6 on my consoles.
Downloaded two games today (DSL hooked up to pfsense again but ipv4 routed via VM).
Noticed download was slow, and was going over the DSL IPv6, and isnt really a way router side to allow IPv6 but force downloads over IPv4, at least not trivially so for now just added a reject rule on IPv6 traffic for consoles as a quick fix, but will probably kill the DHCP6 allocation for them. If I cant get Teredo working, then what I might do is keep IPv6 on, keep the reject rule but allow traffic specifically to the multi player gaming ports.
-
I’m definitely going to be looking at L2TP over 4G once my health improves. I’ve booked AA to help me with the Firebrick config but have told them it won’t be soon, as I’m going through a real health slump just now, having caught the flu from Janet.
What’s the AA L2TP payload MTU ? Can AA take IP_PDU_size=1500 bytes so that the L2TP_PDU_size (= 1500 + L2TP_header_size ) > 1500 ? Mind you, even if AA can, I doubt the 4G carriers can handle more than 1500 bytes - I’m not thinking straight.
-
Offtopic but very sorry to hear that you are both unwell there. Speedy and full recovery. :fingers:
-
Much appreciated. Have been feeling like crap, totally exhausted too.
-
Sad to hear Weaver, I hope you get better quickly.
I dont know how high the MTU can go but if I check my MTU on speedguide analyzer tool with MTU left as unset in pfsense it reports a MTU of 1460 and MSS of 1420. This is on a host connection that has 1500 MTU.
1432 bytes is the highest unfragmented ping I can do.
-
L2TP been something I have never used before I am discovering some things.
So pfsense defaults to 1492 MTU for the L2TP interface, which is too high, so I changed that to 1460, this next thing has me bamboozled though.
The host link which in this case is my VM connection gets a fairly consistent 3ms overhead simply for having L2TP enabled, even if idle. This doesnt happen if I run L2TP inside windows. I dont know if this is normal and expected for L2TP or it indicates a problem. I havent looked yet to see if the same happens using it on top of EE.
-
Is this something to do with non-neutrality in VM then? L4 and/or L7 protocol-aware middleboxes of some sort?
-
I think its probably a pfsense/freebsd problem, tomorrow I will see if the same occurs on top of EE.
-
Its cable modem related, there is a few reports of people finding oddities with dpinger and cable modems, and indeed the issue is not apparent with normal ping from both client machines and even pfsense itself, dpinger is doing something "different". There is no actual performance issues either, download speeds, streaming etc, is fine over the tunnel.
-
I've always found gateway monitoring rather "different" and often not reflecting real-world events. Not least on some occasions it fails completely but pinging the same IP manually works.
-
Ok I have a little update.
I made some adjustments on the L2TP client side, MTU and MRU set to 1460, AAISP left at auto as oddly they dont have a 1460 option for MTU.
I decreased the ping interval on dpinger, and there is now no measurement of increased latency on tbb, pings from any network device, and pfsense itself are also normal, its now only dpinger been odd, and is threads on reddit about dpinger weirdness with bridge mode modems (higher latency vs normal pings), so this seems fine now and just a dpinger oddity.
The SLAAC issue I think is due to a pfsense bug, I have the same issue on some servers in datacentres, the problem seems to be triggered when the gateway is not pingable, when it isnt pingable (which seems common on ipv6) it will stay in pending state causing the routing to never get activated, choosing another IP to ping doesnt resolve it as I think pfsense expects the gateway to always be pingable, the workaround is to disable the monitoring at the loss of having no monitoring data or auto behaviour that relies on the monitoring. I am going to post on redmine bug tracker about it.
There is an existing bug report (fixed in 2.7.0) which was really interesting I read last week which had another similar issue where the debugger posted the process of scripts that are run to explain how he fixed it, which was really nice info but I have ended up losing the link. So will need to check my browser history on that one.
-
Some more info related to dpinger, I checked the historical readings from it on DSL and it was having fluctuations with an average of 1-2ms jitter measurements when on DSL, this doesnt correlate with any historical remote monitoring or live usage observations. So I now consider dpinger as a measurement of jitter/latency to be of low accuracy, this is more of a pfsense/opnsense observational post rather than related to L2TP.
-
@Alex this is the the bug report which I mentioned earlier.
I have linked to the reply as a reference point by someone who did a lot of digging into the monitoring of ipv6 gateways. I am not sure if this is the cause of the perpetual 'pending' state but its a possibly. My theory remains its related to gateways that are not pingable.
https://redmine.pfsense.org/issues/11454#note-23
--
After more digging i came across more bug reports, of which a fix was made by using the "dont wait for RA" option, no such option exists for SLAAC, but I have now moved the configuration to use DHCP6C instead since that part of pfsense has clearly had more dev attention and gateway monitoring now works on IPv6. This option does assign a internet IPv6 to the l2tp interface instead of just link local. The gateway is still not pingable (monitoring is pinging 2001:4860:4860::8844), so the issue might be that pfsense will refuse to consider it online if it only has a link local address.
I also have discovered a few things which I consider to be bugs such as vpn interfaces been brought online using link local address for the physical lan interface instead of the physical wan interface, the aaisp v6 interface is actually having both link locals added to it, but luckily the correct one is added to the routing table.
-
Has anyone got this working on an Edgerouter?
I'm looking at giving it a shot over my VM connection, but looking at the AA wiki it seems there's some strange issues with the Edgerouters.
I've reached out to AA direct to see if they can advise, but figured i'd ask you lovely folk here also :)