Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 2 [3] 4 5 ... 10

Author Topic: 10 Gb, Dual-WAN, segmented home / home office Setup  (Read 7764 times)

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 9103
  • Retd sw dev; A&A; 4 × 7km ADSL2; IPv6; Firebrick
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #30 on: February 03, 2020, 07:29:18 PM »

Unlike the solution suggested by some of the setups AA describe for failover, I was a perfectionist and insisted on keeping all IP addresses constant and keeping TCP connections intact during the failover. So that means that addresses inside the Firebrick’s LAN range don’t suddenly change from what they were before to some RFC1918 crappy new replacement addresses; existing IP addresses carry on exactly as before. My addresses are all globally routable and eternal, unchanged during failover. I use a smaller IPv6 MTU of 1408 too so that TCP connections can continue without being disrupted because the MTU is suddenly forced to drop substantially. It would have to drop because of increased overhead because currently there’s no 3G/4G support for IPv6 so a tunnel is used. I have the MTU permanently low, so it does not reduce - it’s lower than necessary before failover so that when the failover happens nothing actually changes; the increased overhead just causes an MTU to be chosen that matches that already in use, so nothing breaks. IPv4 works differently, uses MTU 1500+8=1508 normally and after failover goes down to MTU 1408. These quite low MTU values are chosen because they are perfection for ATM cell packing efficiency with the DSL overhead that I have in this particular case.
Logged

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #31 on: February 03, 2020, 10:11:43 PM »

I could certainly arrange seamless failover but genuinely don't have any real driver for it. Having connections reestablish is fine.

So from that point of view the solution is quite inferior to yours  :)

If I really wanted seamless I'd be using a pair of tunnels to an SD-WAN appliance hosted on a VPS and be pushing all my traffic that needed seamless through there - that way no IP address changes as encapsulation takes care of the change.

Could have it switch over within at most 100 ms via our fast fail feature but that's probably excessive.

If I had 2 wired LAN circuits I could run either all traffic or a critical subset through them and have a failover time of zero - for that critical, sensitive traffic one link carries the data the other FEC on a 1:1 basis so loss of one link means not a single bit dropped.

But that's excessive, expensive and Openreach won't let me have 2 access lines so academic.
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

aesmith

  • Kitizen
  • ****
  • Posts: 1010
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #32 on: February 06, 2020, 01:39:23 PM »

Unlike the solution suggested by some of the setups AA describe for failover, I was a perfectionist and insisted on keeping all IP addresses constant and keeping TCP connections intact during the failover.
I believe you said at some point that you use actual Internet addressing (AA PA) on you internal network.  Is that correct?  If so then only AA are going to route that address space.  However could you you use an AA L2TP connection for your backup, giving you a completely free hand for 4G provider?  I'd be very surprised if their service wouldn't pass over NAT.  If you really wanted to avoid NAT you could use one of the 4G routers that can bridge the WAN connection to the LAN.  Presumably the Firebrick would be OK receiving a DHCP address, I find it hard to believe it can only work using PPPoE.
Logged

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 9103
  • Retd sw dev; A&A; 4 × 7km ADSL2; IPv6; Firebrick
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #33 on: February 10, 2020, 04:10:51 PM »

> Do you use actual internet addressing?

Yes. I use global routable constant IPv4 and IPv6 addresses for all the hosts on my main LAN, no NAT, no RFC1918 addressing required.(Although I do use RFC 1918 addresses internally; for talking to modems. The modems’ admin interface addresses are exposed as 192.168.n.1 when n is 1,2,3,4 the nth modem. The Firebrick routes these addresses across from the link between the Firebrick and each modem’s admin interface and passes traffic onto the main LAN so that the modems can be queried and so forth)

> L2TP:

Yes, I thought about that. But then there’s the cost of AA L2TP, charged per byte of traffic. And the hassle of setting it up given my pain and concentration levels. That would give me 4G instead of 3G though. And it’s only used once in a blue moon anyway.

The current system keeps me going and is truly seamless because the src IP addresses don’t change so no TCP connections that are established need to be broken; so no protocol breakages at all.

IPv6 MTU does not change at failover, but for the normal DSL state, PPP MTU=1500+8=1508 therefore normal IPv4 PDU MTU=1500 bytes and that will drop, a lot, when in the failed-over state, because of the limited MTU of the 3G ‘dongle’ USB NIC (IP PDU MTU is only 1430 or 1440 or something like that). Because IPv4 can not only change MTU on the fly even while a flow is established so that I’m assuming that PMTUD will detect the sudden reduction in path MTU and fix the problem, but IPv4 can fragment packets at intermediate routers not just at the source. So for this reason, I’m assuming that even before PMTUD kicks in and changes the PMTU, the first IPv4 packets can simply get fragmented if need be and again I’m assuming, perhaps rather optimistically, that that isn’t going to be the end of the world if there’s a bit of IPv4 fragmentation on failover, perhaps only short-term, but I’m not willing to reduce the IPv4 MTU permanently even in the normal case and just for the sake of the smoothness if the failover when fragmentation should I hope work and may only be needed for a while. Does a sudden onset of fragmentation get detected and trigger a change in PMTU that some transport layer by Eg TCP can take advantage of to fix the temporarily or permanently limited MTU problem?

It’s really annoying because the reduced MTU of the USB 3G NIC is way below 1500 bytes, for no good reason at all. If anything it should be oversized something like 1600 bytes, to support tunnel overhead. What is incredibly annoying is that AA’s 4G/3G carrier network AQL/Three does not support IPv6 and this from AA, the leader in IPv6 provision, with IPv6 first offered in something like 2002 iirc.  Have to use AA’s 6in4 tunnelling to get it to work

Logged

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #34 on: February 10, 2020, 05:39:08 PM »

PMTUD relies on the do not fragment bit being set so you'll either have a stall in the connection while the new MTU is worked out or fragmentation.

Fragmentation in the path will break certain traffic. Some applications don't respond well to it.

Only way around this is tunneling and fragmentation, coalescing and reassembly before presentation to the end application.

Firewalls in path may not be fans of fragments either, especially if they arrive out of order.

If it were that easy otherwise I'd be out of a job!
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #35 on: February 10, 2020, 06:09:21 PM »

Regarding the original I may have a possible solution to the dual-WAN dilemma, however I can't go into any detail just yet.

Things are getting interesting. :angel:
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 9103
  • Retd sw dev; A&A; 4 × 7km ADSL2; IPv6; Firebrick
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #36 on: February 10, 2020, 11:55:11 PM »

Perhaps I’ve done a bad thing allowing occasional IPv4 fragmentation then, but the failover state or even just the immediate post-failover period are very rare occurrences so I thought I had better not let such things compromise the normal state of affairs.

It would indeed I suspect be useful if some routers could coalesce fragments in case of a problem with fragment-unfriendly firewalls or apps further upstream.
« Last Edit: February 29, 2020, 08:40:24 PM by Weaver »
Logged

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #37 on: February 11, 2020, 08:43:30 AM »

You've done the best you could without specialised, and expensive, hardware.

The cigar-box size piece of kit on my desk that'll do this for 100 Mb/s of traffic across pretty much as many WAN links as you want to throw at it probably costs about the same as your Firebrick. That you need another one in a data centre somewhere to terminate the traffic or a virtual machine adds to the cost. The licensing is where it gets really fun.
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #38 on: February 29, 2020, 10:45:19 AM »

So I've had a bunch of issues and had to have a rethink.

Ubiquiti kindly started selling a fantastic bit of equipment in the interim meaning I can outsource much of the work.

I have one of these on their way from them. It'll serve as router, switch under stairs, gateway and wireless controller.

I will supplement it with their access points as I go, adding one or two a month depending on budget until every room has superb wireless coverage.

I am hoping to get cabling to a couple of key places in the property, however this may end up being unjacketed ClearCurve along the skirting board  :)
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 30742
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #39 on: February 29, 2020, 02:52:23 PM »

When the moment is ripe, I (and I suspect other members) would be interested to see another (pictorial) diagram of how your LAN is now planned to be deployed.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #40 on: February 29, 2020, 04:16:51 PM »

This depends on how/if I can get cabling to some rooms. That's something that's far from clear just now!
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

Ronski

  • Helpful
  • Kitizen
  • *
  • Posts: 3773
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #41 on: February 29, 2020, 05:13:26 PM »

Ubiquiti kindly started selling a fantastic bit of equipment in the interim meaning I can outsource much of the work.

I have one of these on their way from them. It'll serve as router, switch under stairs, gateway and wireless controller.

I read this, and completely by coincidence read this, don't know what his problem is with it though.
Logged
Formerly restrained by ECI and ali,  now surfing along at 390/36  ;D

CarlT

  • Kitizen
  • ****
  • Posts: 1697
  • Software Defined WAN deployment engineer
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #42 on: February 29, 2020, 05:27:51 PM »

Inevitably going to be software. Not too worried: that can be fixed.
Logged
WiFi: Nighthawk® AX12 RAX120
Routing: pfSense VM
Switching: Mikrotik 2* CRS305-1G-4S-IN, 1 * CRS309-1G-8S+; various cheap and cheerful TP-Link/Netgear
Exchange: Wakefield
ISP: BT Full Fibre 900. Zen Full Fibre 900.

Weaver

  • Addicted Kitizen
  • *****
  • Posts: 9103
  • Retd sw dev; A&A; 4 × 7km ADSL2; IPv6; Firebrick
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #43 on: February 29, 2020, 08:51:49 PM »

I was very put off Ubiquiti WAPs because of RevK - the boss of AA- ’s experience with Apple and roaming and Ubiquiti. (Described in Revk’s blog. Basically he couldn’t get roaming to happen with an iPhone despite huge effort. This might not happen with you though because you’re using all Ubiquiti gear and it could be the mixture of Ubiquiti and Firebrick that somehow triggered his badness.

It would be a death situation for me, an all Apple shop with a Firebrick.
Logged

Alex Atkin UK

  • Kitizen
  • ****
  • Posts: 1639
    • My Broadband History
Re: 10 Gb, Dual-WAN, segmented home / home office Setup
« Reply #44 on: February 29, 2020, 11:10:14 PM »

I was reading that Ubiquiti have pretty poor manufacturing quality and firmware compared to such as Zyxel (their comparable products, not consumer).

Reading their forums you can see the firmware issues, they seem to release products in beta or even alpha state and use their customers to bug fix.

I get the appeal of using a central controller to manage everything, but I'm having second thoughts on their products after really thinking about this.
« Last Edit: March 02, 2020, 08:10:57 PM by Alex Atkin UK »
Logged
INTAKE (ECI) Zen: Home Hub 5A OpenWrt Plusnet: VMG-3925-B10B Three 4G: Hauwei B535-232 Router: pfSense (i5-7200U) WiFi: Ubiquiti nanoHD
Thinkbroadbamd Quality Monitors
Pages: 1 2 [3] 4 5 ... 10