Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[niemand]

Anyone done this? Any gotchas?

Should work just fine I would hope.

The software I was planning to use has a functionality missing right now that I would very much desire from an edge device.

I'm not going to mention specifics as it's on the roadmap but I'm nowhere near important enough to get it bumped higher up the queue.

I will use hardware appliances physically out of path advertising certain subnets to the pfsense or similar VM via iBGP so that they can do their SD-WAN magic.

[Alex Atkin UK]

There was a time this seemed to be the way it was used in businesses, if how often I saw it mentioned on the Netgate forum is anything to go by.

Personally I hate the idea, as I swear blind I can "feel" the difference even between the different CPU power management settings in responsiveness, let alone the extra layers virtualisation would add.

All I'm finding about gotchas is https://docs.netgate.com/pfsense/en/latest/virtualization/virtio-driver-support.html

[Chrysalis]

I have done it on 2 VM's and works just as well as on bare metal, the virtualized network driver is very stable and performs well (in my view better than many bare metal drivers).  I hit a peak of over 900mbit throughput (WAN) on a gigabit port.

To clarify it is forwarding performance I am quoting.

[Alex Atkin UK]

I'm peaking at 941Mbit in, 908Mbit out (plus 13Mbit going out the WAN) on bare-metal from my IoT LAN to the main LAN.  This shows as 983Mbit on the router dashboard graph for some reason.

There may be other factors as QoS is enabled on the LAN.  I'm certainly wondering how hard my box is going to need to work with PPPoE too.

[dee.jay]

Done it that way for several years now. Works great.

[niemand]

Thanks guys. I wasn't anticipating any issues.

It'll be running on the attached host.

Unless the code is truly awful it should be able to handle the required ~2.2 Gb/s throughput that's the maximum it might have to deal with.

As this is a technically minded forum I'll just outline what I have in mind - bearing in mind this is for the new home that I never plan on moving from unless horizontal!

In the Harry Potter room (cupboard under the stairs) will be a https://mikrotik.com/product/crs309_1g_8s_in taking in 2 WAN feeds on GbE ports and sending them to the host via a 10 GbE port. This 10 GbE port will have VLANs for each of the WAN links and a native VLAN for LAN-side traffic.

In my home office will be another CRS309_1G_8S. This will use 2 * 10 GbE ports for connecting back to the Harry Potter CRS309 and forwarding the 3 VLANs to the VM. The rest of the ports will be populated as needed with SFP+ on the native VLAN, initially another 10 GbE for wireless AP which also has 6 GbE ports for use on the home network.

EDIT: I could use the 5 port versions of both however the cost difference between them made me inclined to spend a little more rather than having to buy more 10 G-capable switches later and cascade.

The 10 GbE NIC on the host will be segmented into the 2 * WAN VLANs feeding vNICs on the pfsense host and the native LAN VLAN. The native LAN VLAN will be rate limited to 7.5 Gb/s outbound on the switch to avoid it impinging on the WAN.

The pfsense host will also have a 'lab' interface - this will be to a virtual switch connecting to the VMs terminating my lab network internally on the host only. Those will have WAN vNICs on the vswitch the pfsense lab interface is on, segmenting the lab network from the rest of the home network.

A simple TP-Link smart switch and the existing GbE ports on the host - 2 built in will be fine as I'll be replacing the 2 * GbE NIC on PCI-E with a 10 Gbase-T - will serve as lab port density.

This lab is where the SD-WAN will also happen. The only traffic heading to the Internet from here will be control plane traffic for the SD-WAN and its own proprietary tunnels. There will be some static load balancing rules to pin traffic from the respective SD-WAN appliances to each WAN link.

Nothing too complex: just takes a little time, care and attention to make sure I don't mess up the virtual network configuration. 

[niemand]

Is it just me being a dumbass or does pfsense not support a dedicated interface for management purposes? It seems to be manage it through the forwarding plane or console everything?

This is an incredibly dumb idea if this is the case. Forwarding plane and management plane shouldn't meet or have dependencies on one another if at all possible.

[Alex Atkin UK]

By default, no.  The recommendation seems to be if you want that then use the default LAN as the management and a secondary LAN as your normal one.

You can then easily disable forwarding on the management LAN and maintain the default anti-lockout rules that are intended to prevent you accidentally firewalling yourself from the Web UI. https://forum.netgate.com/topic/105934/dedicated-management-port-for-pfsense/4

[Chrysalis]

By default it isn't setup that way, but you could configure that way with some effort.

There is some design issues within pfsense, probably related to a combination of no motivation from developers to implement different and political issues.

[Alex Atkin UK]

Its more down to a one size fits all.  Plus is there any functional difference between setting up a second LAN with forwarding turned off, blocking traffic to the management ports on the LAN and a dedicated management port anyway?

[niemand]

It's nothing insurmountable I was just checking that I wasn't missing something obvious.

Worst case management can be done out of band as it's a VM.

In happier news a throughput test that was both biased against the pfsense VM, as it was between two other hosts on that same VM, and a single flow so harder to multithread processing of, provided more than enough capacity for my requirements.

No doubt with some modification of CPU affinity and the offload from the Intel NIC it's getting this'll be substantially improved but it's fine.

[Alex Atkin UK]

There's just something deep in my bones that makes me refuse the whole concept of a VM, because it might induce 0.001ms more latency in the connection. 

The same way I plugged the HG612 back in because it consistently gives me 1Mbit more sync rate than the Zyxel.

Plus how it really bothered me that when I iperf3 benchmarked my pfSense box, it can hit a higher NAT speed in one direction than the other.
Same with my desktop, hits 2.22Gbit down from the NAS, 2.35Gbit up, even though I will never saturate that with real-world throughput it bugs the hell outta me its not symmetrical.

[niemand]

There's just something deep in my bones that makes me refuse the whole concept of a VM, because it might induce 0.001ms more latency in the connection. 

There's something deep in my wallet that makes me love the whole concept of a VM.... because it pays my bills. Ka-ching! 

[dee.jay]

I too enjoy a career in Networking

It's great isn't it. Though it does mean I'm constantly faffing (much to my wife's annoyance)

[niemand]

Ah, now that I don't do if I can help it. The day job is quite enough without messing with things at home.

Once the 10 Gb/s upgrade is done and the high availability changes put in place it'll be left alone