Kitz Forum
Computers & Hardware => Networking => Topic started by: Chunkers on November 26, 2016, 05:36:02 AM
-
pfSense self-build router
mistake build
After reading through everyones posts on their pfSense routers in this thread (http://forum.kitz.co.uk/index.php/topic,18944.0.html) I got sucked into a familiar spiral of geeking out reading about people building their own pfSense routers,watching YouTube videos (thank you @underzone that was quite a few hours of my life ;) ) and trawling forums reading about peoples builds.
Along the way I know a little bit more about networking than I did before, a little, and you know what they say about "a little knowledge"
This culminated in me placing an order for a PCEngines 4Gb APU2 (http://pcengines.ch/apu2.htm) board, enclosure and PSU - I already have suitable mSATA SSD. I also have a couple of miniPCIE wireless cards, don't need it but might stick one in just for giggles.
I feel that a combination of superior hardware together with pfSense, which seems much more powerful and better maintained than OEM firmwares, will result in a superior device once I eventually get to grips with the configuration. The only modestly interesting thing about my home setup is that I have dual-WAN and need a load-balancing router - hopefully pfSense and my hardware will be more than up to the task.
My current router has a dual core MIPS 500 Mhz processor and 128 Mb of RAM, to be fair I have never seen the CPU load above 20%. Logic suggests Quad core @ 1 Ghz and 4Gb of RAM will jostle things along nicely.....
I see this a bit of fun, and an opportunity to develop my understanding of pfSense. So my plan is to gradually refine and test my pfSense router until I either give-up or it proves a better performer than my TP-Link TL-ER5120 which will remain in service and as a backup to avoid any family lack-of-internet related disasters. The low risk - high cost approach, I guess ...... I am an engineer after all.
On Black Friday I bought a managed switch to pair with the unit, a Netgear GS108E, VLAN's here I come!
I don't think I have a device old enough to still have a serial port! I read good reports about the Startech (https://www.amazon.co.uk/Startech-1-Port-Modem-Serial-Adapter/dp/B008634VJY/ref=sr_1_1?ie=UTF8&qid=1479981188&sr=8-1&keywords=startech+usb+null) USB ----> null modem cable and went out and bought it.
Here are a few questions :
- Do you run pfSense from RAM disk? Would have thought SSD would be plenty fast enough and I get the sense it is unnecessary.
- Any experience of dual / multi WAN and load balancing?
- I liked the idea of trying to use Squid / something else to cache Windows 10 / IOS / Steam / other updates but I am reading this doesn't work? hmmmm
Cheers big ears!
Chunks
-
No ram disk on mine. Just the ssd.
Amazon for usb to serial adapter ;)
-
Quick update :
Its built, pfSense is installed and running and TRIM is enabled but I won't have set up the dual WAN and other stuff and put it into the service until I get back from work at the end of January.
(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fi43.photobucket.com%2Falbums%2Fe352%2FJolltax%2F20161227_164215_zpsgayvi4vs.jpg&hash=d8eb1cd50fee10a83357bcf7d10fa38d5c3c1c9d)
EDIT : Here a linky to the boot sequence in a text file (http://www.zen101388.zen.co.uk/ChunkspfSenseBoot.txt) if anyone is feeling super helpful / geeky. I can't see anything disastrous in there, but then I don't really know what I am doing ... <smiles in blissful ignorance>
Exciting!
C
-
Looking good, what did the costs end up being?
-
Looking good, what did the costs end up being?
Good question, I haven't really worked it out (until now) ...
APU2C4 board + case + PSU = 118 (was actually 138.4 EU inc shipping)
Import taxes (bastards!) = 36
Null modem cable = 18 (I bought a relatively expensive Startech one)
----------------
£172
I already had an mSATA ssd and WLAN minPCI adapter although I haven't bothered fitting a WLAN CARD yet. Note : I had to buy through my company as they won't sell to public although it available from LinITX (https://linitx.com/category/linitx-firewalls/1086)for £201 shipped inc PSU which doesn't look too bad.
I think your QOTOM unit is better bang per buck tbh although I am sure my device will be overkill for my uses in any case!
Chunks
-
I have the same model as you. Works a treat. What version are you on. I have been running 2.4 plus the latest snapshots and not seen any issues. I did one clean usb/serial install straight to 2 4 and also an in-place upgrade from 2.33 via the gui. Both worked fine.
-
I have the same model as you. Works a treat. What version are you on. I have been running 2.4 plus the latest snapshots and not seen any issues. I did one clean usb/serial install straight to 2 4 and also an in-place upgrade from 2.33 via the gui. Both worked fine.
I haven't really started playing with it yet :
pfSense (pfSense) 2.3.2-RELEASE (Patch 1) amd64 Tue Sep 27 12:13:07 CDT 2016
Bootup complete
FreeBSD/amd64 (pfSense.localdomain) (ttyu0)
*** Welcome to pfSense 2.3.2-RELEASE-p1 (amd64 full-install) on pfSense ***
-
I would at least upgrade to 2.33 via the gui but as I have been on 2.4 for sometime you could again roll up to 2.4 via the gui in one go. Job done.
Image files here if you want to do it manually from scratch for 2.4
https://snapshots.pfsense.org/amd64/pfSense_master/installer/?C=M;O=D
2.33 image files.. https://snapshots.pfsense.org/amd64/pfSense_RELENG_2_3/installer/?C=M;O=D
I apply snap shot updates about once a week from the gui.
-
I think your QOTOM unit is better bang per buck tbh although I am sure my device will be overkill for my uses in any case!
Chunks
Not too dissimilar in price (£177 and I used an old SSD), although I did end up with twice the memory which won't make any difference, and I'm sure mine will be total overkill as well.
-
Hey,
Very happy at the moment, got my pfSense thingy up and running and it seems to be doing a great job :
(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fi43.photobucket.com%2Falbums%2Fe352%2FJolltax%2FCrap%2FUnitdyCrap_zpsbods2flv.jpg&hash=46b32301645fcdf81ada67f54a804bc5d87f93c4)
Note : use of old router as "stand", hehe
Some highlights / observations :
- Got TRIM enabled fairly easily once I worked out that I needed to be in "stupid mode" on the console (or whatever it was called)
- Getting dual WAN with failover to work and load balancing is faffy and annoying, weirdly there are no Load Balancing pools showing, but its deffo working. Anyway its rubbish to do, but seems to work well.
- I have loaded up Squid and its running with 1Gb memory and 8 Gb HD cache, just for fun really, and also because I might use SquidGuard to prevent my kids from accessing *nasty* on the internet - was surprisingly painless compared to the other stuff
- The APU2C4 has been rock solid and is barely warm to the touch, it even survived Mrs Chunks putting a pile of books on top of it while it was running "I wondered what that was". Max load I have seen is 27% CPU, typically its at 0%. I haven't set up the CPU temp monitoring because it looks like a major pain in the arse - I'll wait for it to be properly supported, maybe in a future release?
- Setting up all the DHCP reservations was a bit annoying also, why can't you just cut and paste or upload a list in TXT file or something
- I got my port forwarding sorted which seems to require you to forward the port twice when you have dual WAN, one for each WAN ... pfft
- To do list : I'm going to have all my servers and other junk on my network have their own local DNS names, haven't got round to doing this, I want to set up SquidGuard, remote web access (I think I just have to open a port for HTTP access on the correct port)
- ClamAV seems to give me problems, I guess I need to read more wiki's to get it to work properly
- So many more packages to fiddle with ......
So its working great, and the web interface is sexeh
One thing though : pfSense is definitely not "easy", I have had to do an unpleasant amount of geeky googling to get things working. I guess its an investment in the future.
Quote from Mrs Chunks : "Why do you always have to make everything so complicated, everyone else just has one little box"
Please advise on correct response.....
(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fi43.photobucket.com%2Falbums%2Fe352%2FJolltax%2FCrap%2FPFChunksJan17_zpsmugcanix.jpg&hash=a319759534fe96a83d7fd33f570c88756a7d0649)
IP addresses removed because you are all hackers ....
Yeeeeeeeeeeeeeeeeeeeeeeeeeeeeeehaw
Chunks
-
Quote from Mrs Chunks : "Why do you always have to make everything so complicated, everyone else just has one little box"
Please advise on correct response.....
I would suggest something like: "Because <insert correct and appropriate phrase here>, I do things properly."
As for the <correct and appropriate phrase>, Basil Fawlty would use something like "my little nest of vipers".
-
Nice..
If you want to use the vpn you can run the vpn wizard.. I use the open vpn option. Once you have set it up you will need to install the open vpn client export from package installer. Add the user and tick the certificate box. Then go back to the vpn menu and click on client export. Scroll down and export the settings...
I gathered the basics for setting up the open vpn bits from this but ignore the viscosity bit.
https://www.sparklabs.com/support/kb/article/setting-up-an-openvpn-server-with-pfsense-and-viscosity/
-
Looking good, getting the basics up and running is pretty straightforward, but there is so much to learn (read learn & then forget!) and take in for every thing else, I still have loads I need to do.
-
I was quite fond of LinITX PC-Engines pfSense boxes. An 18 month old 2 core 4 GB APU + SSD pfSense box at home has run perfectly since the outset, when I replaced a HP Microserver running FreeBSD+IPFW. Interestingly, the pfSense main display identifies this as a Netgate APU (which they did once sell) but it is not badged as such.
However, here at my mothers:
- I decided that a 6 month old ALIX (i386) box needed replacement as pfSense 2.4 won't support i386 and nanoBSD is bit of a PITA. Other than that it was fine, other then needing a new CF card now and again.
- The APU box that replaced it would crash every few hours to days. Returned, came back no fault found (and initially no PSU). Still crashing. Eventually returned for a refund.
- While I liked this tiny box, the Netgate SG-1000 was generally slow for speed tests, and specifically very slow for the TBB single-streamed one. People are whinging a bit about this on the pfSense forum. I hope it is a software rather than hardware problem. Returned for an upgrade to a SG-2220 ("reassuringly expensive"?). This should arrive tomorrow. A good thing as ...
- The ALIX box has also started crashing with nothing on the console (much like the APU), suffering filesystem corruption only repairable by rewriting the CF card and restoring the configuration.
Power here in the wilds of Somerset is a bit erratic. I have a UPS but I fear I need to check out everything connected to it.
Do you run pfSense from RAM disk? Would have thought SSD would be plenty fast enough and I get the sense it is unnecessary.
The configuration option for pfSense to use RAM disks for /var and /tmp is there to prevent wear failure on CF cards and the like. It is indeed thought unnecessary to run pfSense on SSDs with /var and /tmp in RAM disks. pfSense of itself doesn't do that much in the way of writes anyway and nowadays SSD write endurance is much better than CF card.
If you mean running more of the system out of a RAM disk, I don't recall seeing anything like that and it would probably be quite difficult to achieve, and harder to update. Look at the way ESXi boots itself :o
Any experience of dual / multi WAN and load balancing?
Some. At home I had a FTTC line of my own and an ADSL line supplied by my employer (until they decided they would leech for free on my connection).
Multi-LAN can be configured for failover which usually works provided you choose an appropriate IP address to monitor. Something always up at your ISP. Not something distant where far away contention can cause failover. 8.8.8.8 and 8.8.4.4 (Google DNS) are not good choices!
Load balancing doesn't work as you might hope it to, especially on wildly disparate speed WANs where you can end up with something which you want fast on the slow line, and if you use HTTPS you will need to use "sticky connections". Works pretty well for torrents :D
Eventually I settled on a failover configuration, with VPN traffic to my employer being specifically aimed at the group with the ADSL connection as the primary.
When I home for long enough I intend to see if I can use USB WiFi to a tethered phone for fall-back.
I liked the idea of trying to use Squid / something else to cache Windows 10 / IOS / Steam / other updates but I am reading this doesn't work? hmmmm
At work I had to set up a hierarchy of Squid servers mainly for getting ClamAV AV definitions and FreeBSD source and package tarballs via a slow internet connection of our own, rather than the fast corporate connection when this was switched to using proxies which only supported NTLM authentication. Squid does take some configuring. Out of the box it broke SVN and cached ClamAV definitions for far too long.
Squid is likely to take quite some tinkering to get it working for your needs.
If you have Windows 10 machines, you might try the "get updates from other machines on my LAN" option.
Other things:
At home, my pfSense box connects with PPoE to a HG612, so no need for a router/modem/access point. Though I have another network segment and switch for a samknows box and for access to the HG612's second port for monitoring. WiFi access points are connected to my internal network.
Here at my mother's I have a BT FTTC line, a Home Hub 5B, a BT YouView box, a Vodafone Sure Signal 3, a Fon access point and the pfSense box de la semaine as the HH5B's DMZ box. Getting the Sure Signal to work with the HH5B was a nightmare. UPNP did not work. Port forwarding needed a startlingly large number of ports. As I was under the desk, today I plugged the Sure Signal into the LAN and removed its port forwarding from the HH5B. Sure Signal working nicely. I will probably keep the HH5B (so I keep BT Wifi), the Fon access point (why not?) and leave the YouView box connected to the HH5B.
The pfSense <> pfSense VPN connection between home and mother's home has been rock solid.
The DNS Override facility in the DNS Resolver (containing both my home BIND DNS server IPs) allows me to access home machines from my mothers by FQDN. Non Windows DHCP machines here pick up the DNS search list so I can access home machines just by name. I'll have to edit the Windows registry for my desktop.
There certainly is a lot to play with and learn from, but "if it ain't broke don't fix it" has a lot going for it too.
The configuration backup and restore facility is wonderful. Do make sure you have backups of working configurations and know what they are.
-
There is a couple of unhappy people on the pfsense forums who brought the official pfSense hardware only to find out it ships with the testing version of pfSense because the official hardware is not supported on the stable build. I think they going to be waiting a while for 2.4 to become gold due to the reported kernel panic's and the nasty traffic shaping bug I discovered (which someone else now confirmed and I am still taking heckles to this date even with the bug confirmed for making it public).
Skyeci has also had issues with his apu unit crashing and as far as I know is now downgraded back to 2.3 to see if it becomes stable.
I had one panic on my braswell unit using pfSense 2.4. A bit of investigation returns the result that there is various people reporting kernel panic's on FreeBSD 11 and pfSense 2.4 (which is based on FreeBSD 11), the one common factor is every single report is using igb network chip's. With a reported workaround been to limit the igb queue depth to just 1 queue. Which is what I have done and so far no more panic's touch wood.
Another issue with the official pfSense kit is I think its overpriced, its a very tiny device with a low spec for the value.
In regards to pfSense GUI itself, yes it is much more settings heavy than consumer stuff like asuswrt, billion, netgear etc. It even lets you tune stuff that I havent seen in ddwrt and tomato usb firmwares. For example I will admit I got stuck with ipv6, and skyeci told me I needed to enable a tracking option which makes the pfSense unit correctly request the ipv6 prefix and give itself a WAN ip.
I also agree the port forwarding is a bit unusual in how it has to be setup, having to set port's twice and also having to setup port aliases, if a service has multiple ports not in a single range. But once you aware of the system in use its not a big problem.
To setup temp monitoring I think is just one option that has to be ticked in the settings, then you should see the core temperatures on your dashboard.
I will be adding snmpd to my unit soon so I can graph everything. :)
another guy who confirmed 1 igb queue halted the kernel panic's.
https://forum.pfsense.org/index.php?topic=123957.msg685254#msg685254
-
Yes, I can see people who bought a SG-1000 without knowing it was running beta software being miffed. I did not mind that as I new before I bought it, but was really miffed about its throughput.
I think that the pricing of the Netgate units may be because of the "free" incentives. The SG-1000 comes with a 1 year Gold subscription. Its two bigger brothers come with 2 paid support incidents for a year. I assume that Netgate are including part of the cost of this in the price. I have not looked at the costs and incentives of the unaffordably more expensive items.
-
To setup temp monitoring I think is just one option that has to be ticked in the settings, then you should see the core temperatures on your dashboard.
I checked the box and got nothing.... according to the PCEngines forum the kernel doesn't contain the correct ID for the AMD processors so I did this hacky thing to make it work (https://forum.pfsense.org/index.php?topic=108262.0).
I transferred the files using SFTP because when i tried to use USB my box went mental, crashed and the rebooted. Won't be trying that again...
Anyway it works now and my temps are all 53C, not bad for a passive device, especially as the cooler is pretty bodgey, uses the case as a passive HS and seemed to be an old design made for the original APU board.
If you have Windows 10 machines, you might try the "get updates from other machines on my LAN" option.
Good idea, I have enabled this.
The annoying thing about Squid is that is doesn't do what I REALLY wanted it to do, cache Apple, Windows and game updates but never mind .....
My understanding is that Apple and MS have made this very difficult requiring dedicated servers etc, I guess they got their reasons.
Chunks
-
adrian it seems also the advertised AES offloading doesnt work in pfsense either due to a lack of a driver, its a bit of an odd choice of hardware to promote.
Chunkers seems it needs a bug report to ask for the amdtemp kernel module to be default compiled on pfSense kernel.
I wont be doing the bug report tho due to the silly responses I got last time.
-
On my apu2 the temp monior does not report anythung under 2.3.2 p1. When the box was on 2.4 though the temp monitor worked just fine.
I guess something was added in the 2.4 beta..
-
adrian it seems also the advertised AES offloading doesnt work in pfsense either due to a lack of a driver, its a bit of an odd choice of hardware to promote.
Which hardware are we talking about? I've given up on the SG-1000 in favour of a SG-2220. Everything seems to work, and for the price it should!
From SG-2220 serial console
cryptosoft0: <software crypto> on motherboard
From web console
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
I did have to tweak some settings in System/Advanced/Miscellaneous, possibly due to my doing the bare minimum to get the box connected and then restoring its configuration from an ALIX backup. Beats setting up from scratch hands down.
I wont be doing the bug report tho due to the silly responses I got last time.
Should I ever submit a pfSense bug, I'll have it polished (sand blasted?) in the forum first.
-
The SG-1000 I am talking about, is some posts on the pfSense forum about the non working AESNI support.
Also yeah I can confirm amdtemp.ko exists on 2.4, so they recitified that in 2.4.
-
Probably best that we stop talking about the SG-1000. As far as I know, I was the only person here using one, and that was for just a few days.
I'll be sticking on 2.3.2-RELEASE-p1 (AMD) on both my AMD boxes until a new release arrives (2.4 might not be far off).
- The SG-2220 down here because it has a Netgate image (notably is not badged Community Edition, has additional Wizards, and possibly some specific tweaks for Netgate hardware), and the only images Netgate release are -RELEASE and -RELEASE-p*. No beta images. When the time comes I will upgrade rather than reinstall.
- The LinITX LinITX APU 1D 4GB (3NIC+USB+RTC) at home because, aside from me being here and it being 160 miles away there, I am really dependant upon it.
I might try to convert my LinITX ALIX 2D2 LX800 (2NIC+USB) box from nano to normal to keep as a back up, but I have rather gone off it, having been forced to rewrite the CF card(s) and reconfigure too many times over the past fortnight. Probably better for me to keep it as-is as a backup for here, and spring for a SG-2440 (gulp) for home, with the APU as a backup.
Or (finally getting somewhat back on topic), what is the current best affordable non APU serial port console (say no to displays and keyboards) amd64 "self build" with at least 3 Intel NICs? I can cope with adding cards, memory, etc, but putting processors onto motherboards or motherboards into cases is something I very much prefer not to do.