Kitz Forum

Computers & Hardware => Networking => Topic started by: Chrysalis on November 16, 2016, 10:36:57 PM

Title: LAN setup
Post by: Chrysalis on November 16, 2016, 10:36:57 PM
Following the post here http://forum.kitz.co.uk/index.php?topic=18942.msg337615#msg337615 which I do not want to hijack, I thought I would show my proposed new LAN which will be live when I built my PFSense device.

So The pfsense box is this baby http://amzn.eu/6Y5E48h
Ram and SSD are already here.  The SSD is way overkill but is a 60gig mini sata SSD (30gig is same price so 60gig is no brainer).

(https://forum.kitz.co.uk/proxy.php?request=http%3A%2F%2Fi.imgur.com%2F7ClEpXm.jpg&hash=de8f0dd5d15cb2219cb9183c15f18e12)
Title: Re: LAN setup
Post by: d2d4j on November 16, 2016, 11:06:28 PM
Hi chrysalis

We use pfsense on some commercial setups, usually on dell poweredge xion dual quad core, 16gb ram, with raid 60 drives, and 2 x 4 1000mb nic cards, as well as onboard 4 1000 nic cards. Works lovely on multi nodes

We also use other firewalls and setups, but for home use, your setup looks ok

I do not use Sam knows box for anything other then Sam knows, and our setup is dynamic for sam knows, so it does not interfere with our IP range

Many thanks and I hope you don't mind my post, but please delete it if you want

Many thanks

John
Title: Re: LAN setup
Post by: Chunkers on November 17, 2016, 09:52:18 AM
Very cool, have always wanted to fiddle with pfsense but have never been brave enough as a primary device.

As a (relevant) aside, personally I have had bad experiences of the passive Celeron microPC's, in my case the Asrock Beebox N3000 which has problems with heat dissipation under any kind of load and also very poor wireless and bluetooth performance - admittedly I was running Windows 10 on it and using it as a HTPC.  I have since switched it to LibreElec and it is much better but I have had to remove the wireless cards and use an external dongle instead and also take out 2.5" disk to keep it cool.

Yours is quite likely a better design that the Beebox and i imagine will be running at a low load but worth having a snoop round perhaps:)

o7

Chunks
Title: Re: LAN setup
Post by: skyeci on November 17, 2016, 10:50:33 AM
love my pfsense box.

I use  pc engines apu2 with 4gb ram and 30gb mini ssd. Works a treat with my sky fibre pro too, supports sky IPV6 along with  open vpn works really well as I got my static IP before sky stopped offering them.
Its 12volt as well so low power use.

Spec http://pcengines.ch/apu2c4.htm (http://pcengines.ch/apu2c4.htm) 
CPU AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache.

Not seen it break into a sweat yet..

Purchased it from LinITX in the UK - came pre built and pre-installed with a full version of Pfsense.

if anyone wants sky settings PM me should you fancy having a go..

sorted  :cool:
Title: Re: LAN setup
Post by: Ronski on November 17, 2016, 08:41:43 PM
Tempted to build one of these, although would probably go for what skieci has, but purchase direct from PCEngines for just over half the price that LinITX want, although that would mean I have to work out how to install Pfsense
Title: Re: LAN setup
Post by: underzone on November 17, 2016, 08:45:27 PM
All the info you will ever need in this series of vids:

https://www.youtube.com/watch?v=agieD5uiwYY&list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk (https://www.youtube.com/watch?v=agieD5uiwYY&list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk)

 ;)
Title: Re: LAN setup
Post by: skyeci on November 17, 2016, 08:49:33 PM
Tempted to build one of these, although would probably go for what skieci has, but purchase direct from PCEngines for just over half the price that LinITX want, although that would mean I have to work out how to install Pfsense

I have a usb install stick I made here with serial console support install if you want to borrow it :)

I was having so much packet loss with my old asus I just needed a next day solution to resolve the issues quickly  ::) I did run it on my i7 with 2 lan cards pre purchae - worked just fine but the power consumption was a bit crazy as it was my gaming pc..
Title: Re: LAN setup
Post by: Chrysalis on November 18, 2016, 02:09:38 AM
Tempted to build one of these, although would probably go for what skieci has, but purchase direct from PCEngines for just over half the price that LinITX want, although that would mean I have to work out how to install Pfsense

I read your post and I was thinking hmmmm.

Because I did check the price of the APU2 before (someone else recommended it to me) was over 200 hence what I got instead, but then thought some more about your post and you mentioned direct, and indeed I noticed the buy direct link and its significantly cheaper than the reseller and cheaper then what I have ordered so is the better choice.  Sort of feel an idiot now but I will persist with what I ordered.  What I ordered does come with the casing and is more compact so its not a loss all round.
Title: Re: LAN setup
Post by: Ronski on November 18, 2016, 06:28:50 AM
Thanks Underzone for the link, and skyeci for the offer - I'll see how I get on.

Chrysalis, I only stumbled upon the buy direct option because I was reading this thread (https://hardforum.com/threads/anyone-tried-a-pc-engines-apu2c4-yet.1901041/) and someone asked how he got it so cheap. You can also buy a case from them as well, in a choice of colours. I prefer the looks of yours, but the APU2 has three NICS, so if I ever got a second line it could load balance as well. I'll probably go back to using the HG612 as the modem, my upload is now down to 4192kbps, whilst D/S is now at 50412kbps, and I know the HG612 will give me more upload.
Title: Re: LAN setup
Post by: Chrysalis on November 18, 2016, 02:38:38 PM
The annoyance with mine is it has realtek nic's and the FreeBSD drivers for realtek are not great, I will probably run them with offloading disabled as my experience dictates.  Whilst the APU2 nic's are all intel.  The cpu on the box i am buying however will be fine without checksum offloading its easily powerful enough.

The APU2 is also more future proof as you said with the extra NIC but not just that but also the extra mini pci-e connectivity on board for future addon cards.
Title: Re: LAN setup
Post by: skyeci on November 18, 2016, 03:43:37 PM
Thats also why I went for apu2, 3 Intel lan ports  :)
Title: Re: LAN setup
Post by: Chunkers on November 18, 2016, 06:49:13 PM
All the info you will ever need in this series of vids:

https://www.youtube.com/watch?v=agieD5uiwYY&list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk (https://www.youtube.com/watch?v=agieD5uiwYY&list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk)

 ;)

These are great, thanks for the link.  I am starting to get excited about the possibility of building my own router :)
Title: Re: LAN setup
Post by: underzone on November 18, 2016, 09:13:03 PM
Yeah me too. When you watch the vid and he says how most routers are 400MHz and 128Mb RAM it makes you realise how good a standalone box @2GHz with 8GB and an SSD could be!
Title: Re: LAN setup
Post by: Chunkers on November 19, 2016, 09:42:00 PM
Thanks Underzone for the link, and skyeci for the offer - I'll see how I get on.

Chrysalis, I only stumbled upon the buy direct option because I was reading this thread (https://hardforum.com/threads/anyone-tried-a-pc-engines-apu2c4-yet.1901041/) and someone asked how he got it so cheap.

I don't want to highjack this thread but have a couple of questions based on your comment and reading through the thread as I really like the idea of getting one, at 140 it seems like a good deal as the 3 NIC version could load balance my dual WAN's:

Feel free to feedback if this needs to broken out as a separate thread, I am pretty sure I will be pricing up and getting some hardware for this myself

Chunks
Title: Re: LAN setup
Post by: Chrysalis on November 20, 2016, 03:03:28 AM
My unit has more cpu grunt, but my opinion is the apu2 also has enough grunt comfortably to do its job, both have cpus way more powerful than current high end retail consumer routers.

If I have to I will add intel nic's via the mini pcie connector or even buy an apu2 at a later date, hopefully I am fine on the realtek's tho.

As a comparison my ac68 is running at 1200mhz (overclocked, 800 is stock), but it has no aesni acceleration and has worse performance per clock than both systems.

A big factor in my decision is that the dev of the firmware on my asuswrt has started to imply he is not willing to do bug fixes that only I have found (he is moving into only fixing for the masses mode) and that a lot of the software behind asuswrt is reliant on very old code due to a locked down closed source driver from broadcom.  I had to manually apply some workarounds to get ipv6 fully stable on my sky connection, and the dev refuses to even acknowledge its broken so that was the trigger point for me. Is a shame the unit I have ordered is using realtek nic's as otherwise I would have considered it perfect (assuming one is ok using a separate WAP and gigabit switch).

Bear in mind regarding the number of network ports, pfsense has very good vlan support, it even has a wizard when you first boot it guiding you through vlan setup, this is all done in mind that alot of people will be running pfsense on devices with limited ethernet ports and can allow things like bonded connections to run via a single ethernet cable, its how I managed to as an example share my wan with lan access to my billion 8800nl stats over one cable.
Title: Re: LAN setup
Post by: Ronski on November 20, 2016, 08:31:48 AM
Bear in mind regarding the number of network ports, pfsense has very good vlan support, it even has a wizard when you first boot it guiding you through vlan setup, this is all done in mind that alot of people will be running pfsense on devices with limited ethernet ports and can allow things like bonded connections to run via a single ethernet cable, its how I managed to as an example share my wan with lan access to my billion 8800nl stats over one cable.

According to the videos linked to earlier you will need a managed switch to use VLANs for the above.
Title: Re: LAN setup
Post by: d2d4j on November 20, 2016, 11:51:54 AM
Hi

I hope you don't mind my thoughts, but we are just talking home network, not commercial grade networks, so my best advice with all networking is to keep it as simple as possible.

This helps with setup and problem solving

If you would like to experiment with networking, I would advice you setup a test network for this purpose and leave your home network alone

Many thanks

John
Title: Re: LAN setup
Post by: Ronski on November 20, 2016, 01:56:21 PM
Hi John, I don't mind your thoughts, but if that was the case we'd still all be using ISP supplied routers, and I think the majority of us posting on here are savy enough to have networks a bit above the most basic, and if we're not then we can learn. I have a server at home, and I'd prefer a much better firewall than my modem/router provides, I often get people from all over the world trying to login to my server, so once I have Pfsense set up most of the world will be blocked/dropped :-)
Title: Re: LAN setup
Post by: d2d4j on November 20, 2016, 02:34:55 PM
Hi ronski

Many thanks

Sorry, I was meaning more in terms of creating vlans and more in depth networking.

I apologise for not been very clear

Many thanks

John
Title: Re: LAN setup
Post by: Ronski on November 20, 2016, 03:07:13 PM
Hi John, no need to apologise I was just thinking whilst stripping the wall paper that you probably meant vlans.

Basically your saying don't complicate it for the sake of it, only if you need to.
Title: Re: LAN setup
Post by: gt94sss2 on November 20, 2016, 05:17:43 PM
Looking at the network map, isn't all the wired traffic supposed to go via the Sam Knows box (as one of the requirements for using it?)

Title: Re: LAN setup
Post by: burakkucat on November 20, 2016, 05:26:12 PM
Looking at the network map, isn't all the wired traffic supposed to go via the Sam Knows box (as one of the requirements for using it?)

Hmm . . . I noticed that a few days ago but failed to mention it.  :-\
Title: Re: LAN setup
Post by: d2d4j on November 20, 2016, 05:36:31 PM
Hi

I could be wrong, so would need to check back but my understanding was the reason for Sam knows to have traffic flowing through it, was so the tests that Sam knows runs, are done at a time of less traffic on your network

Our Sam knows runs under a different external IP to our range, so it runs whenever it needs

Many thanks

John
Title: Re: LAN setup
Post by: burakkucat on November 20, 2016, 05:49:10 PM
Our Sam knows runs under a different external IP to our range, so it runs whenever it needs

Ah, so that might be how Chrysalis has things set-up within his own domain.
Title: Re: LAN setup
Post by: Chrysalis on November 20, 2016, 08:14:18 PM
Looking at the network map, isn't all the wired traffic supposed to go via the Sam Knows box (as one of the requirements for using it?)



it is, but the reason for that is so it always knows when the connection is active.

it doesnt stop the tests from been carried out, but I suppose runs the risk of a skewed result during a download.
Title: Re: LAN setup
Post by: skyeci on November 22, 2016, 09:21:03 PM
LOL just checked the disc space on my ssd. 30GB disc installed & pfsense is using less than 1GB. ;D

Have implemented the mod to keep the sky ip6 stuff on reboot/patch upgrade - working just great after a few reboots to test
ipv4 & ipv6 pinger now working too with a few rules added to the firewall to get it setup properly
Title: Re: LAN setup
Post by: Chrysalis on November 22, 2016, 10:28:50 PM
glad you happy with the mod as I am the one who provided the source files for it ;)
Title: Re: LAN setup
Post by: skyeci on November 22, 2016, 11:28:10 PM
 :thumbs: :thumbs:
Title: Re: LAN setup
Post by: Chrysalis on November 23, 2016, 05:44:00 PM
looks like its finally at the uk depot

Quote
Hi Chris, you have a parcel coming tomorrow.

Scheduled Delivery Date:  Thursday,  24/11/2016

Estimated Delivery Time:  End Of Day
Title: Re: LAN setup
Post by: adrianw on November 24, 2016, 01:26:08 AM
I have been using pfSense for some time with a 64 bit LinITX box and a HG612. No router. Works very well.
More recently elsewhere I have been using pfSense with a 32 bit LinITX box and a BT HH5. Also works well, but I am in the throws of upgrading to a 64 bit box, as pfSense are dropping 32 bit support. As I understand it, the 32 bit release will receive security patches but nothing else.
Title: Re: LAN setup
Post by: Ronski on November 24, 2016, 08:16:44 PM
Well I decided to go for one of these in the end, the version which takes 2.5" SSD as I have an 80GB Intel SSD that's doing nothing.

https://www.amazon.co.uk/dp/B01GBHC62K/ref=pe_385721_37986871_TE_item
http://www.qotom.net/goods-129-QOTOM-Q190G4+4+LAN+Mini+PC.html

There's a thread on the PFsense forums (https://forum.pfsense.org/index.php?topic=114202.0), does it really need 8GB of memory if running quite a few packages?

The PC Engines one ended up looking like it'd cost about 180 including delivery if it got picked up for import duty as Switzerland is not in the EU.
Title: Re: LAN setup
Post by: Chrysalis on November 24, 2016, 11:08:38 PM
just for the curious my mini keyboard works as well with this. Works in the bios etc.

Rii i8+ 2.4Ghz LED Backlit Mini Wireless Keyboard With Touch Pad Mouse UK Layout With Built-in Rechargeable Battery Black
https://www.amazon.co.uk/gp/product/B00T2SJUWA/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 (https://www.amazon.co.uk/gp/product/B00T2SJUWA/ref=oh_aui_search_detailpage?ie=UTF8&psc=1)

Also inside there is a connector and power cable for a normal sata storage device although I recommend a msata ssd like I got and actually removing those sata cables.
Title: Re: LAN setup
Post by: Chrysalis on November 24, 2016, 11:18:33 PM
Well I decided to go for one of these in the end, the version which takes 2.5" SSD as I have an 80GB Intel SSD that's doing nothing.

https://www.amazon.co.uk/dp/B01GBHC62K/ref=pe_385721_37986871_TE_item
http://www.qotom.net/goods-129-QOTOM-Q190G4+4+LAN+Mini+PC.html

There's a thread on the PFsense forums (https://forum.pfsense.org/index.php?topic=114202.0), does it really need 8GB of memory if running quite a few packages?

The PC Engines one ended up looking like it'd cost about 180 including delivery if it got picked up for import duty as Switzerland is not in the EU.

Your find made me realise I suck at searching for stuff :p , you guys all making me feel down with the realtek nics :/

Nice find tho, looks good.  The cpu has no AES tho but at the same time the cpu has a decent amount of raw grunt so it should be able to handle vpn at FTTC speeds.

Also 4 gig should be plenty of ram.
Title: Re: LAN setup
Post by: Chunkers on November 25, 2016, 05:17:32 AM

The PC Engines one ended up looking like it'd cost about 180 including delivery if it got picked up for import duty as Switzerland is not in the EU.

I'll be able to verify this as I am currently trying to get one ordered direct at the moment, the price for a 4Gb board, case and UK PSU is EU 121 + P&P plus I will have to pay VAT, I think. So I am guessing price will be around 135 ish delivered.

I was not able to order without a VAT number so am ordering via family business

I've already git an mSATA SSD

Chunks
Title: Re: LAN setup
Post by: Chrysalis on November 25, 2016, 05:26:26 AM
ok some information for those using pfsense with an ssd.

The pfsense devs are using an old way to configure partitions, in short they -

dont enable trim
dont align partitions to 4k alignment
and they also enable SUJ which I think is best disabled on an ssd.

to fix the alignment follow this guide https://forum.pfsense.org/index.php?topic=86139.0
if you try to do it all manually pfsense will rewrite the partitions, but I have confirmed the above guide does lead to a 4k aligned partition.

The commands to enable trim and disable soft updates journaling are.  (assuming the ssd is on ada0, which it will be if its the only sata device)

Code: [Select]
tunefs -t enable /dev/ada0s1a
tunefs -j disable /dev/ada0s1a

Less important is to have the swap on a swapfile not partition as the partition wont utilise trim, but I think its unlikely a pfsense box with at least 4 gig of ram will even use the swap.

bottom of this page is a couple of commands to make the swapfile http://www.wonkity.com/~wblock/docs/html/ssd.html

so

Code: [Select]
mkdir /usr/swap (although can be put anywhere you like)
Code: [Select]
dd if=/dev/zero of=/usr/swap/swap bs=128k count=16384 (2 gig swap)

add these 2 lines to /etc/fstab

Code: [Select]
# Device        Mountpoint      FStype  Options                         Dump    Pass#
md99            none            swap    sw,file=/usr/swap/swap,late     0       0

then after run
Code: [Select]
swapon -aL
Title: Re: LAN setup
Post by: Ronski on November 25, 2016, 07:03:06 AM
@Chunkers, I allowed 40 Euro for shipping, if shipping was 30 Euro, then with import duty it would come to 153. There's a handy calculator here (https://www.dutycalculator.com/new-import-duty-and-tax-calculation/saved_calculations/view_details/214304922/) (one time use). There's also the possibility the courier will charge a handling fee.

@Chrysalis. Thanks for that very useful information, does this change the swap file from what's set up during installation. You'd have thought the dev's would have made the OS SSD friendly by now.
Title: Re: LAN setup
Post by: Chunkers on November 25, 2016, 07:14:23 AM
ok some information for those using pfsense with an ssd.

The pfsense devs are using an old way to configure partitions, in short they -

dont enable trim
dont align partitions to 4k alignment
and they also enable SUJ which I think is best disabled on an ssd.

to fix the alignment follow this guide https://forum.pfsense.org/index.php?topic=86139.0
if you try to do it all manually pfsense will rewrite the partitions, but I have confirmed the above guide does lead to a 4k aligned partition.

The commands to enable trime and disable soft updates journaling are.  (assuming the ssd is on ada0, which it will be if its the only sata device)

Code: [Select]
tunefs -t enable /dev/adas0p1
tunefs -j disable /dev/adas0p1

Less important is to have the swap on a swapfile not partition as the partition wont utilise trim, but I think its unlikely a pfsense box with at least 4 gig of ram will even use the swap.

bottom of this page is a couple of commands to make the swapfile http://www.wonkity.com/~wblock/docs/html/ssd.html

so

Code: [Select]
mkdir /usr/swap (although can be put anywhere you like)
Code: [Select]
dd if=/dev/zero of=/usr/swap/swap bs=128k count=16384 (2 gig swap)

add these 2 lines to /etc/fstab

Code: [Select]
# Device        Mountpoint      FStype  Options                         Dump    Pass#
md99            none            swap    sw,file=/usr/swap/swap,late     0       0

then after run
Code: [Select]
swapon -aL

This answers one of my (many) questions regarding pfSense config and also the setup of the APU2 box when it arrives, to avoid cluttering this thread with my questions I'll start a new thread - really appreciate the posts from you guys as they will help me a lot as I thrash around trying to make it work, lol.

C
Title: Re: LAN setup
Post by: Chrysalis on November 25, 2016, 08:17:37 AM
@Chunkers, I allowed 40 Euro for shipping, if shipping was 30 Euro, then with import duty it would come to 153. There's a handy calculator here (https://www.dutycalculator.com/new-import-duty-and-tax-calculation/saved_calculations/view_details/214304922/) (one time use). There's also the possibility the courier will charge a handling fee.

@Chrysalis. Thanks for that very useful information, does this change the swap file from what's set up during installation. You'd have thought the dev's would have made the OS SSD friendly by now.

in my case I had configured a 2 gig swap file during install, if you wish to disable it then do this.

open /etc/fstab

comment out the line as follows.

Code: [Select]
#/dev/label/swap0               none            swap    sw              0       0
then run
Code: [Select]
swapoff -a
or simply reboot

if you doing this after you had already added the swapfile then the above command would disable both so run swapon -aL again after or the reboot will take care of it.

Back to my box, I have sourced a dual intel nic mini pcie card, its pricy tho at 50 notes, but the option is there if the box is unstable using realtek.  If purchased it would take the cost of this unit all in, including ram, ssd and the 2xintel nic to about 200 with it been a 4 nic unit (2xrealtek and 2xintel)
Title: Re: LAN setup
Post by: roseway on November 25, 2016, 10:26:17 AM
You may well have already found this information, but according to the FreeBSD documentation at https://www.freebsd.org/relnotes/CURRENT/hardware/article.html#ethernet :

Quote
The re(4) driver supports RealTek RTL8139C+, RTL8169, RTL816xS, RTL811xS, RTL8168, RTL810xE and RTL8111 based Fast Ethernet and Gigabit Ethernet adapters including:

    Alloy Computer Products EtherGOLD 1439E 10/100 (8139C+)

    Compaq Evo N1015v Integrated Ethernet (8139C+)

    Corega CG-LAPCIGT Gigabit Ethernet (8169S)

    D-Link DGE-528(T) Gigabit Ethernet (8169S)

    Gigabyte 7N400 Pro2 Integrated Gigabit Ethernet (8110S)

    LevelOne GNC-0105T (8169S)

    LinkSys EG1032 (32-bit PCI)

    PLANEX COMMUNICATIONS Inc. GN-1200TC (8169S)

    TP-Link TG-3468 v2 Gigabit Ethernet (8168)

    USRobotics USR997902 Gigabit Ethernet (8169S)

    Xterasys XN-152 10/100/1000 NIC (8169)
Title: Re: LAN setup
Post by: Chrysalis on November 25, 2016, 10:52:41 AM
sadly the FreeBSD realtek driver is basic only.  Most stuff that can be tuned on intel nic's is not tunable such as flow control, interrupt moderation and msix.  In addition the driver is buggy that when offloading is enabled it doesnt behave normal.  Otherwise the nic's perform providing the pps load is moderate only :)  I dont expect my home setup to exceed that load and I think will be ok, I have already disabled checksum offloading on pfsense so will see how things go after I have configured the unit and it takes over the router duties, that is a while off yet tho as I have a lot of configuration to import, my router does more then just forwarding packets, it filters malware, mobile ads and performs QoS duties :)  In addition to been my vpn endpoint as well.

I have ran some openssl benchmarks, with AES acceleration disabled its about 12-13 times faster than my ac68 (which itself is overclocked by 50%). With it enabled its about 740 times faster :)
Title: Re: LAN setup
Post by: phi2008 on November 30, 2016, 02:01:14 PM
My router was running VyOS for quite a while(which I really prefer over pfSense - but it doesn't have the community or GUI) and then I switched to pfSense.

I went for a Mini-ITX build with no moving parts, initially I chose a ASUS N3150 board for AES-NI, then I swapped it for an ASROCK AMD board which had a longer PCI-E expansion slot - I use a dual port Intel NIC and avoid the onboard Realtek.

Interesting upside to the AMD board is the AMD A4-5000 CPU - bit worse power consumption, pretty much the same performance (http://cpuboss.com/cpus/Intel-Celeron-N3150-vs-AMD-A4-5000) as the N3150, except in crypto where it hammers the N3150 -

(https://s14.postimg.org/g388oo7o1/crptamd.png)
Title: Re: LAN setup
Post by: Chrysalis on November 30, 2016, 02:05:20 PM
by the way the aes score on cpuboss is a typo, they put a . instead of , making the value seem really low :)
Title: Re: LAN setup
Post by: underzone on November 30, 2016, 02:27:52 PM
Does anyone know if you can use BT TV with a pfSense setup easily? I assume you could use one dedicated LAN port just for the set top box multicast, but like they say - never assume...
Title: Re: LAN setup
Post by: Chrysalis on December 01, 2016, 10:53:30 AM
The mini pcie adaptor just arrived, I decided to buy it as only one source had it in the UK and with limited stock (intel chip variant), better to get it now, then decide later I want it and not been able to source it.
Title: Re: LAN setup
Post by: Chrysalis on December 01, 2016, 12:30:15 PM
This is hard to add whilst keeping things tidy, the issues are.

pci bracket obviously has nowhere to go, I was aware of this of course before hand.
the cables connecting the bracket to the mpcie adaptor are much thicker and longer than I expected, they also are not very flexible.

Right now I have got one end of the unit resting on top of the bracket which is underneath the unit, and raised it the other side so its sort of level, the bottom cover is not in place as impossible to do so.

My plan is to buy longer screws so the base can be secured again but with a gap so the unit would be raised and this gap will be enough for cabling to get in and out the unit as well as enough room in the unit as inside the unit where the cables attach was also too tight to fit in its normal space.

In regards to the pci bracket the two ethernet ports can be detached but I am thinking of keeping them on the bracket and having the bracket sandwiched in the space thats created with using the longer screws, then it should be ok.  I will post some pics later.

So now the unit has 4 ethernet ports, 2x realtek and 2x intel.
Title: Re: LAN setup
Post by: Chrysalis on December 02, 2016, 04:34:35 PM
ok it uses 2mmx5mm screws, I ordered some 2mmx10mm and 2x20mm screws.

Will post some pics, it has a nice blue led light inside which looks really good when the unit is as designed fully enclosed, although it leaks around my room at night with the gap I created to add the intel ports.

Some photos of what is going on. Sorry the inside view of components is blurry didnt realise until now (took 2 days ago), the right blue thing is the msata ssd, to the left of it is the intel nic addon card, in the mpcie slot which was originally populated by the wireless nic card.

The idea is I will have the base screwed on again without any slant, but with longer screws so there is a gap to allow for the new cabling and bracket to fit in place.
Title: Re: LAN setup
Post by: Chrysalis on December 04, 2016, 12:10:39 AM
Looks like pfsense is lacking a lot of packages, 2 packages I will be making are nano and dnscrypt.

There seems a lot of opposition to dnscrypt for some reason in the pfsense forums which I find odd, but regardless I will convert freebsd packages to pfsense.
Title: Re: LAN setup
Post by: roseway on December 04, 2016, 07:18:08 AM
FreeBSD has a simple text editor called 'ee' which is quite similar to nano. You might find that meets your needs.
Title: Re: LAN setup
Post by: Chrysalis on December 04, 2016, 10:37:51 AM
I know about ee but nano is my preference.
Title: Re: LAN setup
Post by: underzone on December 04, 2016, 12:07:03 PM
If you can SSH or FTP into the box, use WinSCP. It is then easy to double click any conf/txt files and enter text and save as easily as notepad on a PC. No skill required, unlike vi...
Title: Re: LAN setup
Post by: Chrysalis on December 04, 2016, 02:05:12 PM
I know of alternatives but regardless the way I work is to use nano. :)
Title: Re: LAN setup
Post by: Chrysalis on December 06, 2016, 02:34:06 PM
So I realised the 2mm screws were wrong, actually should be 3mm, and was fiddling with the bracket positioning then suddenly it was possible to get the base in place with the gap raised slightly for the pcb of the pcb bracket to fit through, I did this and its now snug (will post pics later), of course aware that a possible short could occur and damage to the nic ports, it seems fine although one of the activity leds is broken but the port is working.
Title: Re: LAN setup
Post by: Chrysalis on December 06, 2016, 08:46:39 PM
As it turns out there is a repo for cli packages, but there is no list of what they got in it.

nano
bash
rsync are all there tho.

dnscrypt isnt but I installed the freebsd dnscrypt package which works out of the box.

Its unlikely I will be using my pfsense unit before christmas but I will likely do a online test tho.
Title: Re: LAN setup
Post by: Chrysalis on December 06, 2016, 08:58:03 PM
pics

note it wont be sitting in this location when in production use, is where it is now to be close to monitor.
Title: Re: LAN setup
Post by: Chrysalis on December 06, 2016, 08:58:22 PM
4th pic (3 pics per post limit)

Title: Re: LAN setup
Post by: phi2008 on December 07, 2016, 10:49:20 AM
pics

note it wont be sitting in this location when in production use, is where it is now to be close to monitor.

Talking of monitors, don't know if you're interested but you can buy a cheap 7" HDMI  1024x600 display for about 24 delivered - for those times when only direct access will do, or simply for monitoring things.

Link (https://www.aliexpress.com/item/7-inch-LCD-Panel-Digital-LCD-Screen-and-Drive-Board-HDMI-VGA-2AV-for-Raspberry-PI/32314647847.html?spm=2114.01010208.3.11.2J9EFz&ws_ab_test=searchweb0_0,searchweb201602_2_10065_10068_10084_10083_10080_10082_10081_10060_10061_10062_10056_10055_10054_10059_10099_10078_10079_426_10073_10103_10102_10096_10052_10050_425_10051,searchweb201603_6&btsid=09358aec-0a27-4158-b05f-86e0dfb8744a)

Can do a DIY case if necessary -

https://www.youtube.com/watch?v=UIu7eB18mps
Title: Re: LAN setup
Post by: Chrysalis on December 07, 2016, 01:27:26 PM
cool
Title: Re: LAN setup
Post by: Chrysalis on December 28, 2016, 03:29:41 PM
I am online now using pfsense to test it out.

Is a quick setup.

ipv4 only
using sky dns
no vlan stuff so for now my mdws will be down
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 09:27:39 AM
I have done some more work now

configured dhcp (before asus was still managing my dhcp)
configured nat
configured and enabled pfblockerNG (dnsBL) - this on the asus is what I was using to filter malware etc. but on pfsense it is officially supported in a plugin.
configured unbound partially
configured QoS partially

todo

dnscrypt
finish off unbound
ipv6
vpn
finalise QoS

To compile the dns block list and load it into unbound is now 2-3 seconds, on the asus it took over 2 minutes.

with dnsbl active my ram usage is still only 14% of 4 gig.

QoS is an interesting one, pfsense has a different approach it takes to QoS and I am trying its supported implementation first to see if its good enough for my needs.
Title: Re: LAN setup
Post by: Ronski on January 01, 2017, 10:11:17 AM
I'm also using PFBlockerNG, but I'm not using it to it's best. I currently have just the top 20 countries blocked (excluding UK) under GeoIP. I tried searching for an up to  date guide that explained it clearly but couldn't find one. I get people from around the world trying to hack into my remote access so I'd like to block most the world. It states "It's also not recommended to block the 'world', instead consider rules to 'Permit' traffic from selected Countries only." So if for example I selected just the UK, and changed the List Action to "Permit inbound" would that effectively block the rest of the world?

I'd be interested in knowing how you have yours set up in more detail.
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 10:20:03 AM
ronski I am using a private list and public, there is a couple of other public lists I may also add but this is what I did.

First of all an easy way to test is to use the easylist list as shown in first screenshot, you can add custom lists as well using the 'dnsbl feeds' option.

Third screenshot is my outbound rule I just setup to allow modem stat collecting on shared wan cable, however I could not find a way to add the right ip to the wan interface in the gui, so I had to do that in command line.

Note I am removing easylist later on my router, I added just to help test.  You have to be careful filtering ads/tracking router side as if a site breaks its not so easy to whitelist, also if want to whitelist sites like kitz for ad's that is also harder to do router side.
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 10:22:33 AM
example of how to add a hosts file list to dnsbl

using http://adaway.org/hosts.txt as example
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 10:46:43 AM
with the box more loaded up now temp is only 33C still :)

asus ac68 on passive cooling over 70C even in winter.

latency is lower than the ac68 also

Code: [Select]
C:\Windows\system32>ping -t bbc.co.uk

Pinging bbc.co.uk [212.58.244.23] with 32 bytes of data:
Reply from 212.58.244.23: bytes=32 time=6ms TTL=56
Reply from 212.58.244.23: bytes=32 time=7ms TTL=56
Reply from 212.58.244.23: bytes=32 time=6ms TTL=56
Reply from 212.58.244.23: bytes=32 time=7ms TTL=56
Reply from 212.58.244.23: bytes=32 time=7ms TTL=56

Ping statistics for 212.58.244.23:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 6ms, Maximum = 7ms, Average = 6ms
Title: Re: LAN setup
Post by: Ronski on January 01, 2017, 12:56:33 PM
Thanks for the info, not too fussed about blocking adverts to be honest. What's the Malware one do?

My box is running at about 36 degree's.
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 02:55:40 PM
It blocks domains that have been found to host malware.

I am on pfsense 2.4 now as I have been trying to get ipv6 working but is a no go for me.  Another buddy is trying to help me as he is also on sky but I am stuck for now.
Title: Re: LAN setup
Post by: skyeci on January 01, 2017, 03:18:15 PM
What do you need to check? My 2.4 works fine on sky ipv6 and was fine for 2.3 too.


Test with IPv4 DNS record       
ok (0.096s) using ipv4
Test with IPv6 DNS record       
ok (0.047s) using ipv6
Test with Dual Stack DNS record       
ok (0.661s) using ipv6
Test for Dual Stack DNS and large packet       
ok (1.532s) using ipv6
Test IPv4 without DNS       
ok (0.050s) using ipv4
Test IPv6 without DNS       
ok (0.056s) using ipv6
Test IPv6 large packet       
ok (1.580s) using ipv6
Test if your ISP's DNS server uses IPv6       
ok (2.519s) using ipv6
Find IPv4 Service Provider       
ok (0.473s) using ipv4 ASN 5607
Find IPv6 Service Provider       
ok (0.428s) using ipv6 ASN 560
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 03:22:48 PM
the wan DHCP6 settings, but my settings do match what the pfsense sky guy has posted on skyuser.

On the pfsense dashboard WAN_DHCP6 just stays stuck in pending status.

http://www.skyuser.co.uk/forum/ipv6/58986-sky-ipv6-settings-non-sky-routers-12.html#post463605

Title: Re: LAN setup
Post by: skyeci on January 01, 2017, 03:24:57 PM
Ok give me  a min will send you some screen dumps..
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 03:36:15 PM
do you mind also checking the contents of your /var/etc/dhcp6c_wan.conf file?

here is mine, which doesnt look right.  As it is only requesting dns servers and domain name but no prefix from what I can tell.

Code: [Select]
root@PFSENSE new # cat /var/etc/dhcp6c_wan.conf     
interface igb0 {
        request domain-name-servers;
        request domain-name;
        script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
Title: Re: LAN setup
Post by: skyeci on January 01, 2017, 03:36:31 PM
Done! You have mail... ;)

Did you add tracking on the lan interface on for dhcp6 etc.. have a look at my screen shots. Will have a look at the other bits when I get back.
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 03:55:11 PM
yep I was missing that tracking, thanks is now working. :)

Do you have the unreleased DUID patch on your pfsense yet?

also this looks more like it :)

Code: [Select]
interface igb0 {
        send ia-pd 0;   # request prefix delegation
        request domain-name-servers;
        request domain-name;
        script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc pd 0 {
        prefix-interface igb1 {
                sla-id 0;
                sla-len 8;
        };
};
Title: Re: LAN setup
Post by: skyeci on January 01, 2017, 04:29:23 PM
Nice!

No not yet. Part of the fix is already on the latest snapshot- the do not release request.. I understand the duid bit is now under testing so hopefully out soon. I am happy to wait for it to be released on a patch. I only  use it for the tbb ipv6 monitor so it's not really that much of an issue for my setup  :D

This bit is on the latest patch.

Do not allow PD/Address release
dhcp6c will send a release to the ISP on exit, some ISPs then release the allocated address or prefix. This option prevents that signal ever being sent
Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 04:38:20 PM
An update :)

The GUI method to add the ip for modem stats subnet is under firewall, virtual ip's, simply add the ip as a alias to the wan interface, which in my case is 192.168.2.252 with a /24 subnet mask.

This means so far I have managed to do everything without hacks in the shell.  The only shell modifying I have done is to edit the /etc/rc.initial script to add a 17) option for bash shell. 

On my asus my setup involved a lot of manual cli work.

Also I sent you an email to let you know how to preserve DUID before the patch is commited. :)

Also for both you and ronski, if you want to keep traffic total stats across reboots, do as this post says, I have already confirmed it works, even preserved over 2 pfsense upgrades (I went from 2.2 to 2.3 then to 2.4).

https://forum.pfsense.org/index.php?topic=114753.msg664804#msg664804
Title: Re: LAN setup
Post by: skyeci on January 01, 2017, 04:59:30 PM
Ok cheers.

I did a straight install recently to 2.4. I didn't like the idea of old files being left from previous versions. I also had a crash on my first build which was upgraded from a  working  2.3.. so far on 2.4 I haven't had a crash since doing the clean 2.4 build so hopefully that was perhaps related to the in place upgrades...probably never know but a good 2 weeks since that crash...

Title: Re: LAN setup
Post by: Chrysalis on January 01, 2017, 05:00:37 PM
Yeah I understand where you coming from, it is possible I will redo 2.4 clean at some point especially as I can use ZFS on 2.4 installer, but I have noticed no stability issues from the upgrade process so far.
Title: Re: LAN setup
Post by: Dray on January 01, 2017, 09:10:09 PM
Did you have to upgrade to FreeBSD v11 first?

I think I have to
Title: Re: LAN setup
Post by: underzone on January 01, 2017, 10:12:10 PM
with the box more loaded up now temp is only 33C still :)

asus ac68 on passive cooling over 70C even in winter.

latency is lower than the ac68 also

Code: [Select]
C:\Windows\system32>ping -t bbc.co.uk

Pinging bbc.co.uk [212.58.244.23] with 32 bytes of data:
Reply from 212.58.244.23: bytes=32 time=6ms TTL=56
Reply from 212.58.244.23: bytes=32 time=7ms TTL=56
Reply from 212.58.244.23: bytes=32 time=6ms TTL=56
Reply from 212.58.244.23: bytes=32 time=7ms TTL=56
Reply from 212.58.244.23: bytes=32 time=7ms TTL=56

Ping statistics for 212.58.244.23:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 6ms, Maximum = 7ms, Average = 6ms

Those pings are very impressive! Well done mate.
Title: Re: LAN setup
Post by: Chrysalis on January 02, 2017, 09:18:43 AM
Did you have to upgrade to FreeBSD v11 first?

I think I have to

The OS is upgraded as part of the upgrade process, its a all in one solution.

Now days FreeBSD supports binary updates so its still a very quick process assuming you can download the update packages quickly and the box itself is powerful enough to process all the package updates quickly, on my unit it took about 10 minutes, from 2.3 to 2.4 (which includes the OS update), and about 5 from 2.2 to 2.3.

The bad news with 2.4 I think they have removed hardware crypto support on their openvpn binaries, seems an odd decision, but I may have misunderstood the post, so I have asked for clarification.

The good news is 2.4 has added newer and better QoS queuing systems so basically QoS is enhanced.
Title: Re: LAN setup
Post by: Dray on January 02, 2017, 09:37:54 AM
Oh that's good news, I was concerned about the OS update. I still plan to wait for 2.4 to be the official update so I'm on 2.3.2-RELEASE-p1 (amd64) currently.

Thanks  :cool:
Title: Re: LAN setup
Post by: Ronski on January 02, 2017, 12:42:08 PM
It blocks domains that have been found to host malware.

What options are there for lists?

I see there is Malware Patrol (http://www.malwarepatrol.net), and they seem to offer various lists for a fee, both for PFBlockerNG & Squid, surely you'd only need to use PFBlockerNG (https://www.malwarepatrol.net/howto_pfBlockerNG.shtml) or Squid (https://www.malwarepatrol.net/howto_pfsense.shtml) not both for the purpose of blocking Malware domains?
Title: Re: LAN setup
Post by: Chrysalis on January 02, 2017, 01:27:13 PM
pfblockerNG also adds a lighttpd daemon so it works like this.

Dns lookup is changed to a rfc1918 ip.
Code: [Select]
C:\Windows\system32>nslookup 152media.com.
Server:  PFSENSE.home
Address:  2a02:c7f:<censored>

Name:    152media.com
Address:  10.10.10.1

Then the traffic will be sent to the lighttpd daemon running on pfsense and a blank img is served.

So no squid or other package needed.

You can also whitelist sites listed on alexa to try and avoid accidental breakage of popular sites from FP's.

For this to work, you need to be using the "dns resolver"(unbound) not "dns forwarder".
Title: Re: LAN setup
Post by: Chrysalis on January 02, 2017, 03:58:36 PM
I tested a reboot with the DUID workaround and kept the same ipv6 prefix.

I will be testing the FAIRQ+codel combo later.
Title: Re: LAN setup
Post by: Ronski on January 02, 2017, 05:33:23 PM
Thanks Chrysalis, but what options are there for lists to use? It's not exactly the easiest thing to Google for.

Edit.

Have found this post which seems quite useful

https://forum.pfsense.org/index.php?topic=102470.msg573159#msg573159

Title: Re: LAN setup
Post by: Chrysalis on January 02, 2017, 11:27:22 PM
you already found a good lot of choices there :)

for purely malware lists that are freely available there isnt many.
Title: Re: LAN setup
Post by: Chrysalis on January 03, 2017, 04:11:01 PM
ronski I would say

malcode
mdl
dshield_sd
mpatrol - with alexa
ms2
bbc_dga
bbc_c2

Those all look malware focused including crypto malware, having a quick look at all those lists they dont seem to include any ads/tracking stuff.

just be aware its easier to add bad url's then remove cleaned out ones so it wouldnt surprise me if there is dead domains or domains that are now clean in the lists, but pfsense does allow whitelisting and alexa will whitelist the top ranked sites.

hphosts
swc

those two for sure do include ad/tracking domains.
Title: Re: LAN setup
Post by: Chrysalis on January 03, 2017, 04:24:28 PM
I think sticky DUID feature is now commited.

https://github.com/pfsense/pfsense/commit/c337280901d3eedf98e195bd99d30d2ed9d4df1e
https://github.com/pfsense/pfsense/commit/a5d56253d87abeeb76ef480edced9a7a512ad908
Title: Re: LAN setup
Post by: Chrysalis on January 03, 2017, 05:51:11 PM
I added these which the ac68 would explode in a fit with such a size.

Code: [Select]
===[ DNSBL Domain/IP Counts ] ===================================

 1209183 total
  809831 /var/db/pfblockerng/dnsbl/bambenek_dga.txt
  162237 /var/db/pfblockerng/dnsbl/hphost_fsa.txt
  142657 /var/db/pfblockerng/dnsbl/hphosts_emd.txt
   43810 /var/db/pfblockerng/dnsbl/malwarepatrol.txt
   26329 /var/db/pfblockerng/dnsbl/hphost_psh.txt
   17267 /var/db/pfblockerng/dnsbl/hphosts_exp.txt
    5280 /var/db/pfblockerng/dnsbl/disconnectmalvertising.txt
    1207 /var/db/pfblockerng/dnsbl/malwaredomainlist.txt
     345 /var/db/pfblockerng/dnsbl/zeustracker.txt
     100 /var/db/pfblockerng/dnsbl/malc0de.txt
      73 /var/db/pfblockerng/dnsbl/hphost_hjk.txt
      43 /var/db/pfblockerng/dnsbl/dshield_sdh.txt
       4 /var/db/pfblockerng/dnsbl/bambenek_c2.txt
       0 /var/db/pfblockerng/dnsbl/malwaredomains.txt
       0 /var/db/pfblockerng/dnsbl/disconnect_malware.txt
       0 /var/db/pfblockerng/dnsbl/disconnect_basic.txt

the biggest ones are the cryptolocker domains, device is now using 25% of ram.
Title: Re: LAN setup
Post by: Ronski on January 03, 2017, 07:30:27 PM
Thanks Chrysalis for the further info which is very useful. I did manage to get the Danguardian feed working last night, got tripped up with a few things, didn't realise it wouldn't work until the CRON job ran, also took ages to find/work out what a force update was. I'll detail more in my thread and add other feeds when I've got some time.

Incidentally how often should the CRON job run for pfBlockerNG?

One other thing, I understand this blocks the domain name, but would it block the actual IP address if used directly say by malware?
Title: Re: LAN setup
Post by: Chrysalis on January 03, 2017, 08:47:45 PM
A safe setting is once a day, you dont want to upset list maintainers by swamping thewir servers for updates.

By the way a quick update.

1 - I removed the DGA list which has 800k entries, I did some digging and found these are not verified live domains, they are generated domains from seed's that ransomware admin's were found to be used for domain generation, so its a sort of catch all list that is designed to preempt new domains coming on line and been unfiltered, but my unbound started having some issues, however I may readd it later due to what I found in issue #3 see below.
2 - I added a IP BL for some ransomware servers, this has to be added in the ipv4 section as its not DNS filtered, but firewall filtered. url for list is here, site says is updated every 5 mins but I at least for now set it to once a day. https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt
3 - I discovered that unbound is restarted very frequently if it is set to host DHCP name records, basically if the static DHCP or leased DHCP boxes are ticked in dns resolver settings, pfsense seems to be coded badly in that whenever a DHCP record is updated it will restart unbound which flushes its cache and reloads dns lists, and the 800k list I had added caused unbound to be unresponsive for 30+ seconds when this happened.

Now regarding #3, on my particular network, I can see in the resolver logs, unbound was been restarted every 10-20 minutes which is way too frequent for my liking, so I copied someone else's suggestion which is to manually maintain a DHCP dns list which I load into unbound using the custom config box with an include line and keep those 2 boxes unticked, I am only maintaining for my static DHCP leases, I dont care about dns resolution on dynamic leases.

Just reread your post.

My cron is set to every 12 hours, but I have all this lists set to only update once a day.  Setting the cron to run more often shouldnt be a big deal however it may (if you add lists at different start times) stagger updates which would mean more dns reloading.

Also on the DNSBL config page near the bottom is this section

"DNSBL IP Firewall Rule Settings"

I think but I am only guessing as I have not tried it that if you enable "List Action" setting and select deny, then it may add the resolved ip's to the firewall.

Just be aware that many domains can be hosted on a single ip, so lets say a ransomware dude is hosting his domain on a shared web hosting server mixed in with legal customers sharing the same ip, you could also block all those sites.
Title: Re: LAN setup
Post by: Chrysalis on January 04, 2017, 01:06:41 PM
Was using my phone in bed earlier and there was a internet outage, when I checked pfsense later I found it had a kernel panic and self rebooted, so I seem to have a hardware issue somewhere or a configuration issue.

I think the 2 most likely culprits are the ram and addon intel card (is reported issues in pfsense with intel addon cards), I have disabled msix on the intel ports and also reverted a tunable I played with yesterday and will see if stays stable, if I get another panic I will test the ram.
Title: Re: LAN setup
Post by: Chrysalis on January 04, 2017, 05:32:58 PM
using dnscrypt now, if anyone else using pfsense wants a guide to get it working I will print one.
Title: Re: LAN setup
Post by: Dray on January 04, 2017, 09:36:53 PM
That's good news. I would appreciate a guide  ;)
Title: Re: LAN setup
Post by: Chrysalis on January 04, 2017, 10:06:17 PM
you on pfsense 2.2/2.3 or 2.4?
Title: Re: LAN setup
Post by: Dray on January 04, 2017, 10:10:12 PM
Currently on  2.3.2-RELEASE-p1 (amd64)
Title: Re: LAN setup
Post by: Chrysalis on January 05, 2017, 10:43:27 AM
ok its cli setup 100% as there is no official pfsense package.  This relies on the FreeBSD package.

change to a working directory e.g. /root
Code: [Select]
cd /rootdownload the FreeBSD 10 dnscrypt-proxy package
Code: [Select]
fetch http://pkg.freebsd.org/FreeBSD:10:amd64/latest/All/dnscrypt-proxy-1.8.1.txzinstall the package
Code: [Select]
pkg install dnscrypt-proxy-1.8.1.txztest if there is no runtime errors, by running the binary with no arguments, should just see generic output telling you that syntax is needed
Code: [Select]
dnscrypt-proxynow the next bit is dependent on your own config, there is various dnscrypt guides around the web, We assume you will be using opendns dnscrypt servers. I cannot paste mine as its using my private dns server.
So run this command which will use the built in database to connect to opendns (cisco)
Code: [Select]
dnscrypt-proxy --ephemeral-keys --local-address=127.0.0.1:65053 --daemonize -R ciscoyou should see a warning that they do logging and also that opendns has no dnssec, but no other output aside from those 2 lines, you can verify if its running with this.
Code: [Select]
ps ax | grep dnsand look for this
Code: [Select]
dnscrypt-proxy --ephemeral-keys --local-address=127.0.0.1:65053 --daemonize -R ciscoif its running then you want it to startup auto on boot so the following 2 commands.
Code: [Select]
sysrc dnscrypt_proxy_enable=YES
sysrc dnscrypt_proxy_flags='--ephemeral-keys --local-address=127.0.0.1:65053 --daemonize -R cisco'

Now it is done but isnt actually been used.

You have created a encrypted tunnel for dns between your router and opendns, but you still need to tell the router to use that tunnel, and in this case to use the tunnel you need to forward dns queries to 127.0.0.1 port 65053

I dont think pfsense supports custom ports in its GUI so in the dns resolver settings scroll down to where you see a box for custom options, and add this

Code: [Select]
forward-zone:
        name: "."
        forward-addr: 127.0.0.1@65053

now unbound will forward all internet queries to the tunnel after you save and apply the settings.

That is finally done.

Notes

If you ever update pfsense to 2.4, the binary will stop working, you will need to uninstall the package, and then install the FreeBSD 11 package.
Pfsense wont manage the package meaning if you want to keep up with new versions of dnscrypt-proxy you need to keep an eye on the FreeBSD repo for updates.  An easy way is checking on freshports.org.
Title: Re: LAN setup
Post by: Dray on January 05, 2017, 11:10:45 AM
Thanks Chrys I'll give it a go.

Incidentally I saw some of your posts on the pfSense forums (there are so many forums there), that fellow who was droning on about dnscrypt being unnecessary clearly lives in the USA and has no clue what life is like in other parts of the world. I doubt he's travelled outside his home town.
Title: Re: LAN setup
Post by: roseway on January 05, 2017, 11:19:51 AM
One small comment: it's not necessary to fetch the package before installing it. Just type (as root)

Code: [Select]
pkg install dnscrypt-proxy
which will fetch and install the latest version of dnscrypt-proxy in the repositories.
Title: Re: LAN setup
Post by: Chrysalis on January 05, 2017, 11:32:21 AM
no it wont eric as it doesnt exist on the pfsense repos.

pkg install without a local package in the syntax will only install packages on the pfsense repositories.

Code: [Select]
root@PFSENSE pfblockerng # pkg install dnscrypt-proxy
Updating pfSense-core repository catalogue...
Fetching meta.txz: 100%    940 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%    2 KiB   1.7kB/s    00:01   
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.txz: 100%    940 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%  121 KiB 123.7kB/s    00:01   
Processing entries: 100%
pfSense repository update completed. 444 packages processed.
pkg: No packages available to install matching 'dnscrypt-proxy' have been found in the repositories
Title: Re: LAN setup
Post by: roseway on January 05, 2017, 11:54:51 AM
Fair enough, I failed to see your comment that it isn't in the pfSense repos.
Title: Re: LAN setup
Post by: Chrysalis on January 05, 2017, 11:57:18 AM
No worries.  Obviously if it was in the repos that would be the way to install it, but when I requested it to be added, as dray mentioned I had hostile responses telling me there is no point in the package.

I also think adding the FreeBSD repo in the pfsense config (meaning pkg install would search the FreeBSD repo) is a bad idea as that could cause issues for other packages, this method I feel is the safest way.

If one wanted to remove one step of the dns processing, then they could disable the dns resolver and bind the proxy to port 53, however you then lose dns caching as well as other benefits of the unbound resolver such as been able to filter with pfblockerNG.
Title: Re: LAN setup
Post by: Chrysalis on January 05, 2017, 12:09:11 PM
Incidentally how often should the CRON job run for pfBlockerNG?


I have made my cron more frequent now at every 2 hours, the reason is at midnight for some reason 3 of the feeds couldnt download, and so it auto retries to the next cron, but with my cron been every 12 hours it meant 12 hours to wait for a new attempt.

So the update frequency in the feed settings controls how often updates will happen for the feed, but the cron setting is how often it will 'check' if updates are needed, so a cron every 2 hours will not do updates every 2 hours if the feed is set to once a day.

The issue I have with frequent cron runs is if the dnsbl feeds get staggered at different intervals since each dnsbl feed update reload's unbound.  So I will keep an eye on it and if I find they get staggered then will revert to infrequent crons again.
Title: Re: LAN setup
Post by: Dray on January 06, 2017, 07:12:50 AM
ok its cli setup 100% as there is no official pfsense package.  This relies on the FreeBSD package.

change to a working directory e.g. /root
Code: [Select]
cd /rootdownload the FreeBSD 10 dnscrypt-proxy package
Code: [Select]
fetch http://pkg.freebsd.org/FreeBSD:10:amd64/latest/All/dnscrypt-proxy-1.8.1.txzinstall the package
Code: [Select]
pkg install dnscrypt-proxy-1.8.1.txztest if there is no runtime errors, by running the binary with no arguments, should just see generic output telling you that syntax is needed
Code: [Select]
dnscrypt-proxy

I have a problem at this point, this command just returns "Command not found" so I tried
Quote
/usr/local/sbin/dnscrypt-proxy
which gives the error "Undefined symbol crypto_core_hchacha20"

I think crypto_core_hchacha20 is part of libsodium, but I'm stuck now.
Title: Re: LAN setup
Post by: Chrysalis on January 06, 2017, 12:10:19 PM
yeah you need to check if these files exist

Code: [Select]
root@PFSENSE inc # ldd /usr/local/sbin/dnscrypt-proxy
/usr/local/sbin/dnscrypt-proxy:
        libltdl.so.7 => /usr/local/lib/libltdl.so.7 (0x80123a000)
        libsodium.so.18 => /usr/local/lib/libsodium.so.18 (0x801443000)
        libkvm.so.7 => /lib/libkvm.so.7 (0x8016b3000)
        libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x8018c1000)
        libm.so.5 => /lib/libm.so.5 (0x801ac4000)
        libc.so.7 => /lib/libc.so.7 (0x800823000)
        libthr.so.3 => /lib/libthr.so.3 (0x801cef000)
        libelf.so.2 => /lib/libelf.so.2 (0x801f16000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80212a000)

run ldd on your own system tho as since I have FreeBSD 11 my /lib ones will be different.

Code: [Select]
root@PFSENSE inc # pkg which /usr/local/lib/libsodium.so.18
/usr/local/lib/libsodium.so.18 was installed by package libsodium-1.0.11_1

I think you right, I must have had it already installed as a dependency of another pfsense package.

So
Code: [Select]
pkg install libsodium should fix it.
Title: Re: LAN setup
Post by: Dray on January 06, 2017, 01:39:32 PM
yeah you need to check if these files exist

Code: [Select]
root@PFSENSE inc # ldd /usr/local/sbin/dnscrypt-proxy
/usr/local/sbin/dnscrypt-proxy:
        libltdl.so.7 => /usr/local/lib/libltdl.so.7 (0x80123a000)
        libsodium.so.18 => /usr/local/lib/libsodium.so.18 (0x801443000)
        libkvm.so.7 => /lib/libkvm.so.7 (0x8016b3000)
        libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x8018c1000)
        libm.so.5 => /lib/libm.so.5 (0x801ac4000)
        libc.so.7 => /lib/libc.so.7 (0x800823000)
        libthr.so.3 => /lib/libthr.so.3 (0x801cef000)
        libelf.so.2 => /lib/libelf.so.2 (0x801f16000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80212a000)

run ldd on your own system tho as since I have FreeBSD 11 my /lib ones will be different.
I ran ldd but not all of the files you listed were there
Quote
So
Code: [Select]
pkg install libsodium should fix it.
I tried that and it installed from the pfsense repository but it didn't work so I had a look on pkg.freebsd.org/FreeBSD:10:amd64/latest/All/ and I saw libsodium-1.0.11_1.txz there which i fetched and installed and now /usr/local/sbin/dnscrypt-proxy runs  :)
Title: Re: LAN setup
Post by: Chrysalis on January 06, 2017, 02:40:40 PM
ok but I find that odd as I never had to do that and mine did work when I was on pfsense 2.2.  But it works for you now which is good :) but just hope that dependency been installed from FreeBSD doesnt break any pfsense packages that have the same dependency.
Title: Re: LAN setup
Post by: Dray on January 06, 2017, 04:34:04 PM
I dont think pfsense supports custom ports in its GUI so in the dns resolver settings scroll down to where you see a box for custom options, and add this

Code: [Select]
forward-zone:
        name: "."
        forward-addr: 127.0.0.1@65053

now unbound will forward all internet queries to the tunnel after you save and apply the settings.
Unfortunately this causes my DNS to stop working :(

Is there any debugging I can do?
Title: Re: LAN setup
Post by: Chrysalis on January 06, 2017, 09:54:18 PM
The commands I gave to set on bootup sadly wont be valid for pfsense.

You need to install the shellcmd addon package in the GUI.

system -> package manager -> available packages - select shellcmd

When its done goto services and select shellcmd

set shellcmd type to shellcmd and the command in left box and description in right box.

Regarding your broken dns lookups, you can check your live unbound.conf with the command 'cat /var/unbound/unbound.conf'  Stick the output on pastebin and if you want to keep it confidental pm me the link.
Title: Re: LAN setup
Post by: Chrysalis on January 07, 2017, 01:27:37 AM
As I thought would happen the mini pcie Jetway card I got for intel ports is no longer sold.

this is what I put in the unit.

https://webcache.googleusercontent.com/search?q=cache:gy2pXys8PpMJ:https://linitx.com/product/13534+&cd=6&hl=en&ct=clnk&gl=uk
Title: Re: LAN setup
Post by: Dray on January 07, 2017, 06:32:54 AM
Unfortunately this causes my DNS to stop working :(
It seems I have to add another line to this to make it work, like this
Quote
do-not-query-localhost: no

forward-zone:
 name: "."
 forward-addr: 127.0.0.1@65053
I now have another problem in that OpenDNS servers don't support DNSSEC, so I have to turn that off
Title: Re: LAN setup
Post by: Dray on January 07, 2017, 06:56:00 AM
I've added Shellcmd with the following:

sysrc dnscrypt_proxy_enable=YES    shellcmd    dnscrypt start on bootup    
   
sysrc dnscrypt_proxy_flags='--ephemeral-keys --local-address=127.0.0.1:65053 --daemonize -R cisco'    shellcmd    set dnscrypt parameters on bootup    
   
Title: Re: LAN setup
Post by: Dray on January 07, 2017, 11:58:07 AM
It seems it's possible to tell if dnscrypt is working by doing
Quote
nslookup -querytype=txt debug.opendns.com
The answer includes the line
Quote
debug.opendns.com  text = "dnscrypt enabled (01234567890123456789)"
Title: Re: LAN setup
Post by: Chrysalis on January 07, 2017, 12:13:17 PM
the command to add in shellcmd is this

Code: [Select]
/usr/local/sbin/dnscrypt-proxy --ephemeral-keys --local-address=127.0.0.1:65053 --daemonize -R cisco
Title: Re: LAN setup
Post by: Dray on January 07, 2017, 04:37:09 PM
Thanks  :)
Title: Re: LAN setup
Post by: Chrysalis on January 08, 2017, 10:50:30 PM
As good as pfSense is their (core) developers seem to have attitude, I been wasting my time writing up detailed bug reports as they just seem intent on hitting the reject button to keep the bug count down.  So I will probably stop posting on the pfSense forum soon as well and just keep my discussion on here.
Title: Re: LAN setup
Post by: Chrysalis on January 09, 2017, 06:26:10 PM
An update regarding my hardware.

When I was debugging some ALTQ stuff I was testing with the 2 realtek ports, at which time I discovered one of the ports is not working, someone else with the same unit has reported the same exact issue as well, so be wary if ordering the unit I purchased and are not adding extra ports.
Title: Re: LAN setup
Post by: Chrysalis on January 09, 2017, 10:12:09 PM
this is with FAIRQ+codel

and a custom rule I added to prioritise all ack's.  I think this one was with a 93.5% cap of max upload bandwidth.

http://www.dslreports.com/speedtest/8577920

This one was set to 97%.

http://www.dslreports.com/speedtest/8574510
Title: Re: LAN setup
Post by: Chrysalis on January 10, 2017, 08:01:39 PM
pfblockerng has a very nice feature which can be used for traffic classifying.

see this data

Code: [Select]
Alias Count Packets Updated
pfB_Dhshield24 40 1 Jan 10 18:15   (2)
pfB_Dshield 101 0 Jan 10 18:15   (2)
pfB_Emerging 1731 0 Jan 10 18:15   (2)
pfB_MalwareExploits 135 0 Jan 10 18:15   (2)
pfB_Netflix 36 0 Jan 10 18:15   (1)
pfB_RansomCryptoware 11548 0 Jan 10 18:15   (2)
pfB_Steam 64 0 Jan 10 18:15   (1)
pfB_bbc 41 0 Jan 10 18:15   (1)
pfB_blizzard 44 0 Jan 10 18:15   (1)
pfB_google 6759 5 Jan 10 18:15   (1)
DNSBL_MalwareExploits 213791 5 Jan 09 19:45:33
DNSBL_PrivacyFraud 191072 2 Jan 09 19:46:15
DNSBL_Cryptolocker 830095 0 Jan 08 23:49:34

Aside from malware lists you may notice I have entries like netflix, google,steam and bbc.

What I have done is entered the ASN information into pfblockerng and it can grab the ip ranges those companies manage into a firewall table.  I then used that information to add rules to send traffic based on those tables to the QoS queue I want.

For steam this is working perfect.  It also works on google services if I disable ipv6.  Netflix, iplayer are not that easy tho as the bbc is outsourcing to limelight networks and not using their own ip space for iplayer, netflix is coming over from local sky nodes.  Also when ipv6 is enabled google and netflix come over it and the ASN feature is ipv4 only.  But still I thought this was worth mentioning as it is a powerful tool.

I do have the traffic shaping working quite nicely now albeit after I had to configure it differently than how pfSense documention suggests and I caused a ruckus on the pfSense forum when trying to explain the issues I had with the default behaviour.

--edit--

ipv6 ASN's are supported, google now working good over this system as well :)
Title: Re: LAN setup
Post by: Chrysalis on January 17, 2017, 02:01:24 AM
Since steam downloads from port 80 I could not lower its priority in conventional ways, this is classified via ASN match.

Title: Re: LAN setup
Post by: Chrysalis on January 25, 2017, 02:49:03 PM
I am planning to install 2.4 fresh using ZFS so I can ditch UFS.  Will post here when I do the work.
Title: Re: LAN setup
Post by: Chrysalis on January 26, 2017, 08:48:36 AM
Ok I have now successfully migrated to zfs since I am not confident of using ufs for reliability, there is some gotchas that occured which I will mention here.

Please read everything before you consider doing this as is some gotchas.

So the process I did was as follows.

1 - Run the backup wizard under diagnostics menu to make a backup, make sure everything is included in the backup, a box is ticked by default which doesnt backup traffic usage data, I unticked it.
2 - If you have any custom files anywhere on the filesystem then back them up as they will be lost.  Also if you want to preserve any logs back them up also.
3 - Download the pfsense 2.4 installer, which is on the development download section, pfsense 2.4 is required if you want native support for 4k alignment, trim and zfs.
4 - Put the installer files on your install media, in my case is a usb stick and I used rufus usb tool.
5 - Reboot the pfsense unit and boot of the install media.
6 - Choose zfs (guided) install
7 - select striped and pick your storage device
8 - enable forced 4k alignment option.
9 - GPT enable if you have UEFI bios or disable if you dont.
10 - proceed with install and let it finish and reboot
11 - after reboot access the web UI with default admin/pfsense login details.
12 - when the wizard appears ignore it and instead access the diagnostics menu and choose backup and restore.
13 - restore your backup and making sure the option to also restore packages is enabled (was enabled by default on my unit).
14 - Watch the console as it restores your packages to see if any issues.
15 - When completed the core pfsense system and all official plugins should be restored.

Gotchas

The first issue I had is after restoring my backup and it rebooted, it could not get internet access to redownload and install the packages, the reason for this is unbound was not working for 2 reasons (I initially thought was just one reason).  This was because in my case dnscrypt was missing and as such I had no working dns tunnel, and also that unbound didnt start due to missing pfblockerng files causing a syntax error.  I noticed on the console it was retrying every minute or so so I simply edited /etc/resolv.conf as follows to make the router use google dns temporarily.

Code: [Select]
nameserver 8.8.8.8
The restore packages process then successfully finished.
I then had to pkg install dnscrypt again, but that was all I had to do for that as the earlyshell cmd configuration was intact so on a reboot it started properly.
Unbound however was down still because it was trying to load pfblockerng dnsbl files that were missing, so again I had to temporarily enable google dns on the router manually, and then in the pfblockerng gui I manually ran the cron process, which downloaded all the lists and created its configuration files, after that unbound runs normally.

People who dont use pfblockerng dnsbl lists and dont use dnscrypt wouldnt have this problem.

I restored my custom files like custom loader.conf and services.inc (to fix unbound restarts) and all seemed well.

One final gotcha.

The traffic totals plugin is missing at this point, it was not restored when the restore wizard ran, I also forgot to backup its data when I backed up my custom files.  So this plugin has to be reinstalled manually post restore.

Also I enabled a zfs feature that makes extra copies of every stored file, the command to do so is this.

Code: [Select]
zfs set copies=3 zroot
Bear in mind tho this will not make copies of existing files unless they get rewritten, only new writes have the automated copies, it is also nowhere near as good as a zfs mirror setup but is better than a plain single copy on a single drive setup.  the reason to make 3 copies instead of 2 is so when zfs detects corruption in a copy it can determine the correct copy by a majority rules system where by where 2 copies match, they will be determined to be the correct copy.
Title: Re: LAN setup
Post by: Chrysalis on January 26, 2017, 09:15:45 AM
I decided to check the raw smart data of the ssd, and considering I have used this pfsense unit for about a month I am surprised at how much has been written.

175 gig of data written on the ssd.

In comparison my desktop ssd, which hosts games and the OS, and I have used for over 2 years has 8.34tb of writes.

Bear in mind tho the ssd in my pfsense unit is only 60gig vs 512gig of my desktop ssd and its planar nand vs 3d nand in my pc.  So the endurance of the ssd in the pfsense unit is way less than whats in my pc.  both are MLC nand.

The largest dnsbl feed with 800k domains the file when downloaded is circa 100mbyte in size, this file is then converted to another format to be compatible with unbound, the converted file is 44mbytes in size.  So on every update is about 150mbyte of writes from that feed, the updates are once a day but I have probably done a few dozen manual updates on top of that when testing stuff.  Now I have zfs compression in play tho will continue monitoring and see if the rate of writes slows down (with 3 copies set it may well actually jump up), there is also the possibility this drive was not shipped unused to me (a return) as I didnt check before using it if the stats were all in a new state.
Title: Re: LAN setup
Post by: Chrysalis on January 26, 2017, 03:00:38 PM
Been doing some more testing with the traffic shaper, whilst I find FAIRQ+codel superior for upstream traffic, it doesnt seem to be perfect for downstream and I am now currently using hfsc+codel which has showed better results in some quick tests using steam, dslreports and tbb speedtesting but will need time to see how it goes with days of internet use.

Code: [Select]
pfTop: Up Queue 1-14/14, View: queue, Cache: 10000                                                                            14:55:55

QUEUE                             BW SCH  PRIO     PKTS    BYTES   DROP_P   DROP_B QLEN BORROW SUSPEN     P/S     B/S
qInternet                        19M fair             0        0        0        0    0                     0       0
qACK                               0 fair    6   307812 19973104        0        0    0                     0       0
qDefault                           0 fair    4   248278  159647K        0        0    0                     3     207
qOthersHigh                        0 fair    5    13414  1717003        0        0    0                     0       0
qOthersLow                         0 fair    3   143551 82363014        0        0    0                     0       0
qICMP                              0 fair    7     1728   113444        0        0    0                     2     221
qBulk                              0 fair    2      649   256446        0        0    0                     0       0
root_igb1                        67M hfsc    0        0        0        0        0    0                     0       0
 qDefault                      3357K hfsc        220764  249884K        0        0    0                     2     357
 qICMP                         1342K hfsc           428    31672        0        0    0                     0       0
 qACK                          3357K hfsc         67140  3883408        0        0    0                     0       0
 qOthersHigh                     33M hfsc         13584  1638300        0        0    0                     0       0
 qOthersLow                      13M hfsc        193416  284731K       35    52990    0                     0       0
 qBulk                         6714K hfsc        179147  263734K     3894  5824621    0                     0       0

Steam for me remains the ultimate test, as that floods the line with dozens of tcp sessions (seems they designed it to get round poor isp's) and under normal circumstances it will cause packet loss on small packets like ssh packets and pings.  Steam testing has been more positive using hsfc than fairq and priq.
Title: Re: LAN setup
Post by: Chrysalis on January 26, 2017, 05:34:38 PM
found this gem, add /status.php to end of the ip url so e.g. https://192.168.1.252/status.php and you get a very nice informational page :)
Title: Re: LAN setup
Post by: adrianw on January 27, 2017, 12:29:35 AM
found this gem, add /status.php to end of the ip url so e.g. https://192.168.1.252/status.php and you get a very nice informational page :)
Thanks!
I can see that being very useful at times.
Title: Re: LAN setup
Post by: Chrysalis on January 30, 2017, 05:52:46 PM
this looks pretty sweet for anyone considering pfsense.

http://www.asrock.com/ipc/overview.asp?Model=NAS-9601

Newer gen version of my CPU and the unit has 6 intel ports, and it has VGA out so no serial console stuff needed.
Title: Re: LAN setup
Post by: Ronski on January 30, 2017, 07:28:05 PM
Very nice, but seems it's not readily available.
Title: Re: LAN setup
Post by: Chunkers on January 30, 2017, 11:17:51 PM
this looks pretty sweet for anyone considering pfsense.

http://www.asrock.com/ipc/overview.asp?Model=NAS-9601

Newer gen version of my CPU and the unit has 6 intel ports, and it has VGA out so no serial console stuff needed.

That IS nice but
Very nice, but seems it's not readily available.
AKA does anyone sell it?

OTOH, bet its going to be expensive, plus I have an Asrock BeeBox N3000 (http://www.asrock.com/microsite/beebox/) and its a pile of unreliable crap.

Chunks
Title: Re: LAN setup
Post by: burakkucat on January 30, 2017, 11:29:15 PM
this looks pretty sweet for anyone considering pfsense.

http://www.asrock.com/ipc/overview.asp?Model=NAS-9601

Newer gen version of my CPU and the unit has 6 intel ports, and it has VGA out so no serial console stuff needed.

I looked at the on-site images and could not see a VGA port . . . So I downloaded the data sheet. That mentions VGA under the "Graphics" heading but the section under the "Rear I/O" heading states "VGA 0"!

All a bit of a mystery.  :-\
Title: Re: LAN setup
Post by: Chrysalis on January 31, 2017, 10:46:18 AM
I have reduced zfs copies back to the default 1 on /tmp and /var and left at 3 for the rest of the system.

With this it is averaging 5 gig writes a day to the ssd, which I am pretty sure is mostly due to dnsbl feed processing. If we assume my ssd was at 0 bytes writes when installed, then I believe this to be a similar rate of writes as ufs.  This is because the ssd is a sandforce ssd which has native compression, on a normal ssd which doesnt compress I think zfs will reduce writes.

On pfSense /var is mostly storage for logs and dynamically generated files (including dnsbl files), so any corruption to these files should not have any persistent effects hence me changing it back, likewise /tmp is only housing short lived files.

I also changed globally logbias on zfs from latency to throughput which is more friendly to ssd's.

I am going to examine the pfblockerng code to see where it writes its files during the update process and see if some of the work can be moved to a ramdisk, I dont want to enable the built in pfsense ramdisk as it is by default as that makes alot of stuff non persistent such as logs and processed dnsbl files, meaning on a reboot logs are lost and pfblockerng has to download a new set of files due to the old ones been on volatile storage.  What I am aiming to do is have the original downloads and any in between processing files on ram storage and then only the completed dnsbl files on persistent storage.

/tmp also houses the pf rule set and a config.cache file which are frequently rewritten but both are only small files.

I also have been experimenting with different power states for the cpu.

Since I had another panic last week, I disabled some power saving functions in the bios which can cause instability, with the most important one I believe disabling DVFS on the ram (so ram stays at stock clocks and voltage throughout).  My current config now also has powerd set to not let the cpu drop below its stock speeds, but still allow it to ramp upto turbo speeds (so things like dnsbl updates are faster), this also removes low voltage states from the cpu which can cause instability.  This config adds probably about 3-5C to my average cpu temps.

So the experiment has been to let 2 cpu cores enter C2 state when idle (very low performance cost) and the other 2 cores to C3 (moderate performance cost when going from idle to load).  This actually did not achieve anything significant so I will be reverting all to C2, or even leaving at the default C1.

Here is some data, note core 0 seems to be always in a wake state doing something so is unaffected mostly by any changes.

cores 0,1 set to C2 cores 2,3 set to C3

Code: [Select]
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.3.cx_usage_counters: 1608486 118501 844080
dev.cpu.3.cx_usage: 62.56% 4.60% 32.82% last 33390us
dev.cpu.3.cx_lowest: C3
dev.cpu.3.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.2.cx_usage_counters: 287976 14077 383181
dev.cpu.2.cx_usage: 42.02% 2.05% 55.91% last 16161us
dev.cpu.2.cx_lowest: C3
dev.cpu.2.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.1.cx_usage_counters: 1925350 1301238 0
dev.cpu.1.cx_usage: 59.67% 40.32% 0.00% last 43us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 39609431 170 0
dev.cpu.0.cx_usage: 99.99% 0.00% 0.00% last 102us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1000

This shows core 0 99.99% of time is in C1 state.
Core 1 about 40% of time it manages to get in C2 state.
Cores 2,3 similar to core 1 except they drop to C3 instead of C2.

The temperatures for this config? well here it is.

Code: [Select]
dev.cpu.3.temperature: 38.0C
dev.cpu.2.temperature: 38.0C
dev.cpu.1.temperature: 37.0C
dev.cpu.0.temperature: 37.0C

Pretty much no benefit from either C2 or C3, C3 actually seems to make things worse as there is some work involved for the cpu to move between states and C3 takes much longer than C2 to move in and out of.

Here is data for this morning after it been idle and also no heating on.

Code: [Select]
root@PFSENSE tmp # sysctl dev.cpu |grep cx
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.3.cx_usage_counters: 2572229 196691 2064691
dev.cpu.3.cx_usage: 53.21% 4.06% 42.71% last 16775us
dev.cpu.3.cx_lowest: C3
dev.cpu.3.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.2.cx_usage_counters: 800620 38919 1055540
dev.cpu.2.cx_usage: 42.24% 2.05% 55.69% last 58977us
dev.cpu.2.cx_lowest: C3
dev.cpu.2.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.1.cx_usage_counters: 3184420 3416255 0
dev.cpu.1.cx_usage: 48.24% 51.75% 0.00% last 702us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 112912517 423 0
dev.cpu.0.cx_usage: 99.99% 0.00% 0.00% last 252us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1000
root@PFSENSE tmp # sysctl dev.cpu |grep temperature
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 35.0C
dev.cpu.0.temperature: 35.0C

Everytime i check the pattern is fairly reliable that cores 2 and 3 have higher temps than core 1, core 0 can get higher sometimes.  The power savings from these mode in terms of raw watts are also very low, as the cpu itself is only rated at 6 watts, so doesnt use much power to begin with.

Quick update, C2 is actually providing a meaningful benefit, I first dropped cores 2,3 to C2, and this was the result. A 1C drop to match core 1.

Code: [Select]
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 36.0C
dev.cpu.0.temperature: 36.0C

Then watch what happens when I lock core 1 to C1 only.

Code: [Select]
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 39.0C
dev.cpu.0.temperature: 36.0C

and one minute later

Code: [Select]
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 40.0C
dev.cpu.0.temperature: 36.0C

However core 0 , even tho spends 99.99% of time in C1 doesnt have temps that high which is interesting, likewise if lock core 0 to C1 it has no affect as its 99.99% of time in that state anyway.
Title: Re: LAN setup
Post by: Chrysalis on January 31, 2017, 11:00:48 PM
Found the cause of the writes, its not pfblockerng, its the rrd graphs generated by pfsense.

in /var/db/rrd is the graphing databases and they are updated every minute, on my system they are 7.2meg in size in total.  Multiply that by 60 minutes and thats 480meg of writes every hour, or 11.5gig every 24 hours. (when uncompressed).

I took the idea from the scripts deployed to backup the traffic stats, so did the following.

1 - temporary disabled graphing.
2 - created a backup script which will run at a set interval, and ran it to create an initial backup.
3 - wiped /var/db/rrd but left the directory in place so can mount on it.
4 - created a small ramdisk and mounted to the location which in my case I chose 200meg which should easily be enough.
5 - created a restore script and ran it to restore the files.
6 - reenabled graphing again.

I will next set a cron entry to run the backup script at intervals, probably once an hour or maybe every 15 minutes.
Also will be the script added to shellcmd to generate ram disk and restore the backup at boot.
Title: Re: LAN setup
Post by: Chrysalis on February 16, 2017, 04:18:19 PM
Ok the SSD I put in my pfsense unit as it turns out has a warranty write limit higher than my much more expensive samsung 512 pro.

The kingston msata ssd 60gig has a 218TB warranty coverage.
Title: Re: LAN setup
Post by: Chrysalis on April 19, 2017, 03:36:03 PM
since i had to take unit apart for it been stuck in hibernation mode after a power cut i took a new pic to show how tight a fit to add the intel i350 nic ports

https://drive.google.com/file/d/0B7P3Ne0hzKcGd3FrWGxDdUhDbkk/view?usp=drivesdk
Title: Re: LAN setup
Post by: Chrysalis on April 19, 2017, 03:50:44 PM
I looked at the on-site images and could not see a VGA port . . . So I downloaded the data sheet. That mentions VGA under the "Graphics" heading but the section under the "Rear I/O" heading states "VGA 0"!

All a bit of a mystery.  :-\
yeah it has a vga header on board but no port on case so needs a attachment
Title: Re: LAN setup
Post by: nallar on April 20, 2017, 01:00:04 PM
You're running 2.4, you could try fq_codel instead of fairq+codel. Should be better.

https://forum.pfsense.org/index.php?topic=126637.0
Title: Re: LAN setup
Post by: Chrysalis on April 20, 2017, 02:24:23 PM
on todo list :)
Title: Re: LAN setup
Post by: Chrysalis on September 24, 2017, 12:15:51 PM
look like kingston have withdrawn the models of their msata from the market below the 120 gig size, the 120 gig hasnt fell down to the price range they were at so basically you can no longer get a new kingston msata in the 30 price range.  All I can find now in that price range for a msata ssd brand new is chinese brands like kingfast.
Title: Re: LAN setup
Post by: camieabz on December 09, 2017, 06:07:38 AM
Unrelated to the OP / thread:

Many of the posters have their kit setup in their sigs. This is getting parsed for Google searches, and as a result, when I searched for "Billion" "8800" and other search-related data, this thread popped up, but is not relevant at all.

Might I suggest that sigs not include kit, or that the site prevents parsing of sigs?
Title: Re: LAN setup
Post by: Dray on December 09, 2017, 06:51:41 AM
Funnily enough the same thing happened to me when I googled squiggle
Title: Re: LAN setup
Post by: Chrysalis on December 10, 2017, 04:05:24 PM
camieabz this thread may have come up, but then you would find the 8800nl in my sig is clickable and points to one of my 8800nl threads.

Also i have pfsense in my sig which would then have this thread come up for pfsense searches as well.

Swings and roundabouts really.
Title: Re: LAN setup
Post by: Chrysalis on June 18, 2018, 11:55:37 AM
dnscrypt-proxy v1 is now considered obsolete, and is been removed from FreeBSD repo's in due course, so I have v2 a replacement client developed by another developer running, so far so good and will post a guide later, there is also an official guide for it as well here.

https://github.com/jedisct1/dnscrypt-proxy/wiki/Installation-pfsense

I will post my own guide also, largely similar but added notes in relation to my tuning and using the amd64 binaries (official guide uses ARM binaries), and getting it to work with your own personal dnscrypt server.

Also a comparison with the older proxy software here.

https://github.com/jedisct1/dnscrypt-proxy/wiki/Differences-to-v1

The original dnscrypt-wrapper is still been developed and works with this v2 proxy.  The v2 proxy also works with unbound DNS servers using crypto v2 or DNS over http/2.