I have reduced zfs copies back to the default 1 on /tmp and /var and left at 3 for the rest of the system.
With this it is averaging 5 gig writes a day to the ssd, which I am pretty sure is mostly due to dnsbl feed processing. If we assume my ssd was at 0 bytes writes when installed, then I believe this to be a similar rate of writes as ufs. This is because the ssd is a sandforce ssd which has native compression, on a normal ssd which doesnt compress I think zfs will reduce writes.
On pfSense /var is mostly storage for logs and dynamically generated files (including dnsbl files), so any corruption to these files should not have any persistent effects hence me changing it back, likewise /tmp is only housing short lived files.
I also changed globally logbias on zfs from latency to throughput which is more friendly to ssd's.
I am going to examine the pfblockerng code to see where it writes its files during the update process and see if some of the work can be moved to a ramdisk, I dont want to enable the built in pfsense ramdisk as it is by default as that makes alot of stuff non persistent such as logs and processed dnsbl files, meaning on a reboot logs are lost and pfblockerng has to download a new set of files due to the old ones been on volatile storage. What I am aiming to do is have the original downloads and any in between processing files on ram storage and then only the completed dnsbl files on persistent storage.
/tmp also houses the pf rule set and a config.cache file which are frequently rewritten but both are only small files.
I also have been experimenting with different power states for the cpu.
Since I had another panic last week, I disabled some power saving functions in the bios which can cause instability, with the most important one I believe disabling DVFS on the ram (so ram stays at stock clocks and voltage throughout). My current config now also has powerd set to not let the cpu drop below its stock speeds, but still allow it to ramp upto turbo speeds (so things like dnsbl updates are faster), this also removes low voltage states from the cpu which can cause instability. This config adds probably about 3-5C to my average cpu temps.
So the experiment has been to let 2 cpu cores enter C2 state when idle (very low performance cost) and the other 2 cores to C3 (moderate performance cost when going from idle to load). This actually did not achieve anything significant so I will be reverting all to C2, or even leaving at the default C1.
Here is some data, note core 0 seems to be always in a wake state doing something so is unaffected mostly by any changes.
cores 0,1 set to C2 cores 2,3 set to C3
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.3.cx_usage_counters: 1608486 118501 844080
dev.cpu.3.cx_usage: 62.56% 4.60% 32.82% last 33390us
dev.cpu.3.cx_lowest: C3
dev.cpu.3.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.2.cx_usage_counters: 287976 14077 383181
dev.cpu.2.cx_usage: 42.02% 2.05% 55.91% last 16161us
dev.cpu.2.cx_lowest: C3
dev.cpu.2.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.1.cx_usage_counters: 1925350 1301238 0
dev.cpu.1.cx_usage: 59.67% 40.32% 0.00% last 43us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 39609431 170 0
dev.cpu.0.cx_usage: 99.99% 0.00% 0.00% last 102us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1000
This shows core 0 99.99% of time is in C1 state.
Core 1 about 40% of time it manages to get in C2 state.
Cores 2,3 similar to core 1 except they drop to C3 instead of C2.
The temperatures for this config? well here it is.
dev.cpu.3.temperature: 38.0C
dev.cpu.2.temperature: 38.0C
dev.cpu.1.temperature: 37.0C
dev.cpu.0.temperature: 37.0C
Pretty much no benefit from either C2 or C3, C3 actually seems to make things worse as there is some work involved for the cpu to move between states and C3 takes much longer than C2 to move in and out of.
Here is data for this morning after it been idle and also no heating on.
root@PFSENSE tmp # sysctl dev.cpu |grep cx
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.3.cx_usage_counters: 2572229 196691 2064691
dev.cpu.3.cx_usage: 53.21% 4.06% 42.71% last 16775us
dev.cpu.3.cx_lowest: C3
dev.cpu.3.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.2.cx_usage_counters: 800620 38919 1055540
dev.cpu.2.cx_usage: 42.24% 2.05% 55.69% last 58977us
dev.cpu.2.cx_lowest: C3
dev.cpu.2.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.1.cx_usage_counters: 3184420 3416255 0
dev.cpu.1.cx_usage: 48.24% 51.75% 0.00% last 702us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1 C2/2/500 C3/3/1000
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 112912517 423 0
dev.cpu.0.cx_usage: 99.99% 0.00% 0.00% last 252us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1000
root@PFSENSE tmp # sysctl dev.cpu |grep temperature
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 35.0C
dev.cpu.0.temperature: 35.0C
Everytime i check the pattern is fairly reliable that cores 2 and 3 have higher temps than core 1, core 0 can get higher sometimes. The power savings from these mode in terms of raw watts are also very low, as the cpu itself is only rated at 6 watts, so doesnt use much power to begin with.
Quick update, C2 is actually providing a meaningful benefit, I first dropped cores 2,3 to C2, and this was the result. A 1C drop to match core 1.
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 36.0C
dev.cpu.0.temperature: 36.0C
Then watch what happens when I lock core 1 to C1 only.
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 39.0C
dev.cpu.0.temperature: 36.0C
and one minute later
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 36.0C
dev.cpu.1.temperature: 40.0C
dev.cpu.0.temperature: 36.0C
However core 0 , even tho spends 99.99% of time in C1 doesnt have temps that high which is interesting, likewise if lock core 0 to C1 it has no affect as its 99.99% of time in that state anyway.