Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Alex Atkin UK]

Yesterday my N100 suddenly rebooted then failed to start up again due to seemingly ZFS corruption.  Kinda ironic given I never had a failure using UFS and supposedly ZFS is meant to be more reliable.

I have no clue how to recover this without reinstalling, plus no idea if its a bad hardware (that will teach me buying an appliance with SSD & RAM already supplied from China), or something else.  I've never had pfSense die on me before, though admittedly this machine has rebooted itself a few times prior which I assumed was a problem with my custom scripts.

The kicker is I thought I had backed up the config.xml a few weeks back, but apparently I hadn't as the last backup is 7 months old.  Though upon booting my backup N5105 it looks like I only lost some DHCP allocation changes, maybe some changes to my policy routing, but I'd like to recover the configuration if I can.

[Ragnarok]

This week, My Config on my OpenWRT router got corrupted by my mother somehow managing to turn off the router without a clean shutdown.  Pressing the power button normally shuts it down cleanly, I have no Idea how she held it down. Even Power cuts haven't managed this in the past!!

It was weird the internet connectivity was broken partially, but not completely, but enough to break the voip service. I just couldn't find the issue with it. apparently the issue was so bad it couldn't wait a few more minutes till i got home.

Make regular backups especially if you change anything is the moral of the story!!!

Thankfully i had a working config from a few months back when I was back on BT before getting moved to Trooli fiber. I got back up and running on the latest open WRT quite quickly.

[Chrysalis]

Alex I ended up not using the ram and ssd they supplied, the test windows install I had on prior to moving to pfSense, had corruption with very little usage.  (the corruption was bad enough that DSIM couldnt even run its check command).

I then couldnt pin point via SMART, cluster and ram testing the culprit, so just replaced both the ram and NVME that was supplied and its been fine since.  My second N100 was ordered bare bones.

Also I do have active cooling on it, without it the NVME was pegged at 70C its cap/throttle temp.  I have 2 tiny fans which are quiet, and then slowed them down even more with a tiny voltage regulator device.  They are on top of the unit, might post a pic later.  So my NVME now runs about 45C in late spring when I added them and about 50C in the current weather, since the ram is very close to the NVME that was also likely running really hot before as well.  The N100 case design is its weakness, its quite small, and doesnt have the cooling at the sides like my older NUC.

FreeBSD doesnt seem to activate idle NVME power saving, which is a contributing factor to the temps, but there is a command which I ran, that can throttle the active power, and it knocks a few Celsius of the NVME drive, it has to be ran every boot up, so I have it as a automated boot command.

Have you tried sticking the NVME in another machine? boot up live FreeBSD, and then can can load up the ZFS pool to access the data.

[Chrysalis]

Did some digging, there is some hope for you.

So mount the drive in a system can live boot FreeBSD (could even just keep it in the N100 for this, and boot of FreeBSD via USB).

'zpool import nameofpool' for getting the pool loaded.
Then mount the dataset you want with the zfs flag '-o ro' this is read only, and there is people on TrueNAS forums who have had success this way.

You can check the NVME temp with the smartctl command, if you curious how hot its getting in the unit.

[Alex Atkin UK]

Thanks, I actually booted Fedora 39 and was going to try from there but couldn't remember the pool name (its gonna be pfsense isn't it).

A lot of people report temp problems with N5105 and N100 passive appliances, but I found mounting them vertical on the wall so that convection can do its job optimally makes a huge difference.  What I do think is dumb is my TOPTON unit has no vents to help the NVME drive, my N5105 does.  But the vertical orientation seemed to prevent the drive getting hot.

I'm not convinced the problem was heat as I found a thread on the Proxmox forums of people with this unit claiming early N100 microcode is buggy and causes crashes.  It seems Intel just can't get the microcode of their CPUs right lately.

[Chrysalis]

If you not sure of the pool name the '- a' flag might catch it as that will import all pools.

If you dont mind sharing that thread I will have a read, I think it might be a bit quick to consider microcode is the problem before swapping parts but I am interested. Also If I recall correctly newer microcode is loaded during boot by default.

My pool is named 'pfSense'.

The dataset which holds the root files is probably 'pfSense/ROOT/default' with the config in 'pfSense/ROOT/default/cf'

[Alex Atkin UK]

Question is, how often do pfSense update the microcode?  Sounds like its tied to the FreeBSD release used.

I've lost all faith in Intel at this point given https://community.intel.com/t5/Processors/Microcode-0x129-Update-for-Intel-Core-13th-and-14th-Gen-Desktop/m-p/1622436 has not dropped for Linux yet and I'm anxiously awaiting it, seeing as I have a 14700k in the server.  The whole darn point was to not have to touch it for the next 5 years as its such a pain to work on.

At least it will eventually thankfully as the MSI board in my server is on the first BIOS to support 14th gen, as you can't update the ME firmware from Linux,  which it claims is required for newer BIOS versions.  Still pondering if I should swap it with an ASUS TUF Gaming as they merge the ME update into the main BIOS update.  Is it even important to update if the microcode gets loaded via Linux anyway?

[Chrysalis]

If you are convinced your chip is unstable, then I dont know what to say as an opinion.  In my case I replaced the chinese branded NVME and ram, cooled the unit and it has been fine since.

I couldnt find a proxmox thread that matched what you described but did find some where people fixed their systems after replacing the ram.

Intel have said the 0x129 fix only affects CPUs with a TDP over 65w, the N100 is nowhere near that, even then it only affects "some" CPUs one's that have a voltage request over 1.55v, and those kind of requests are only made for chips at clock speeds over 5ghz using p-cores.  The N100 is a e-core only chip that doesnt clock anywhere near that high.

My 13700k as an example doesnt go over 1.344v so I wont even be bothering with the bios as my chip wont be affected.  If you have a very poorly binned 14700k, that might need the microcode, but the N100 is not affected at all by that issue.  The N100 also isnt affected by the eTVB bug, as that only affects i9 gaming chips.  Also N100 is Alder Lake not Raptor Lake, that should give you piece of mind as Alder Lake chips are not affected at all by the recent Intel problems, not even the top end i9 chips.

The microcode update is a FreeBSD package which means it is not tied to OS updates, however on pfSense a lot of packages only get updated when they update the OS.  I just had a look, the latest portable microcode update I can find for the N100 is dated November 2023, Revision 26 fixes erratas adl074 adl075 and adl076.  Typically fixes like this are extremely rare conditions, you have likely been running systems for years not up to date on microcode, I attached the bug details for that update.

This is the version of the package on my system.

devcpu-data-20230617_1

Looks like from June 2023, so it isnt a "really old" microcode.  But not the November microcode.

The bios fixes for high end Raptor Lake were released yesterday, so I think it will appear very soon for your motherboard on your 14700k rig. 0x129 wont drop for linux as Intel have announced its a motherboard only update, so you will have to update the bios for your board to get the microcode.  The benefit of that is it will be loaded no matter what OS you use.

On further investigation pfSense has the latest version of the devcpu-data package, however that package is now replaced with a newer package, which they may be unware off, so right after this edit I will be submitting a request for them to switch to the new package, as they might not be aware of it.  Details here.

https://www.freshports.org/sysutils/devcpu-data/

[Alex Atkin UK]

My 13700k as an example doesnt go over 1.344v so I wont even be bothering with the bios as my chip wont be affected.

I think you misunderstand the problem, its the transients that can go above 1.55v which may not show up in software monitoring but will slowly (or quickly depending on how well binned the CPU is) kill the ringbus.

It varies dramatically by workload, so can be hard to catch even if it does show up in software.
I guess the microcode is not releasing to Linux because it requires motherboard vendors to make other changes to adhere to it (or not):
https://www.youtube.com/watch?v=TOvJAHhQKZg

I guess I may stick to my current solution, aggressively limit the maximum clock as it seems to be the only way to keep my CPU within the limit of the low-profile cooler anyway.  Limiting the current seems to do nothing to prevent it clocking aggressively.

I know this is unrelated to the N100, its just showing a pattern of Intel releasing bad microcode.

[Chrysalis]

Got a reply from pfSense dev's, the new package is already switched to the new microcode package in snapshots. So should be used in the next release onwards.

[Alex Atkin UK]

Does it load microcode automatically on boot like I believe Linux does?

When I was searching I found a lot of older information where pfSense refused a request to include microcode at all and like a lot of the official pfSense documentation, what is on their site is clear as mud.

[Chrysalis]

Yes loaded on bootup.

[Chunkers]

Not sure whether our units are in similar packages but my N305 is also passive and seems to behave pretty well even during warm periods with the indicated temps not moving much, admittedly mine runs on a very low load and I chose the enclosure with the larger heatsinks, unless running very high load I suspect not heat related

I bought mine barebones and sourced branded memory and SSD to try avoid any issues

[Alex Atkin UK]

Nope, mine has really tiny fins (only a couple of mm) which TOPTON seems to have revised in later models to more like your N305.

The gamble of buying off Aliexpress, but then they do cost about 1/3 of what you'd pay from a UK supplier so it would still work out cheaper if I have to replace it with a newer model.

[Chrysalis]

Mine has tiny fins, chunkers unit does look perhaps more suited to the job.

For sure I will always buy barebones moving forward.