It seems that my router, the Firebrick FB2900 has died. RIP. The whole LAN went crazy this morning, internet access was lost and I wasted a lot of time blaming WAPs, partly because of insane nonsensical error behaviour from iOS / iPadOS. It turned out that all the madness was due to the failure of DHCP. When the penny finally dropped it was because I went to AA’s control panel website Clueless which showed that both DSL lines supposedly went down at 7:30 on Friday morning. The principal way that Clueless determines up/down line state is from the non-reply to PPP LCP echo requests (PPP LCP ‘pings’). It is the Firebrick that replies to these LCP pings not the modems and so when the Fb2900 died, it made it look as if the lines were down as far as AA could see. Well, they weren’t effectively working, so a reasonable state report. Then I asked Janet to take a look at the Firebrick, no LEDs showing at all but the unit was still quite warm which says to me that the (internal) PSU was not completely dead.
I emailed AA about it, but no reply from a human; not surprising as there would have been no one around probably, given that by then it was Friday mid afternoon and every one would be heading home.
Luckily I have my FB2500 in the emergency equipment box, and a Solwise 4G wireless router, for total disaster situations. Getting the FB2500 set up properly was a real fiddle, which was my own fault as I have neglected it for five years and have no installed o/s updates not pushed new config versions into it. The old o/s was not compatible syntactically with the current modern-day config XML. The modern config has some XML attributes that the old o/s has not heard of, so I had to delete various bits to get it to load the XML config. I should have updated the o/s first but I needed to get the device working 100% on the internet so I wanted the Firebrick’s XML config to be good enough for the modern day state-of-affairs first. It did work ok-is even with the 2017 config so perhaps I should have upgraded the o/s first, benefit of hindsight when not in a kerfuffle state. Upgrading the o/s wasn’t simple. First I upgraded the boot loader, just to be a conscientious citizen. Then upgraded the o/s, which failed and I didn’t notice the fact - so UI could have been better surrounding the upgrade procedure. The reason it failed, so I discovered was that the o/s was so old that you can’t upgrade all the way in one leap - you have to upgrade in a number of intermediate steps. I think this is what are called the system of ‘breakpoints’ in the releases website. Again, the upgrade procedure UI could be better on the Firebrick. When it tells you to upgrade to v.vvv.vvv then it should be more greatly emphasised and it should explicitly say loudly that that the version number shown is not the one you desired, but an intermediate one and explain that. It took several intermediate upgrades to get there, but I managed it in the end.
After this it should be possible to upload the normal config. Because an FB2500 is not feature-compatible with an FB2900/FB2700 because the FB2500 lacks the 3G USB NIC ‘dongle’ feature, any mention of USB causes the config-load to fail. I wrote an automatic FB2900/FB2700-to-FB2500 converter program, first in awk, then in iOS Shortcuts, which was a pain as it’s incredibly slow. This gets run automatically if you try to use my standard uploader tool to upload an FB2900/FB2700 config file into an FB2500 router.
So, a very fraught day, with three hours of panicking and debugging and then wrestling to bring the neglected FB2500 up to modern standards. Note to self: every six months or so, upgrade o/s and config of backup routers in storage.
I see that AA will rent out Firebricks now. This might be quite attractive if it means that you can upgrade models. Will have to ask. I suppose there ought to be some way though. I will be always wanting to have two Bricks in stock. Since the Firebrick supports VRRP, I should really be exploiting that I feel. There are a couple of concerns about this:
1. I have been concerned about the scenario where a lightning strike takes out two Bricks not just one. I have lost a Firebrick to a lightning strike before, and that time AA just gave me a free replacement, which I thought was amazing. I now have a small 8-way ZyXEL switch between the modems and the Firebrick which now is really only there as lightning protection for the Firebrick. So perhaps that extra protection is enough to protect two Firebricks in VRRP. I presume that you would be mad to try two dissimilar models in VRRP pairing?
2. Janet asked the excellent question (she is starting to think like a sysadmin, so I tell her in praising her question) of how does the sysadmin know if a router has died if VRRP is hiding the state ? I have no idea, I said to her. I’m sure one of our number will know. There could be a Firebrick feature meant to address this, or some standard feature associated with VRRP, or an add-on standards-based monitoring solution might be required.
