Kitz Forum

Broadband Related => Broadband Hardware => Topic started by: Weaver on April 22, 2019, 04:19:46 AM

Title: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on April 22, 2019, 04:19:46 AM
I have a Huawei USB 3G NIC fitted in my Firebrick FB2700 router for failover-backup. Supplied by A&A.

It disappeared at around 10:00 UTC on Sunday morning. Now there is no USB device shown in the ‘dongle status’ info.

This has happened several times before but it could easily take a couple of months wait to reproduce it. On earlier occasions, I rebooted the Firebrick and then the USB NIC was shown and was working as normal.

It would seem that I unwittingly wiped out valuable information when I rebooted the brick so I won’t do that this time until someone has had a chance to look at it.

This is clearly a bug, but I’m not sure exactly where. If there is something naughty going on in the USB NIC then there needs to be a workaround in the Firebrick to handle the failure and reinitialise it. Because after all, the Firebrick knows or can know that there is a failure. If it thinks that the hardware has been physically removed then it can try reinitialising it anyway, just on the off chance.

Has anyone got any ideas?
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: burakkucat on April 22, 2019, 03:20:11 PM
I have a Huawei USB 3G NIC fitted in my Firebrick FB2700 router for failover-backup. Supplied by A&A.

It disappeared at around 10:00 UTC on Sunday morning. Now there is no USB device shown in the ‘dongle status’ info.

This has happened several times before but it could easily take a couple of months wait to reproduce it. On earlier occasions, I rebooted the Firebrick and then the USB NIC was shown and was working as normal.

It would seem that I unwittingly wiped out valuable information when I rebooted the brick so I won’t do that this time until someone has had a chance to look at it.

I would suggest that the best persons to take a look would be either Firebrick support (https://www.firebrick.co.uk/about/contact-us/) or A&A support (https://support.aa.net.uk/AAISP). Remote access would, of course, be required.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on April 22, 2019, 03:43:59 PM
Agreed. Have emailed A&A.

Wondered if anyone else had seen this.

Am I right in calling this a bug? Even if something bad happened on the network or there was some bug in the Huawei device, is it fair to say that the FB2700 should ‘just handle it’? - doing a workaround even, as necessary.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: burakkucat on April 22, 2019, 04:13:06 PM
I would hope that the Firebrick could handle the "mishap", in one way or another.  :-\
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: andrew-AAISP on April 22, 2019, 05:07:26 PM
Are you on s/w version 1.51.010?

There was a fix in that which may help.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on April 22, 2019, 05:31:40 PM
Afraid I already was on that release?
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Chrysalis on April 23, 2019, 02:31:08 PM
yeah I hope you and aaisp can figure it out as I think you are the only one on here weaver that has used firebrick.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on April 24, 2019, 02:15:34 AM
@Chrysalis I seem to recall that at one time we had other Firebrick users in our number, perhaps I’m imagining it though. Burrakucat has an earlier model Firebrick somewhere around, maybe not right next to his cat-bed.

I suspect A&A staff have been run off their feet what with needing to catch-up after the long break in England and I’m far far from critical, but haven’t heard back from a human yet. Which is fine, give them time, let them get on with the important stuff and people who are actually down, and then let them get their breath back.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Hibbard on April 24, 2019, 08:49:43 AM
The USB code has always been rather unreliable as the device eventually becomes unresponsive only pulling and re-inserting the USB kicks it back into life.

I gave up on it personally.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: vic0239 on April 24, 2019, 08:52:07 AM
^^^ Ditto.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: burakkucat on April 24, 2019, 03:32:24 PM
@Chrysalis I seem to recall that at one time we had other Firebrick users in our number, perhaps I’m imagining it though.

There are other users of Firebrick devices . . . I'm just having a problem in remembering exactly who. Ah, yes, Ixel has a FB2900 (https://forum.kitz.co.uk/index.php/topic,21359.msg369919.html), the current device and, from their responses, I presume both Hibbard & vic0239 are also Firebrick owners.

Quote
Burrakucat has an earlier model Firebrick somewhere around, maybe not right next to his cat-bed.

Yes, correct. I have a second generation FB105, stored in the grotto, for special use cases. (Such as performing a cat-in-the-middle interception of data flowing through an Ethernet link.)

My understanding --
The first to fourth generation Firebricks are now termed legacy devices.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on April 24, 2019, 04:23:41 PM
Is it something internal to the USB device going wrong?

Whether it is or not, the Brick does know that badness has happened, because the device always disappears from the usb hardware tree displayed in ‘dongle status’. So, thank goodness, the FB2700 has in fact detected the problem but just doesn’t have a sufficient quantity of clue to make it take some useful action. And, internal or not, the workaround would be to reinitialise the software module itself and reinitialise the brick hardware.

AA staff are looking into it for me anyway.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on August 08, 2019, 03:24:09 AM
This has happened once again now. Don’t know how long ago. The start of it is more than a week back as suggested by the long blank history in AA’s clueless.as.net.uk CQM server which is monitoring link cwcc@a.0 (representing the USB NIC on 3G/4G). So unfortunately, due to my negligence and lack of oversight and watchfulness I did not pick up on the failure in a timely fashion. At least this time round though I have thus far managed to restrain the urge to fix the problem by resetting the Firebrick, which revives the 3G NIC but would destroy diagnostic information.

I emailed AA on Sunday but no response as yet. Am hoping they will log into the Firebrick directly with the usual account that was set up at the dawn of time for AA support staff’s usage. With any luck there will be some diagnostic info still in there?
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on August 09, 2019, 02:19:03 AM
* Problem solved, I strongly suspect.

My stupidity truly knows no bounds and I owe AA a huge ‘fruity bun’ for wasting their time.

AA staff pointed out that they were seeing a ‘usb hub’ and asked me about it. I said ‘no’, I just have a straight usb cable from the Firebrick to the USB ‘dongle’ 3G NIC, but I had grown suspicious. I asked Mrs Weaver to fetch me the said cable. I had asked mrs weaver whether she had plugged in a usb hub and she said no, but in fact I had never seen the cable that Mrs Weaver had used. Looking at it the far end looks way too large and at perhaps 5m in length it has to be too long for the passive usb limit, no? So perhaps it is an active usb repeater. Maybe such things look like usb hubs to the upstream hardware. It only has one socket on the far end of it, so that’s why Mrs Weaver understandably didn’t consider it to be a usb hub.

Anyway, surely we have found the answer now, I suspect that either the amazon cable in question is rubbish or else it draws too much current from the Firebrick and so occasionally the voltage droops. Or there is a dI/dt problem, also causing a voltage droop. Perhaps needs a capacitor in a strategic place. Perhaps the whole setup would be ok if we were using an externally powered usb hub.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on August 19, 2019, 06:20:52 AM
There was a catch.

Without the cable, the 3G link is very unreliable, link down-up down- more down than up. So perhaps this is all about signal strength and despite my confidence in the suitability of the USB NIC’s placement in the room when plugged directly into the Firebrick with respect to the window and line of sight to the basestation maybe I was wrong and it does need to be right in the window. I can’t move the Firebrick with dongle in it very easily, it would be a fair bit of disruption haveing tinget mains and a couple of long ethernet cables just for an experiment. So I thing it would be easier to move the dongle again on its own but either with a different active cable, or a much shorter passive extension cable to a halfway position that has to be fairly good in placement terms.

Perhaps what I should really do is get an externally powered usb hub and plug the dongle into that. Can things be set up with a hub so that power to the dongle is supplied or supplemented by an external dc source rather than drawing possibly too much power from the Firebrick?

If I can get such a USB hub if very high quality, perhaps I could combine it with the original active cable or a shorter passive cable and test both. The active cable thing would be yet another thing to go wrong, but maybe it would not misbehave with the hub, since it wouldn’t have to supply power - major over-optimism. Anyway it would be worth a test. And a test with hub plus shorter passive cable might improve placement enough so as to be able to make a signal strength improvement that is significant, if such is there to be had even.

Could anyone recommend any known reliable high quality usb hardware for this? Known excellent active cables or even better externally dc-powered hub if such exists ? Stuff that you have tested yourself and know well?
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on August 22, 2019, 12:28:07 AM
Got an email back from AA with some excellent news. Developers have been able to reproduce the problem described as a ‘lock up’ in the NIC; I’m told the NIC becomes unresponsive when a certain condition develops but no details yet as it’s early days. But now that it can be reproduced the devs will track it down and be able to address it.

I’m going to go back to the long cable for now.
Title: Re: Firebrick FB2700 - disappearing USB ‘dongle’
Post by: Weaver on August 13, 2022, 02:33:54 PM
Several years on, and this fault is now a rare occurrence, but today I have once again noticed the disappearing 3G dongle. Last thursday at 23:35 BST there was a clueless.aa.net.uk log entry saying "LNS planned switchover". Not surprisingly, the clueless.aa.net.uk CQM graph shows no PPP link after that, but either the PPP connection is not established or somehow CQM graphing isn’t happening for another reason (CQM graph shows purple background). When I noticed the problem today, I commanded the Firebrick to cycle the power supplied to the USB interface in order to reinitialise the 3G NIC, and then all was well.

The LNS thing is perplexing: why should that have anything to do with the 3G NIC getting into a bad state? Something happened on or after Thursday that was genuinely bad, because it required a 3G-NIC-related fix action, although nuking the USB i/f was perhaps overly drastic.

I have played with the f8lure.mouselike.org service in the past. I should be able to get this to ping-monitor the 3G USB NIC and alert me via email when the 3G NIC goes down? Is that correct ? What do I need to check ?
I got stuck with this procedure though, because pinging from an external source address failed; presumably either due to some firewalling somewhere, but I can’t see it, or a routing choice, given that this is a failover interface and only supposed to be active during failover. So will have to check the routing.


Checking routing settings in clueless for that single IPv4 address, all looks correct. So will need to ask AA as I’m stuck.