Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Weaver]

[Context: see http://forum.kitz.co.uk/index.php/topic,17364.msg317505.html#msg317505]

I have just taken delivery of a new router. I had some problems setting it up, as documented in an earlier thread, (see http://forum.kitz.co.uk/index.php/topic,17779.msg323470.html#msg323470), but Andrews and Arnold, my ISP, sorted the device out and brought it back from the dead for me, by return of post. So it came back all happy and pre-programmed with my own current config in it taken from my Firebrick FB2500.

The reason I bought it was in an attempt to cure the "dripping blood" problem, loss of PPP LCP ping echoes or their responses, sent by Andrews and Arnold's CQM system (see - http://aaisp.net/kb-broadband-cqm.html) for link quality testing. See also -
    http://forum.kitz.co.uk/index.php/topic,17364.msg317505.html#msg317505

After a lot of testing and messing about with swapping kit and hours and hours talking to AA tech support, I decided that the only thing I could think of was to swap out the router, on the theory that port 1 on it - which went to one of the three modems attached to it - was bad.

Well here's the bad news. After spending £900 on a brand new router, the problem is still there, quite unchanged. Brain failure, total. I don't know what to do next. Just go over everything again, systematically I suppose.

Theory: perhaps if I swap ports 1 and 4, port 4 being connected to the LAN switch, it will avoid the problem, because then port 1 won't be going to a modem any more. But if this is right, then this would mean that all Firebricks have this port-1 related problem, so why are other users not seeing it, and why did it suddenly appear one Thursday evening at the end of March? Speculation: could have been something triggered by a software update, but there would have to be some secondary condition involved too, as there was no software update that day, yet there had been one a short(-ish) while before. Another point, I think the default config for the Firebrick is to have the LAN on port 1, so perhaps that's how most people use it. More speculation: perhaps there are very few people who use LAN=port_4 and also have multiple modems. Anyway, putting the blame on the Firebrick is far from certain.

I need to go over everything again, swapping modems out - I have yet another new modem. I also need to swap out cables just in case. What a nightmare.

[PhilipD]

Hi

Have you tried setting up another ping monitor, like the one from Thinkbroadband to see if that has the same problem?  If it doesn't, you know it's not your line or your equipment.

What if you ping out, set up repeating pings and see what packet loss you get on the way out. Not sure what OS you are using but from Windows then

Code: [Select]

ping bbc.co.uk -t will send pings until you stop it.  I can run that without any loss of packets/pings over the course of many minutes, if you can, it would again suggest there is no problem with your equipment or line and the pings are being blocked elsewhere.

Regards

Phil

[Weaver]

Thanks for that PhilipD, the "pings" in question are PPP LCP echo-requests, not the usual ICMP pings, so not IP at all, a PPP-only thing.

I can certainly ICMP-ping various targets, my router itself can ICMP-ping things, and I have various iOS apps that can do it, as well as a number of external server that can ICMP-ping in the inbound direction. But that's a different issue. What the loss of PPP LCP frames _means_ is unclear. I don't know whether it is part of a wider problem, because TCP can cover up the issue, and after all there are two other modems so that doesn't help with debugging either. I can of cause disconnect the other two modems and see what the TCP throughput is like then.

[Weaver]

Explanation of PPP LCP, see -
    https://technet.microsoft.com/en-us/library/cc957992.aspx
and the links at the end of
    https://en.m.wikipedia.org/wiki/Link_Control_Protocol

[Bowdon]

I'm not sure if this is related or not. But I remember when using the BT HH5 (not sure if it was both types or one) there used to be a lot of people with PPP LCP problems. I'm not sure if it was solved. I had it for a short time and the only thing that fixed it for me was swapping out the router. I moved from the HH5 setup to the RT-N66U + HG612 setup. I haven't had the PPP LCP issue since.

[aesmith]

I suggest swapping the modem connections just get the first insight.  Calling them line 1, 2, 3 etc then let's assume currently line 1 connects to modem 1 and then to port 1 on the Firebrick.   The Firebrick was where the problem appeared to lie before.   Let's swap round so line 1/modem 1 goes to port 2, and line 2/modem 2 goes to port 1.   Leave the cables unchanged, the only change is to move the end of the cable at the FB from one port to another.

If the dripping blood moves to line 2 then we don't have a mystery, we have an equipment problem which can be raised with A&A.

[burakkucat]

As I understand it, the bigger FireBricks (the FB2500 & FB2700) only have four ports, unlike the little one (the FB105 (as located in The Cattery)) which has five ports -- split as 1xWAN and 4xLAN.

Human nature being what it is, I would expect that in a "one uplink" scenario (one line feeding to one modem feeding to the FireBrick) the modem would be connected to the far left-hand port (numbered one) and the various "downlinks" would be made to the other ports, sequentially, left to right. It would be reasonable, by extension, to expect that in a "multi-uplink" scenario the other "uplinks" would then be to the next free ports sequentially after port number one.

Hence my feeling is a quick test would be to remove the "one downlink" from port 4, move all "three uplinks" one port to the right and re-establish the "one downlink" at port 1.

Try as I might to list all the possible permutations of three modems, three patch cables (modem - FireBrick) and usage of the four ports of the FireBrick, my applied statistical analysis has failed to produce the number (due to the passage of the years). 

There might be some sense in my slightly frivolous suggestion made in one of your previous threads . . . that is to invite Adrian Kennard to have a long weekend at Skye Shepherd Huts and, whilst in the vicinity, take a first-hand look at your problem.

Just one final question . . . If you were not using the A&A monitoring facility would you actually notice anything odd during your everyday usage of the Internet? 

[aesmith]

Hence my feeling is a quick test would be to remove the "one downlink" from port 4, move all "three uplinks" one port to the right and re-establish the "one downlink" at port 1.

I don't know the Firebrick, but wouldn't that mean configuration change?  Whereas swapping modems between two ports wouldn't (I assume).   I like simple tests, testing one thing at a time.

[Weaver]

> If you were not using the A&A monitoring facility would you actually notice anything odd during your everyday usage of the Internet?

That's an excellent point. Tested today, I disabled two lines leaving only line 1 active, and measured the TCP download throughput using speedof.me. The result was extremely healthy at 2.45 Mbps, and will improve further when I reapply the 3 dB downstream target SNRM which was set at 9 dB for testing. So that seems to suggest that it might be a PPP LCP-specific issue.

[Weaver]

Two tired to do any experiments today, so swapouts recommence tomorrow afternoon.

[aesmith]

Would be interesting to see what the KBD diagnostics say about that line.   Throughput is a fairly coarse measure in my opinion, I certainly found essentially no reduction in download speed even when the error rate was bad enough for DLM to keep knocking the speed back.

[ejs]

> If you were not using the A&A monitoring facility would you actually notice anything odd during your everyday usage of the Internet?

That's an excellent point. Tested today, I disabled two lines leaving only line 1 active, and measured the TCP download throughput using speedof.me. The result was extremely healthy at 2.45 Mbps, and will improve further when I reapply the 3 dB downstream target SNRM which was set at 9 dB for testing. So that seems to suggest that it might be a PPP LCP-specific issue.

I wouldn't really call that everyday usage of the Internet, unless all you do is run speedtests. 

[Weaver]

@ejs I'm writing without filling in the gaps. As far as everyday internet goes, I had though that my internet speed was possibly down a bit, so I did a specific speedtest to see if there's anything real or if it's all down to my imagination. The answer would seem to be the latter. The speed will pick up still further when I get the target SNRM on line #1 back to what it used to be, as I mentioned.

[Moderator edited to fix a "finger slippage" typo.]

[Weaver]

*** PROBLEM SOLVED ***

This afternoon's dripping blood tests

Test #1. Swap out modem 1 for fresh modem. (Powered back up at 14:05)
Result: dripping blood _still_ on line @a.1. (during 14:05-14:35)
Rules out modem 1 as culprit

Test #2. Swap DSL cables to modems 1 and 2 at the wall sockets. (Powered back on at 15:09)
line @a.1 now  --> FB port 2
line @a.3 now  --> FB port 1
Result: dripping blood now shows on line @a.3  (!)
Line 1 exonerated
FB port 1 implicated

16:27 swapped modems 1 & 2 back
line @a.1 now  --> FB port 1
line @a.3 now  --> FB port 2

16:50
Port allocation changed to :
LAN port 1
Wan1 port 2
Wan3 port 3
Wan4 port 4
Result: no dripping blood at all, as port 1 is no longer doing PPPoE. Tada!!

[Weaver]

It seems there's a regression in the Firebrick release of mid March (?) or whenever, need to look back at the earlier thread. The Firebricks have some kind of badness relating to the use-case of PPPoE on port 1, possibly this may be a red herring in that it only affects PPP LCP pings and does not affect normal IP, but on the other hand it may be that it's dropping or corrupting a certain fraction of all packets of certain types at who knows what layer, or it's dropping or messing up the responses.

It is present in two units, my earlier FB2500 and my new FB2700. It's always only port 1. It started very suddenly at a definite point in time, and since then it's been roughly constant.

We've completely ruled out modems and the line has no problems, it's a matter of whichever line you link through port 1. And it is not a matter of one bad Firebrick.

Giving the evil port 1 a different job, non-PPPoE, cures the problem, luckily.

If I hadn't been so drugged up and fatigued and confused and generally out of it, I could have worked this out six weeks or so earlier, and £900 sooner. Ho hum.