Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[iMx]

Good Morning,

I've recently migrated to Zen, from BT, and am facing an annoying problem - head banging against a brick-wall has started.  For the last 3 years my current setup has functioned fantastically on BT, since the migration to Zen things seem to have gone a bit wonky - the Zen aspect may or may not be related.

I have an OpenBSD 5.7 router connected to either an HG612 or ECI modem, via a switch, I encounter the same problem with both.  The problem?  PPPoE drops frequently after between 1 - 15 minutes of connected time and reconnects, then repeats, the modem sync is not dropping.  The router has an OpenVPN (UDP) VPN connection that routes all traffic to the OpenVPN server in the DC.  I should add, I still have another line still with BT with the exact same setup and this does not encounter the problem and has been up for some 70 days.

Between migrating from BT -> Zen, the only thing that changed on the OpenBSD router was the PPPoE username/password.  From the moment the migration occurred, this problem started occurring.

Thing's I have ruled out:

- Cabling, no errors on switch ports but all cables have been replaced
- Not HG612 or ECI related, problem happens with both.  Initially thought it could be the HG612 bug with UDP/VPNs, however the modem is unlocked and running the latest release.  The trick of unplugging and reseating the eth cable doesn't make any difference.
- OpenBSD config, there is minimal kernel PPPoE config same setup works with BT and continues to work
- OpenBSD OS versions (tried 6 different images/releases)
- Rolled back RFC4638 patches,  i.e for MTU 1500.  The Max Payload is negotiated successfully during the connection process, so I don't believe this is the issue but have tried without anyway.
- LCP echo/replies are all being sent and responded to in a timely manner, there are no ignore/dropped echos/replies before the 'term-req' is received'

Enabled debug on the OpenBSD pppoe interface and it seems to me, that Zen are sending 'term-req' - although I need to make sure my reading of the logs is correct i.e 'lcp input' is the ISP/Zen.  However, the below logs also show 'Down event (carrier loss)' but there is no carrier loss (the modem stays in sync) and all ethernet ports between the modem/switch/router stay up, no errors, etc - although this could be because the term-req has already been received and the disconnection is in process.

Code: [Select]

Jun 28 21:15:56 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2eb output -> 84:26:2b:a2:3c:da, len=139
Jun 28 21:15:56 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2eb output -> 84:26:2b:a2:3c:da, len=139
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp input(opened): <term-req id=0x0 len=4 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-3f-75-80-e0-83-59-1b-c3>
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp opened->stopping
Jun 28 21:15:56 rtr00 /bsd: pppoe0: phase terminate
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp down(opened)
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp opened->starting
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp close(starting)
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp starting->initial
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp send terminate-ack
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp output <term-ack id=0x0 len=4>
Jun 28 21:15:56 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2eb output -> 84:26:2b:a2:3c:da, len=12
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp TO(stopping) rst_counter = 0
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp stopping->stopped
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase dead
Jun 28 21:16:06 rtr00 /bsd: pppoe0: timeout
Jun 28 21:16:06 rtr00 /bsd: pppoe0: disconnecting
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp down(stopped)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp stopped->starting
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase establish
Jun 28 21:16:06 rtr00 /bsd: pppoe0 (8863) state=1, session=0x0 output -> ff:ff:ff:ff:ff:ff, len=18
Jun 28 21:16:06 rtr00 /bsd: pppoe0: Down event (carrier loss), taking interface down.<7>pppoe0: lcp close(starting)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp starting->initial
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase dead
Jun 28 21:16:06 rtr00 /bsd: pppoe0 (8863) state=2, session=0x0 output -> 84:26:2b:a2:3c:da, len=38
Jun 28 21:16:06 rtr00 /bsd: pppoe0: session 0x2ee connected
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp open(initial)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp initial->starting
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase establish
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp up(starting)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp starting->req-sent
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp output <conf-req id=0x5 len=14 05-06-d2-4f-8e-af-01-04-05-d4>
Jun 28 21:16:06 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2ee output -> 84:26:2b:a2:3c:da, len=22
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp input(req-sent): <conf-req id=0xa7 len=19 01-04-05-d4-03-05-c2-23-05-05-06-1a-89-24-a4-00-00-00-00-00-00-00-00-00-00-00-20-85-80-10-ff-1c-cb-c5>

Next steps:
- Test with the TG589 router supplied by Zen, although putting this in bridging mode doesn't seem particularly easy (GUI reconfigure for bridge doesn't seem to work, suspect it misses the 101 VLAN tag).
- Let another device do the PPPoE aspect, and hand an RFC1918 address to the OpenBSD router, see if the problem still occurs..

.. If anyone has any suggestions, or seen anything similar previously, I'm all ears

[iMx]

Just tried with the Zen TG589, in bridge mode, same problem.  Have fired off an email to the OpenBSD misc mailing list, will open a case with Zen in case they can shed any light.

To me, this either has to be a bug in the OpenBSD PPPoE implementation (although why it would work on BT, AA is beyond me), or a fault :-/

[iMx]

Tried with the Zen TG589 in normal router mode doing all the PPPoE, with the OpenVPN UDP tunnel, problem still occurs - looks like this is potentially a bug or incompatibility somewhere with the OpenBSD/Zen PPP implementation, although I know of a number of other OpenBSD users who also use Zen with a bridged FTTC modem.

[burakkucat]

This does look rather like an OpenBSD problem and is rather puzzling because, as you say, the set-up works with other suppliers (ISPs/CPs). 

[iMx]

This does look rather like an OpenBSD problem and is rather puzzling because, as you say, the set-up works with other suppliers (ISPs/CPs). 

Thanks for the reply - also the ISP/BT/Zen seems to be sending/initiating the disconnection:

Code: [Select]

Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp input(opened): <term-req id=0x0 len=4 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-3f-75-80-e0-83-59-1b-c3>

Every aspect of the setup, from hardware, to config, to cables, to physical ports, works fine (or so it seems)... just not on Zen and/or not for me

[Dray]

Can you post a traceroute to bbc.co.uk please?

[iMx]

Can you post a traceroute to bbc.co.uk please?

Sure, this is through the TG589 with the PPPoE session:

Code: [Select]

                                                         My traceroute  [v0.86]
rtr00 (0.0.0.0)                                                                                 Mon Jun 29 19:22:13 2015
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                Packets               Pings
 Host                                                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. dsldevice.lan                                                                              0.0%    19    0.3   0.3   0.3   0.5   0.0
 2. losubs.subs.dsl2.th-lon.zen.net.uk                                                         0.0%    19   18.4  18.5  18.2  18.9   0.0
 3. ae1-115.cr2.th-lon.zen.net.uk                                                              0.0%    19   30.5  21.6  18.2  34.3   5.5
 4. ge-2-0-0-0.cr1.th-lon.zen.net.uk                                                           0.0%    19   34.9  24.3  17.9  74.1  15.1
 5. 82.71.254.134                                                                              0.0%    19   19.0  19.8  18.3  33.4   3.5
 6. ???
 7. ae0.er01.cwwtf.bbc.co.uk                                                                   0.0%    19   19.7  19.8  19.2  21.2   0.2
 8. 132.185.255.165                                                                            0.0%    19   20.4  20.9  19.8  26.4   1.4
 9. fmt-vip133.cwwtf.bbc.co.uk                                                                 0.0%    18   19.2  19.5  19.2  19.9   0.0

[kitz]

also the ISP/BT/Zen seems to be sending/initiating the disconnection:

hmmm...    the Zen/PPP drops seem to rear its head every so often and I don't think we've ever truly got to the bottom of what's causes it.

Splbound was about the most recent whereby he had issues where he said that his HG612 was retaining sync, but losing PPP to Zen.   It got a bit more confusing though because his SNRm was dipping and at one point I wondered if what something to do with idle time as his router was reporting "pppd:No response to 3 echo-requests"...   but going back even further jeff had an issue with frequent loss of PPP on Zen (but still had sync).

[iMx]

No such timeouts for me I'm afraid, the LCP echo request/reply goes back and forth - the last, 4 seconds before the disconnect:

Code: [Select]

Jun 29 13:20:42 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x16 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-b1-d0-1d-2b-00-00-00-00>
Jun 29 13:20:42 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:42 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x16 len=8 9b-cc-13-a7>
Jun 29 13:20:45 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x17 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-31-92-e8-9b-00-00-00-00>
Jun 29 13:20:45 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:45 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x17 len=8 9b-cc-13-a7>
Jun 29 13:20:55 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x18 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-54-f8-2b-14-00-00-00-00>
Jun 29 13:20:55 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:55 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x18 len=8 9b-cc-13-a7>
Jun 29 13:20:58 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x19 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-44-34-28-75-00-00-00-00>
Jun 29 13:20:58 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:58 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x19 len=8 9b-cc-13-a7>
Jun 29 13:21:01 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x1a len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-fc-ff-39-26-00-00-00-00>
Jun 29 13:21:01 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:21:01 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x1a len=8 9b-cc-13-a7>
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp input(opened): <term-req id=0x12 len=4 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-fc-08-46-fa-00-00-00-00>
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp opened->stopping
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp send terminate-ack
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp output <term-ack id=0x12 len=4>

[iMx]

Am trying a hunch that has so far proved to be correct - but more time is needed to prove the theory.

On my connection I have/had an OpenVPN UDP tunnel on my router, this routes all traffic to a box in the DC - the FTTC connection for all intents and purposes is an access line into the DC network over the VPN.  The only traffic that Zen will 'see' is UDP based, or the large majority, other than random port scans/pings to my external Zen IP.

Switching the tunnel to TCP, so far, no more 'term-req' from Zen and the PPP session has been up an hour.  This *never* happened when the OpenVPN tunnel was set to UDP.  Potentially, my set up exacerbates the problem - as it is all UDP - potentially other people who have seen this problem, only trigger it sometimes when certain traffic goes out?

Does anyone know if Zen to anything funky with UDP traffic? i.e assuming it's torrents, trying to balance me to another server over and over?  Long shot... but so far, the UDP OpenVPN tunnel seems to be the trigger for me.  Or a bug in the Zen LNS (assuming this would be LNS area)?

Will see how things go....

[iMx]

Possibly a red herring, have switched back to OpenVPN UDP, bounced the PPP session.... PPP has stayed up longer than it ever did previously with this setup, no term-req yet.  Wondering if Zen support have found something in the background whilst I was testing, or if I ended up on another LNS somewhere along the way.  I just hope I'm not one of these people that see's this issue with Zen, then it disappears, only to return...

.. enough geeking for the day, going to the beach :p

[iMx]

When I had (potentially still have) this problem it just randomly went away, whilst I was trying a number of things - my setup went back to exactly what it was to start with, i.e router/modem/etc, problems disappeared.  What was 100% clear from my diagnostics, is that it was Zen that was sending the 'term-req' and pulling the PPP session down - the problem being, no one from Zen could tell me why.

I did notice that at times when I was getting disconnected I was going through a gateway that was suffering high latency, dsl4 at the time seemed to be the worst for me, there are other threads relating to this on TBB.  Shortly after the PPP session died after Zen killed it from their end, most of the time, I ended up on a different gateway and it would stay up for a bit longer.  Then it would drop again, but dsl4 seemed to be consistently bad for me for the first 3-4 days of my line transferring over to Zen.

Current PPP session is just under 3 days, with 4.5 days modem sync.  Am I happy since migrating to Zen?  Not overly to be honest, I wasted considerable time looking into this and going through the support-script but with no clear outcome - if I'd wanted that, I could have stayed with BT!  With hindsight, I wish I hadn't bothered and just stayed with BT.  Time will tell if the service they provide changes my mind...

[iMx]

Zen PPP Session got up to around 5 days, then reset... then it reset every day or so, then a few times a day, now I'm back to every 1 - 20 minutes or so.  :-/  Problems can be seen clearly in the below Mouselike graph, modem sync continues to be rock solid:

https://f8lure.mouselike.org/archived_graphs/sv-vdsl0.no-ip.net_day8.png

tMy 2nd BT FTTC line continues to have a PPP up time of around 15 days.

[iMx]

For completeness sake, as I haven't posted proof of the MAC sending the term-req yet - tcpdump from the pppoedev interface, LCP echo-request/reply goes back and forth, nothing dropped, then the remote end sends the 'term-req' @21:49:22.681482.

94:de:80:8f:a4:23 is the local MAC of the pppoedev interface, 84:26:2b:a2:3c:da is/was the remote for this session

Code: [Select]

21:49:12.583483 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 10
        LCP: Echo-Request, Magic-Number=1419402018
21:49:12.583505 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 30: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 10
        LCP: Echo-Reply, Magic-Number=2119939257
21:49:15.624809 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 10
        LCP: Echo-Request, Magic-Number=1419402018
21:49:15.624828 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 30: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 10
        LCP: Echo-Reply, Magic-Number=2119939257
21:49:18.604635 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 10
        LCP: Echo-Request, Magic-Number=1419402018
21:49:18.604655 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 30: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 10
        LCP: Echo-Reply, Magic-Number=2119939257
21:49:22.681482 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 6
        LCP: Terminate-Request
21:49:22.681509 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 26: PPPoE-Session
        code Session, version 1, type 1, id 0x0264, length 6
        LCP: Terminate-Ack
21:49:32.742741 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 32: PPPoE-Session
        code Session, version 1, type 1, id 0x0265, length 12
        LCP: Configure-Request, Magic-Number=-82992445[|lcp]
21:49:32.787984 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0265, length 21
        LCP: Configure-Request, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709, Vendor-Ext
21:49:32.787987 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0265, length 12
        LCP: Configure-Ack, Magic-Number=-82992445, Vendor-Ext
21:49:32.788012 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 41: PPPoE-Session
        code Session, version 1, type 1, id 0x0265, length 21
        LCP: Configure-Ack, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709[|lcp]
21:49:33.367008 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
        code Session, version 1, type 1, id 0x0265, length 21
        LCP: Configure-Request, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709, Vendor-Ext
21:49:33.367041 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 41: PPPoE-Session
        code Session, version 1, type 1, id 0x0265, length 21
        LCP: Configure-Ack, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709[|lcp]
Resetting on average of every 5 minutes at the moment.

[iMx]

Beginning to again suspect Zen do something funky when the traffic seen is all UDP (or even primarily UDP), the moment I enable an OpenVPN UDP tunnel PPP starts getting reset again - between 1 and 20 minutes - traffic is generally pretty low, 10-100K/S, apart from the odd burst when rsync backups happen.  So, I shut down the OpenVPN tunnel on the Zen routing domain and then loop downloaded a 100M speedtest file from Linode, so all traffic is TCP (other than maybe the odd DNS request).....PPP stays up.

It is interesting that the days I saw latency problems there were no PPP resets, after they did maintenance on DSL4 (confirmed) latency improved but the PPP resets started again.  I have absolutely nothing concrete to back this up, but I'm beginning to wonder if there is something load-balancing-wise on their end that was disabled for a few days, latency gets worse, they re-enable it and I'm back to the PPP resets when most traffic is UDP.  Are they potentially seeing it as P2P/Torrent traffic and trying to move me around to cope with the additional load of Wimbledon?!

Whilst it could be a coincidence, this started again yesterday afternoon about 14:00 when latency started to increase again - this was when Andy Murray was playing, things went mad last time he was playing in the tournament.