Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Robbie]

Are there any ISPs piggy-backing off Openreach's network that offer FTTP without PPPoE, other than TalkTalk?

(I know the SINs presume PPPoE, so looking for exceptions).

 

[XGS_Is_On]

So Openreach SINS don't presume PPPoE that's just how many roll and I believe the only other one that doesn't use it is Sky, so either their Sky Broadband or Now TV brands.

[Dave Jones]

Just out of curiosity, what's the problem with PPPoE?

[Chrysalis]

It gets asked a lot, primarily two things, it natively doesnt support 1500 bytes MTU, although a work around exists using baby jumbo frames.

It also can add considerable processing cost for internet throughput, ISP devices often get around this but can be an issue on open source solutions.

[Alex Atkin UK]

Also the fact even if you have baby jumbo frames, technically your connection will be slightly slower due to the protocol overhead.  I don't think its a noteworthy amount, but I think we all would prefer the most efficient method if possible.

[Robbie]

Just out of curiosity, what's the problem with PPPoE?

PPPoE can be tricky to handle.  Some cheap dedicated hardware chips can offload the PPPoE element in much of the same way that NAT can be offloaded in a cheap ISP provided box.  Unfortunately when higher bandwidths are needed these cheap dedicated bits of silicon run out of steam and routers start to lean on multicore Atom CPU chips and above.  These higher class CPUs do not have PPPoE offloading, although the better server or mini-server orientated CPUs do tend to have more advanced and dedicated crypto handling, relieving the CPU of that task.

Where it all gets messy is that almost all current OSs & CPUs handle PPPoE through a single core.  This reduces your theoretical throughput on your typical multi-core beast to that of a single core machine.  A lot of higher-end routers run on Linux & BSD and they all share the single-core PPPoE issue.  Whilst we lag behind a lot of countries in terms of FTTP even Openreach is starting to push >1 Gbps products.  PPPoE bottlenecks will just cause more and more issues.

My router is reasonably powerful, with QAT offloading, multiple fully-routed interfaces, all with modern Intel NICs.  It has a brace of 10 GbE ports and multiple 2.5 GbE ports.  In simple routing it can route 10 Gbps WAN traffic with ease; with typical higher-end features enabled it would drop to around 7 Gbps.  I would guess with a PPPoE WAN it probably struggle at around 1.5 Gbps, maybe a little less.  It is an £800+ OEM router....

So you can see the problem.  Those who need higher bandwidths are probably the exact same people running QoS, advanced firewall features, perhaps intrusion monitoring, VPNs and alike.  Being forced to use PPPoE on high-bandwidth links will require very expensive hardware so that a single core can work very hard on the PPPoE problem, leaving the other cores with little to do.

PPPoE was fine at low bandwidths on dedicated silicon.  The list of countries using PPPoE on >1 Gbps is vanishingly small and none of the major CPU manufacturers see a viable market in dedicated silicon for PPPoE. 

BT/Openreach plan on introducing a 1.8 Gbps FTTP service later this year.  PPPoE WANs have a problem.

☕️

[bogof]

Because of this issue (seen on Ubiquiti gear amongst others) I built a SFF PC that I use running OpenWRT just for PPPoE offloading, and then have a router behind it.  My ISP (AAISP) gave me a block of 8 public IPs so I just use one for the downstream router to avoid double NAT.  Works great, and means the router doesn't run out of steam thanks to menial PPPoE duties.
If we get 1.8G Openreach services here I'll update the Quad 1G ethernet card in the SFF PC to Quad 2.5G.

[craigski]

If I understand this correctly, PPPoE has been implemented inefficiently, single threaded, in certain types of OS. The solution for end user running one of these OS is to 'throw hardware' at the issue when running higher bandwidth services.

What I don't understand yet, is how the ISPs have got round the potential performance issue(s) with 1000's of PPPoE connections. I'm assuming they could have a similar issue at the PPPoE server end, so what hardware and OS are they running on the servers/access concentrators?

[Alex Atkin UK]

BT/Openreach plan on introducing a 1.8 Gbps FTTP service later this year.  PPPoE WANs have a problem.

Its not quite as bad as it was a few years ago thanks to low-end laptops/embedded CPUs having high IPC CPU cores, but less of them than desktops.

I'm not 100% sure, but I think my Celeron N5105 might be able to do PPPoE at >2Gbit on pfSense, on OpenWRT it should do more thanks to the Linux PPP being multi-threaded.

Its definitely a waste of energy/CPU cycles, but at least its not limited to expensive hardware any more.

[Alex Atkin UK]

What I don't understand yet, is how the ISPs have got round the potential performance issue(s) with 1000's of PPPoE connections. I'm assuming they could have a similar issue at the PPPoE server end, so what hardware and OS are they running on the servers/access concentrators?

ISP equipment is designed for the job so likely has full PPP offloading from the CPU.  As each unit will handle hundreds maybe thousands of customers, any price premium for it is written off as it can be used regardless of what technology the customer is using to connect.

Its less common on the consumer side simply because its been cheaper to do it on the CPU and a lot of countries don't use PPP.  An ISP provided gateway may in fact have it as they know what they need for their customers, but overall the market is much smaller for consumers buying their own router so its less likely to be there.

There's plenty of ISPs worldwide that don't let you use your own gateway, just support some sort of a bridge mode or DMZ, so PPP would still be handled in their box.

[Robbie]

If I understand this correctly, PPPoE has been implemented inefficiently, single threaded, in certain types of OS.

It's more of a CPU issue but the underlying issue was with IP itself.  I'm simplifying here but it was never designed to be multithreaded and the presumption was that you would always want to keep streams on the same path to avoid packets arriving out of order, dislocation of fragments causing malformed packets or triggering collisions.  The retransmissions triggered by these errors would drive the workload higher and effective throughput would plummet.

In order to get IP to work across modern multicore CPUs the incoming packets are hashed, so that they can be tracked (and error checked) before being reassembled, in order, no matter which core or thread they traversed.  It does not work with 100% reliability but the increase in processing bandwidth more than makes up for the inherent possibility of packets being out-of-order.

The problem with PPPoE is that it isn't, in any way, an IP standard - it was never designed to be so.  It is a particularly well-named protocol as it was designed for point-to-point use only, with this specific variant designed for ethernet use.  It cannot physically traverse the internet - so upstream nodes, routers, BGP etc never have to deal with it.

So in this context when we send data we wrap our internet-destined IP packet in a PPPoE straightjacket and at the first internet hop it is forced to shed the PPPoE wrapper and become just a regular IP packet again before it is allowed to bounce around the world to its final destination.  When ISPs / Openreach apply PPPoE it is only ever in the context of the first or last hop from to or from the customer's premises.

☕️

[meritez]

A lot of higher-end routers run on Linux & BSD and they all share the single-core PPPoE issue. 

☕️

but

I'm not 100% sure, but I think my Celeron N5105 might be able to do PPPoE at >2Gbit on pfSense, on OpenWRT it should do more thanks to the Linux PPP being multi-threaded.

Its definitely a waste of energy/CPU cycles, but at least its not limited to expensive hardware any more.

So is PPPoE Single Core or Multi-threaded?

[craigski]

So is PPPoE Single Core or Multi-threaded?

I assume it can be both, depending on the implementation/OS.

I don't see PPPoE CPU issue on Draytek router. Its CPU normally around 4%, if I fully load the 1GBs PPPoE WAN port using PPPoE CPU goes upto 6%

I think the CPU (NPU?) is GRX350 or similar and OS is DrayOS.

https://assets.maxlinear.com/web/documents/anywan%20soc%20grx350_flyer.pdf

[Robbie]

So is PPPoE Single Core or Multi-threaded?

The quote about Linux PPP being multi-threaded its not mine and my understanding of OpenWRT is more limited; no doubt Alex can expand on his information.

My understanding is that Linux does not utilise more than 1 core for PPPoE.  However, there is an accelerator package available for some Linux distros that works both at user-space and (critically) kernel-level to spread the work across threads.  It's a highly credible and open-source project [ACCEL-PPP] - perhaps this has been absorbed into OpenWRT in an effort to overcome the native issue?

The CPUs used in Dreytek routers are dedicated silicon with offload engines for NAT and PPPoE, so that workload does not impact the CPU load.  Where these CPUs become unstuck is with traffic that is ineligible for offload, forcing everything through the CPU.  If all your traffic can be offloaded they work just fine.  Start doing intensive QoS, VPNs, traffic monitoring, packet inspection, heavy encryption etc they will fall on their knees as they are just not designed for those roles.

[XGS_Is_On]

If I understand this correctly, PPPoE has been implemented inefficiently, single threaded, in certain types of OS. The solution for end user running one of these OS is to 'throw hardware' at the issue when running higher bandwidth services.

What I don't understand yet, is how the ISPs have got round the potential performance issue(s) with 1000's of PPPoE connections. I'm assuming they could have a similar issue at the PPPoE server end, so what hardware and OS are they running on the servers/access concentrators?

BT Wholesale use these among other things: https://www.nokia.com/networks/ip-networks/multi-access-gateway/