Kitz Forum

Broadband Related => FTTC and FTTP Issues => Topic started by: Ixel on May 25, 2021, 01:45:23 PM

Title: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on May 25, 2021, 01:45:23 PM
EDIT: Linked to PDF file of the scan of the generic letter hand delivered recently by Lightning Fibre. I would've attached it but it's too large to attach :D.

PDF Link: http://fr89.uk/files/lightningfibre.pdf (http://fr89.uk/files/lightningfibre.pdf)

---

This thread will serve as a diary for my experience in hopefully signing up to Lightning Fibre soon. I will update it every so often when I have something significant to mention. Some people on here might be curious about Lightning Fibre or just generally my overall experience. I already have FTTP, albeit via Openreach and I paid for this to be installed around 2-3 years ago I believe. I know I'm out of contract again, on native FTTP, so I know I'm definitely over two years of having the service. I got a reasonably low-end quote at the time, certainly compared to the average quotes I've seen people mention on some forums more recently.

Briefly, Lightning Fibre are a local altnet ISP which are rolling out gigabit symmetrical FTTP across where I live and a few nearby cities/towns. The ONT's they install are capable of 10Gbps, but for now they are rolling out up to 1Gbps with a future aspiration of offering more than that. They tend to dig their own trenches and build their own cabinets. They don't appear to share much of Openreach's infrastructure from what I understand, although I believe they will use Openreach's pole across the road to deliver FTTP to me.

Another nice thing about their service is that, according to their website, the first month is free (for home users only). This is to allow time to cease the previous connection, effectively avoiding paying for two ISP's in the same month.

More information can be found on their website at https://www.lightningfibre.co.uk/

I intend to go for the home 1,000 megabit package on the longest contract term offered (free installation).

On another note, Cityfibre are also busy rolling out their FTTP network here too. Plenty of roadworks going on thanks to all this :P.



25/05/2021 - 1pm:
Approximately 1-2 weeks ago I received a hand delivered letter from Lightning Fibre that stated they were about to do some necessary roadworks/streetworks to lay the fibre. The whole process is usually finished in a month or two and if I have any questions to phone or email them.

Well, I emailed them some months before I had this recent letter and they took approximately one working day to reply to my question regarding availibility of an IPv4 /29 block. Unfortunately they "done" (yes, that was a typo in their email I guess) offer this yet but the network director is currently looking into it. They also stated they would get back to me with a further update before the original ETA of me being able to signup. Originally the ETA for my street to be live was between December 2020 and June 2021, clearly it's going to likely be much sooner than that.

I've since emailed them again, twice, but sadly nearly two weeks have passed by without any response at all. This doesn't inspire me with much confidence on what my experience with customer service might be like if I was to become their customer. I could phone them but it's not an urgent matter, especially as I'm looking at possibly around two months before I can consider signing up.

I know I initially had a problem emailing them. The mailshot email that I had from their community engagement person said to reply with some information like my current ISP and/or questions. I did reply, but I later found out that they apparently never received my reply. I'm hoping this is just another situation where, for some reason, their email system is silently not allowing my messages to be received. If that's the case then I don't know why, my server isn't blacklisted, has a valid RDNS/PTR, a valid SPF and has DKIM configured. No other email address I've sent to so far has had such a problem.

In the meantime I've managed to workaround the additional IPv4 addresses problem by using AAISP's L2TP business service, which provides me with the number of additional IPv4 addresses that I need (a /29). Unfortunately the service is capped to 100Mbps but it's better than potentially having no additional IPv4 addresses at all. I don't generally use it for my NAT traffic, especially due to the speed cap considering that my current FTTP connection is on something like 1000/115.

AAISP have been super helpful (Andrew on their IRC), as usual, with assisting me to configure my Firebrick to accommodate the additional IPv4 addresses without my NAT traffic also going through the L2TP. The only time the NAT traffic will go through the L2TP is if my PPPoE connection to Cerberus Networks (my current ISP) is down, in which case it will go through the USB mobile dongle that's connected to the Voxi network.

I'll also scan in the letter I received, the one I mentioned earlier in this post, later today and attach it for those curious.



26/05/2021 10am:
I've had a reply on my other email, after trying to email them from my gmail instead. Long story short, they say they won't know for at least a few more months as to whether they can offer additional IPv4 addresses to customers. They will update me as soon as they have more details. On another note, it appears as if the reply was written in a little bit of a hurry. Some shorthand usage and typos were present, e.g. 'tho' instead of 'though', but as long as the service itself is good and if and when a repair needs doing it'll get done then that's what I'm mainly interested in! :)

I probably won't be able to provide any further updates for the next few weeks now (or perhaps 2-3 months), as I need to wait until the necessary works are done so that I can begin to signup. They also mentioned in the reply that it could easily be until August before I can signup.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: meritez on May 25, 2021, 03:03:42 PM
Quote
Lightning Fibre is a brand new broadband network for East Sussex
We are a cutting edge, community minded full fibre network operator and ISP based in Eastbourne, East Sussex.

Unlike a Ďfibreí network, which may include old copper wire, weíre undertaking a major infrastructure upgrade in Eastbourne, Hastings, Hailsham and Heathfield to lay a brand new Ďfull fibreí broadband network that will eventually reach almost every home and business in these towns.

I'm curious, but I doubt they'll build to Junction 15 of the M1
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: kjw on May 25, 2021, 03:49:53 PM
Who would have thought Hastings/Eastbourne would eventually have access to 3 fibre networks! Crazy :)

What makes you want to leave your OR FTTP behind?
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on May 25, 2021, 04:05:24 PM
Who would have thought Hastings/Eastbourne would eventually have access to 3 fibre networks! Crazy :)

What makes you want to leave your OR FTTP behind?

Indeed.

Higher monthly cost and a lack of symmetrical speed, those are mainly the reasons for considering moving. While I don't have the exact price to hand, I'm paying Cerberus somewhere near £150 per month for Cerberus FTTP Ultra2 with a block of IPv4 /29. Even with just AAISP's L2TP service providing the /29 and Lightning Fibre's top home plan on a long contract, it's considerably less cost per month and for a symmetrical gigabit speed too. Even allowing for upkeep of a 4G backup connection, it's still cheaper than sticking with Openreach's FTTP.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: kjw on May 25, 2021, 05:18:14 PM
Ouch! Is that an FTTPoD line or are they really that expensive?!

Obviously the symmetrical part has OR beat, but someone like Zen could probably supply a block of IPS on the 1000/110 package for a lot less.

I wonder how many of these ALT's will still be around when the dust settles, new ones are popping up every day and at some point all that investment will need paying back :)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on May 25, 2021, 05:24:39 PM
Ouch! Is that an FTTPoD line or are they really that expensive?!

Obviously the symmetrical part has OR beat, but someone like Zen could probably supply a block of IPS on the 1000/110 package for a lot less.

I wonder how many of these ALT's will still be around when the dust settles, new ones are popping up every day and at some point all that investment will need paying back :)

It was an FTTPoD line, this is native FTTP now and yes going through Cerberus (they go via BT Wholesale, which as I understand it isn't cheap) and paying a small fee per month for the extra IPv4 addresses does work out to near that per month. I used to be with Zen a long time ago but due to some bad experiences (around single threaded download speed issues) I now avoid them. I had to leave them a negative Trustpilot review at the time, which got someone's attention and they released me from my contract early as well as refunding me, after which I updated the review to mention this. So while I didn't leave Zen on a bad note, I'm hesitant to consider them again.

Regarding the alts, indeed, who knows how many will survive in the long term.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: grahamb on September 04, 2021, 08:15:55 PM
@Ixel

Having seen your post in the News Articles section, and reading through this thread, are you located around the Latimer Rd area? I've only just decided to register an interest and I'm wondering where Lightning Fibre have planted/are going to plant their little grey cabinet and when exactly the service will be going live. They were digging up the pavement and putting down cable outside the terrace where my flat is located  two or three months ago, as you probably already know.  :)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on September 06, 2021, 10:05:29 AM
@Ixel

Having seen your post in the News Articles section, and reading through this thread, are you located around the Latimer Rd area? I've only just decided to register an interest and I'm wondering where Lightning Fibre have planted/are going to plant their little grey cabinet and when exactly the service will be going live. They were digging up the pavement and putting down cable outside the terrace where my flat is located  two or three months ago, as you probably already know.  :)

No but it's only a few streets, it's nearby. I can't be certain but I presume the vicinity might be connected to a cabinet in Firle Road or just off of it. I haven't seen it for myself as I don't go that way, but someone I know recently told me something about a cabinet around there, unless they were just joking with me :D. I had a letter through the letterbox a while back, so I'm hoping for more news soon. I'm keeping a close eye on the roadworks via one.network. I recently saw two guys from Lightning Fibre around the back, looking at the pole and the various manhole covers for Openreach infrastructure, presumably assessing/surveying. A family member about 5 minutes down the road (on foot) is booked in this week to have theirs installed, after being phoned late Friday evening that it's now ready and would they like to place an order.

I'm also curious to find out, eventually, how the connection is authenticated. I'm hoping they don't use PPPoE, and looking at their gigabit speed test on YouTube I would say they probably don't (doesn't look like there's overheads for it). I'm also waiting to hear back regarding customers having an option of purchasing a block of IP addresses instead of just a single static IP. However I won't need multiple IP addresses from them now as I've got a block via OVH using a GRE tunnel and routing them to my local network, albeit at least the downstream is rate limited to around 250Mbps before the latency can rise up to +100ms, so I've capped it to 200Mbps downstream. I'm not sure if upstream will be similar until I have Lightning Fibre's gigabit service here. Either way that's still ample speed for the server stuff.

Also 'little cabinet' might be wrong ;). Have you seen the one in Old Town near the Co-op at Albert Parade?
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: grahamb on September 06, 2021, 06:53:54 PM
I've not been around the Albert Parade area for a few years now (since having to have my car scrapped) and no good reason to get on a bus/walk there, although my interest has now been piqued. I was just going by the size of the cabinets I have seen dotted around in, for example, Enys Rd and elsewhere.  :)

Also, I read through your second paragraph and the more words I read, the less I understood...  ;)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on September 06, 2021, 07:00:53 PM
I've not been around the Albert Parade area for a few years now (since having to have my car scrapped) and no good reason to get on a bus/walk there, although my interest has now been piqued. I was just going by the size of the cabinets I have seen dotted around in, for example, Enys Rd and elsewhere.  :)

Also, I read through your second paragraph and the more words I read, the less I understood...  ;)

I guess the cabinet size may vary depending on location and number of premises to connect, but there's photos of the one at Albert Parade at https://twitter.com/LightningFibre/status/1410331687356088331. As for the second paragraph, sorry I may have gone a bit technical I guess :). I haven't been to Old Town area for a while now myself, bus fares are high for single trips and I don't drive either. Mostly working from home so I don't really have a need to travel far on a regular basis thankfully. If I do need to go somewhere that's not within walking distance I'd rather just get a taxi these days.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on October 23, 2021, 09:46:36 AM
Another update, not majorly significant but one I'll share anyway.

I had a letter through the letterbox yesterday, virtually identical to the one I had before (which I'm now starting to wonder if I should've had in the first place). However, the major difference this time is that they've been previously working on the other half of the longish road I'm on. Now there's roadworks planned by Lightning Fibre virtually at the telegraph pole I'm connected to. I believe I possibly received the first letter in error due to longish the road and the funny route it takes. Once again the letter says I should be able to connect to Lightning Fibre within a month from now (or something to that effect), which would co-incide with the planned roadworks here and a few weeks leeway after the planned finish date.

Exciting times ahead perhaps, maybe I'll be able to signup and go live before Christmas :fingers: :yay:.

I've not had a follow up regarding the request for multiple IP addresses, but I suspect that might not be possible at this time. When I look at how many they appear to have on their ASN, which I assume includes allocations for customer connections, it's unlikely to be adequate for those who would need or want multiple IP addresses at the moment (512 + 512 + 32,768 + 2,048).
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on October 23, 2021, 03:07:45 PM
Exciting times ahead perhaps, maybe I'll be able to signup and go live before Christmas :fingers: :yay:.

Ah, but before which Christmas?  :D
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on October 23, 2021, 08:19:37 PM
Ah, but before which Christmas?  :D

I hope this Christmas. :fingers:
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: SaltyWagyu on June 02, 2022, 07:37:43 PM
What router do Lightning Fibre provide? Is it possible to set it up as modem-only mode (or bridge, in other words) so that I can use my existing Asus RT-AX86U router with it?
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: tubaman on June 03, 2022, 08:35:07 AM
What router do Lightning Fibre provide? Is it possible to set it up as modem-only mode (or bridge, in other words) so that I can use my existing Asus RT-AX86U router with it?
Their website support page (https://www.lightningfibre.co.uk/support) suggests it's a dual band Zyxel device, but it doesn't mention the model.
"Check the LEDs on the Zyxel Router (the freestanding box). The ďPowerĒ, ďGlobeĒ, ď2.4GĒ and ď5GĒ LEDs should be green..."

If you wanted to use your own router you'd ideally connect it directly to the ONT and not use their one at all. However, Zyxel routers can be locked to the ISP's settings so if you really need to use your own I'd contact them and ask before committing to anything.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: SaltyWagyu on June 03, 2022, 12:53:07 PM
If you wanted to use your own router you'd ideally connect it directly to the ONT and not use their one at all. However, Zyxel routers can be locked to the ISP's settings so if you really need to use your own I'd contact them and ask before committing to anything.

Thank you, I somehow missed that they supply a modem and router seperately. Hopefully I can use the Asus with the modem using PPPoE in that case, if that's what they use. I will contact them before ordering.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on June 03, 2022, 04:39:27 PM
. . . I somehow missed that they supply a modem and router seperately.

There is no "modem" with a FTTP service. The equivalent device is an ONT (optical network terminal). (Sometimes referred to as an ONU (optical network unit).)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: j0hn on June 03, 2022, 11:13:20 PM
It's also unlikely they will use PPPoE.
They more than likely use IPoE/DHCP.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on June 24, 2022, 11:37:55 PM
Hi, I thought I'd drop by with an update. I'm unable to edit the original post.

Earlier today I had a phone call from Lightning Fibre finally giving me good news, my street is live. The order process was easy, they had most of my details from the 'register my interest' form ages ago. The only information they needed was for setting up a direct debit, of course. I've been given a fantastic deal on their gigabit package (making the total price per month less than £30/mo compared to the current £44/mo), especially given the lengthy wait and how long I've been out of contract with Cerberus paying a rather high price each month.

I asked how the connection is established, e.g. PPPoE or IPoE. While the person on the phone wasn't certain, they believed like me that it's IPoE. No authentication details are given for the connection, at least I've not seen any on their members portal and had no such details via email. I was also told that they might be re-evaluating their pricing for a static IP address (at £10/mo), as they've observed that it's a fair bit higher than many other ISP's via Openreach's network. At the moment I've not opted for a static IP address, I may not need it even though I use a GRE tunnel (Mikrotik supports using a subdomain/domain name for the 'remote address' on a GRE tunnel so I can likely use DDNS).

They are aware that I will want to use my own router instead of theirs which is fine, they also suggested to make sure that the engineers are aware of this on the day of installation.

By the middle of next month, or sooner if there's an earlier installation slot that becomes available, it should be installed. Once it's done then I can sort out cancelling my FTTP service with Cerberus Networks.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on June 25, 2022, 04:13:23 PM
Hi, I thought I'd drop by with an update.

Thank you. I was wondering what was the current status with your FTTP service.

Quote
Earlier today I had a phone call from Lightning Fibre finally giving me good news, my street is live. The order process was easy, they had most of my details from the 'register my interest' form ages ago. The only information they needed was for setting up a direct debit, of course. I've been given a fantastic deal on their gigabit package (making the total price per month less than £30/mo compared to the current £44/mo), . . .

Good news, indeed.  :)

Quote
I asked how the connection is established, e.g. PPPoE or IPoE. While the person on the phone wasn't certain, they believed like me that it's IPoE.

From what I have seen, in my reading, I'm fairly certain that it is IPoE.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on June 25, 2022, 05:28:28 PM
There's only one thing I wasn't aware of until I read a page somewhere on their website. It talks about if you're living in a rented property then permission is needed (a signed wayleave) from the landlord. I wasn't asked about this on the phone either. I rent a house and Openreach never asked for this when I had FTTPoD installed. However, I'll contact my letting agent on Monday and ask them about it, I strongly doubt the landlord will be awkward.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: SaltyWagyu on June 26, 2022, 02:15:45 PM
They've been very difficult to get any information out of. I asked on their community page two questions https://www.facebook.com/LightningFibreBroadband/posts/989354918426245 after getting no response from the enquiry form, then I emailed them like they asked, still nothing today.

Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on June 26, 2022, 02:39:46 PM
They've been very difficult to get any information out of. I asked on their community page two questions ... after getting no response from the enquiry form, then I emailed them like they asked, still nothing today.

Not good.

Well, for your second question while I can't say for absolute certain... I highly doubt they would remove the Openreach master socket as they are neither authorised to do so and it's not their 'property'. As for the first question, it remains a mystery for now. I've not had any authentication details (beyond an invitation to access their members portal), I have a feeling that it's IPoE and not PPPoE (which is good as I'd prefer IPoE :)).

When I asked them about using my own router they said that's fine and to mention this to the engineers on the day of installation so that they can assist with getting it working on Lightning Fibre if needed. However I'm fairly certain I can set that up myself, so long as I know the type of connection it makes (presumably IPoE).

The only details mentioned on their members portal are the current IPv4 address, current IPv6 address and current IPv6 prefix which presumably will show some values when the connection is established. Other details are the ID, address, contract end date, scheduled installation date and time, price per month and any discount, install cost, service type, provisioned status, SLA level, static IP status and finally the service type (e.g. Home Hyper Lightning 1000M/1000M for me).
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: SaltyWagyu on June 27, 2022, 06:46:16 PM
Thanks Ixel, keep us posted on getting it working with your own router  :fingers:
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on June 28, 2022, 09:36:13 PM
Thanks Ixel, keep us posted on getting it working with your own router  :fingers:

Will do.

I've also been told today by someone else who's using Lightning Fibre that it's IPoE/DHCP. It's thankfully not PPPoE.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on June 29, 2022, 02:34:30 AM
Any telco that is starting from scratch would be crazy to use PPPoE, its only a requirement for legacy reasons on Openreach (so neither OR nor the ISP has to use two different authentication methods across their different packages).

I do wish PPPoE would die, but it seems unlikely any time soon as even if everyone was on fibre it would be a pain getting all the customers to change the settings on their routers.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 08, 2022, 11:49:18 PM
Had a courtesy call earlier today checking if Monday was still fine, which was nice. At first I was slightly worried it might've been a call needing to reschedule the install due to an unforseen circumstance, thankfully not though.

Only a few days to go until the fun begins!
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on July 09, 2022, 06:36:14 AM
My anxiety was through the roof once I had the install date, constantly expecting a cancellation.

Unfortunately its how I ended up with the ONT and CSP lower down than I would have liked and a disconnected analog line. I was basically "whatever gets this thing live ASAP", with hindsight I should have marked exactly where I wanted everything.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 09, 2022, 09:52:09 AM
Ouch, yeah I can imagine :D.

Even though I have FTTP with Cerberus Networks I'll be glad when I have Lightning Fibre not just because it's symmetrical but also because they have faster ports at both LINX and LONAP. Cerberus Networks was just 1G each on LINX (until recently perhaps as I now see 10G listed too on their PeeringDB profile). I say this because I've been noticing congestion on LINX (at least perhaps until 10G was listed, haven't really checked again) and LONAP. While Cerberus Networks has 2x 1G's at both internet exchanges, the congestion was still noticeable on traffic which went via those internet exchanges.

This morning I'm working on moving my configuration from one RouterOS installation to another, the idea being that I have a spare RouterOS to configure Lightning Fibre on without interupting the current network. Once I've confirmed I've got it working on that router then I can move most of the configuration back to it again and all being well that's it :fingers:.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 09, 2022, 04:11:11 PM
The countdown has well and truly commenced.  ;)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on July 09, 2022, 06:08:00 PM
I'd never thought to look up peering before (well not in the last decade or so), very interesting.

PeeringDB is really useful as it means you can figure out which peer a traceroute refers to.  In fact, for ping/traceroute tests I can pick a host over different peers to see how each is behaving, neat.

I guess AAISP VoIP should work pretty well with peering straight in Telehouse.  Mind you, I've been impressed with how fast I can get to Europe now, its not much slower than VDSL often was within the country.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 10, 2022, 09:34:10 AM
I'd never thought to look up peering before (well not in the last decade or so), very interesting.

PeeringDB is really useful as it means you can figure out which peer a traceroute refers to.  In fact, for ping/traceroute tests I can pick a host over different peers to see how each is behaving, neat.

I guess AAISP VoIP should work pretty well with peering straight in Telehouse.  Mind you, I've been impressed with how fast I can get to Europe now, its not much slower than VDSL often was within the country.

Yeah PeeringDB is useful, along with other places such as bgp.he.net, to check the ISP's connectivity. I started delving into PeeringDB and related resources when I setup my own ASN with RIPE :).

Going slightly off topic for a moment...

As I'm now using BGP from my home router, instead of OVH's /24 (well, I'm in the process of migrating) it has been extremely useful in helping to understand the best choices on which transits to work with (e.g. Mythic Beasts, CDN77/Datapacket, Vultr/Choopa, to name a few). That however will possibly need re-evaluating when I'm on LF next week. Overall cost for my ASN, IPv4 /24 and IPv6 /48 is actually less than I was paying OVH for just the dedicated server too which is nice. Motivators have primarily been that LF doesn't offer an IPv4 /29 and OVH's ASN is constantly being marked as spam by UCEProtect level 3. Someone also told me that LF's IPv6 implementation is currently broken, not sure if that's true though but yeah that's not really relevant for me since I have my own IPv6 /48 with BGP anyway. Another nice thing is I can traffic engineer to some degree, meaning I can avoid announcing my prefix(es) to a specific ASN, internet exchange or such in order to forcefully manipulate the route that the traffic takes.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 11, 2022, 12:46:29 PM
Hi all,
Installation complete. I've swapped it over to my MikroTik router and somehow I'm getting above the speed I should at the moment :D.

(https://www.speedtest.net/result/13392067642.png) (https://www.speedtest.net/result/13392067642)

Installation lasted about 2 hours. A few vehicles and engineers turned up.

It's definitely DHCP, no PPPoE nonsense.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 11, 2022, 01:48:01 PM
Another success story. Congratulations.  :)

Do you still, currently, have the old service active just in case an ELF rears its head?
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 11, 2022, 02:26:05 PM
Another success story. Congratulations.  :)

Do you still, currently, have the old service active just in case an ELF rears its head?

Thanks :).

Yes, for now I have the old service on standby. I'll give it a week or two before I contact them to arrange cancellation. As it is I've already had a problem which I'm fairly certain is something to do with my MikroTik's configuration. I've had internet connectivity disappear and it was restored as soon as I forced the ethernet interface down and up, so that's an issue I'm currently trying to figure out the cause of. If it persists I will use their Zyxel router temporarily to confirm it's my equipment and not a problem that's out of my control.

Other than that hiccup, so far so good!

EDIT: The problem may have been caused by my somewhat strict firewall configuration. Anything on the input chain of WAN that's TCP or UDP and is a 'not yet established' connection is dropped. The DHCP client from what I understand uses UDP port 68, which I've now allowed. However, as an added precaution I've also made a scheduled script on my router which forcefully renews the DHCP client lease every 15 minutes. So far the connection hasn't dropped since then.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 12, 2022, 12:11:44 PM
It worked well yesterday, the maximum speed was adjusted a few hours after the installation so I guess something was misconfigured or not fully configured on their end.

So... yesterday at least for traffic not going via LINX and instead going via LONAP I was seeing single thread download speeds of around 900+ megabits. On LINX however I was seeing an average of 250-350 megabits. Upload speeds were consistently 900 megabits+ for traffic going via either internet exchange on the various servers in London that I did test with (Vultr VPS, Mythic Beasts VPS, UK Dedicated Servers dedicated server, OVH (Erith) dedicated server and Misaka.io VPS which uses CDN77/Datacamp/Datapacket's network).

I initially saw a fast download single thread speed (via LONAP) somewhere around 7:45am this morning, but unfortunately since then I've not seen such speed since possibly 9-10am onwards. Typical results appear to be a brief burst at about 600+ megabits and then it settles around 200-350, no matter whether it's going via LINX or LONAP for the above mentioned servers to connect to my router here. Multiple thread speed appears to be fine.

I'm not 100% certain if it's my end currently or not but I have doubts that the problem is my end. I'll do a brief test at around 10pm tonight to see if speeds improve (e.g. with the possibility of less people using LF due to going to sleep). If I'm able to pretty much conclude it's not my end then I will get in touch with LF's support and see what they say, after I put their router back in place.

Here's a recent speed test from TBB at least:
(https://www.thinkbroadband.com/_assets/speedtest/button/1657623120387514555.png)

Obviously this is more than adequate for video streams of high quality though for example, I realise that!

On other news, it looks like the timing to change network was convenient as I believe my Openreach ONT's power supply plug is developing a fault. I was plugging something in and heard some 'electrical noise' as if something was making and breaking perhaps. Observing that nothing else on the extension block was showing any possible problems, I looked at the ONT's power LED and noticed it was orange and discovered it was running off of the BBU. The power supply plug was firmly in the extension block too, but I was hearing some kind of electrical noise from the plug. I've unplugged it and the BBU's power to the ONT at the moment.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 12, 2022, 06:06:19 PM
On other news, it looks like the timing to change network was convenient as I believe my Openreach ONT's power supply plug is developing a fault. I was plugging something in and heard some 'electrical noise' as if something was making and breaking perhaps. Observing that nothing else on the extension block was showing any possible problems, I looked at the ONT's power LED and noticed it was orange and discovered it was running off of the BBU. The power supply plug was firmly in the extension block too, but I was hearing some kind of electrical noise from the plug. I've unplugged it and the BBU's power to the ONT at the moment.

Was the BBU supplied as part of the installation? (I forget how many years ago it was when you decided to take a FTTPoD service.) From your mention of the colour of the ONT's power LED, I deduce that there is a telemetry cable installed between the ONT and BBU. So that implies either a Huawei 4+2 or a Huawei 1+1 ONT. So I guess the answer to my question is "yes".  ;)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 12, 2022, 06:57:09 PM
Yes the BBU was supplied as part of the installation. I believe this was around, or at least close to, the time when Openreach was phasing BBU's out. It's been quite some time since I've had FTTPoD installed. I have a Huawei 1+1 ONT and there is indeed a telemetry cable between the BBU and ONT.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 14, 2022, 08:56:35 AM
Here's an update.

With the help of another Lightning Fibre customer, a business customer, I've been able to confirm that the low single thread download speed issue isn't isolated to me. It's a somewhat strange behaviour in my opinion because it seems to start off near gigabit generally and then a few seconds later drop down to 200-400 megabits as if there's some kind of burst shaping going on, but that's probably not the case and just looks like it.

Me and another Lightning Fibre customer performed a local test, pretty much, using iperf3 between our Lightning Fibre IP addresses. The results showed near gigabit speed in either direction.

They also did a speed test on speedtest.net (single thread) using the Lightning Fibre server and they unfortunately had a similar result to me. They also did a speed test from their dedicated server in London using the Lightning Fibre server and were able to achieve multiple gigabits no trouble.

With their permission I've passed on this data to Lightning Fibre and am currently waiting for their next response.

Their initial response, while extremely quick, wasn't really inspiring. I was asked to use the speedtest.net app (Windows, Android or Mac) instead of the website as they said the website test can't generally keep up and showed me multi threaded test results comparing both the Windows app and the website on their laptop. I pointed out in my initial ticket message that it's single thread speeds as well as the result of that test. I also gave them the results I had on day one which were far better.

Unfortunately the speedtest.net Windows app doesn't appear to have an option for doing a single threaded test. Android however does so I plugged in their equipment temporarily and stood within a meter using my mobile to do a test, albeit wireless, and sent them the result of that test. Neither my system (an overclocked Threadripper 3990X 64-Core CPU) or the other Lightning Fibre's customer's system (an i9-12900K) are slow systems :D.

I also sent Lightning Fibre some additional iperf3 test results using two dedicated servers both on different networks and both with 10 gigabit ports. The tests were performed between my Lightning Fibre connection and the dedicated servers. Unfortunately they also showed similar slow single thread speeds. I've checked if this is depends on time of day and it's consistently on the slow side no matter if it's early morning, midday, early evening or late evening.

So, from this I at least know I'm not the only customer who could be experiencing a slower than imagined single thread download speed. I know it's not my equipment and I also know their network locally is performing fine so it's likely a problem somewhere between their network and the internet (issues or congestion with their transit(s), an underlying fault somewhere or possibly a misconfiguration). I'm hoping my findings will eventually be passed on to someone who has much greater access to their network infrastructure, e.g. someone in their NOC, to look into this and hopefully resolve it.

It's not a deal breaker but it's not what I was expecting. Perhaps at peak times I would've expected slower single thread download speeds but not all day long after day one.

This is likely a good test run of how their customer service is. Hopefully a positive outcome will come from this! :fingers:
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 14, 2022, 03:55:11 PM
With the help of another Lightning Fibre customer, a business customer, I've been able to confirm that the low single thread download speed issue isn't isolated to me.

Obviously I would not expect you to post any details regarding that Lightning Fibre (business) customer but perhaps you could say if they are either (1) Eastbourne based, (2) Sussex based or just (3) England based. I.e. I am wondering how local you are to each other.

It is good that you are able to test using iperf3 between your respective IP addresses.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 14, 2022, 05:00:13 PM
Obviously I would not expect you to post any details regarding that Lightning Fibre (business) customer but perhaps you could say if they are either (1) Eastbourne based, (2) Sussex based or just (3) England based. I.e. I am wondering how local you are to each other.

It is good that you are able to test using iperf3 between your respective IP addresses.

Their office is in Eastbourne.

Indeed, I thought it might be useful for Lightning Fibre if they take this matter somewhat seriously. I've not had another update yet so I'm hopeful that it's probably due to waiting for feedback from someone else who has sufficient access to see what's happening on their network.

Unfortunately the problem may be getting worse. Earlier today I did a random iperf3 test with the OVH dedicated server and here's a comparison of the results from yesterday vs today.

Yesterday:
Code: [Select]
Accepted connection from x, port 60782
[  5] local x port 5201 connected to x port 60784
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  78.4 MBytes   657 Mbits/sec
[  5]   1.00-2.00   sec  72.6 MBytes   610 Mbits/sec
[  5]   2.00-3.00   sec  70.4 MBytes   590 Mbits/sec
[  5]   3.00-4.00   sec  39.9 MBytes   335 Mbits/sec
[  5]   4.00-5.00   sec  41.2 MBytes   345 Mbits/sec
[  5]   5.00-6.00   sec  43.0 MBytes   361 Mbits/sec
[  5]   6.00-7.00   sec  46.8 MBytes   392 Mbits/sec
[  5]   7.00-8.00   sec  27.0 MBytes   227 Mbits/sec
[  5]   8.00-9.00   sec  59.2 MBytes   497 Mbits/sec
.....

Today:
Code: [Select]
Accepted connection from x, port 32866
[  5] local x port 5201 connected to x port 32868
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  9.45 MBytes  79.2 Mbits/sec
[  5]   1.00-2.00   sec  16.2 MBytes   136 Mbits/sec
[  5]   2.00-3.00   sec  11.2 MBytes  94.6 Mbits/sec
[  5]   3.00-4.00   sec  13.7 MBytes   115 Mbits/sec
[  5]   4.00-5.00   sec  17.6 MBytes   148 Mbits/sec
[  5]   5.00-6.00   sec  12.3 MBytes   104 Mbits/sec
[  5]   6.00-7.00   sec  13.2 MBytes   110 Mbits/sec
[  5]   7.00-8.00   sec  11.7 MBytes  98.2 Mbits/sec
[  5]   8.00-9.00   sec  21.7 MBytes   182 Mbits/sec
[  5]   9.00-10.00  sec  15.0 MBytes   126 Mbits/sec
[  5]  10.00-10.01  sec  42.8 KBytes   199 Mbits/sec
.....

Also a recent TBB test from just a moment ago:
(https://www.thinkbroadband.com/_assets/speedtest/button/1657814649564068355.png) (https://www.thinkbroadband.com/speedtest/results.html?test=1657814649564068355)

I'm starting to wonder if I'm expecting too much from gigabit FTTP, but then I recall having much better single thread download speeds on Cerberus Networks FTTP - so I'm not sure. However, if the single thread download speed continues to degrade further... well, it won't be too far away from potentially affecting UHD streams :D.

Here's the iperf3 test between two Lightning Fibre connections from yesterday:
Code: [Select]
Accepted connection from x, port 59072
[  5] local x port 5201 connected to x port 59074
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  88.6 MBytes   743 Mbits/sec
[  5]   1.00-2.00   sec  97.3 MBytes   816 Mbits/sec
[  5]   2.00-3.00   sec  99.7 MBytes   837 Mbits/sec
[  5]   3.00-4.00   sec  99.5 MBytes   835 Mbits/sec
[  5]   4.00-5.00   sec  97.8 MBytes   817 Mbits/sec
[  5]   5.00-6.00   sec  91.9 MBytes   774 Mbits/sec
[  5]   6.00-7.00   sec   108 MBytes   909 Mbits/sec
[  5]   7.00-8.00   sec  98.9 MBytes   830 Mbits/sec
[  5]   8.00-9.00   sec  98.1 MBytes   823 Mbits/sec
[  5]   9.00-10.00  sec   102 MBytes   854 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec   982 MBytes   824 Mbits/sec                  receiver

Accepted connection from x, port 59076
[  5] local x port 5201 connected to x port 59078
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  91.2 MBytes   765 Mbits/sec
[  5]   1.00-2.00   sec  90.1 MBytes   755 Mbits/sec
[  5]   2.00-3.00   sec  97.1 MBytes   815 Mbits/sec
[  5]   3.00-4.00   sec   104 MBytes   872 Mbits/sec
[  5]   4.00-5.00   sec   105 MBytes   879 Mbits/sec
[  5]   5.00-6.00   sec  93.4 MBytes   783 Mbits/sec
[  5]   6.00-7.00   sec  80.4 MBytes   674 Mbits/sec
[  5]   7.00-8.00   sec   101 MBytes   850 Mbits/sec
[  5]   8.00-9.00   sec  96.2 MBytes   808 Mbits/sec
[  5]   9.00-10.00  sec   102 MBytes   857 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec   961 MBytes   806 Mbits/sec                  sender
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  receiver

As you can see it's a much better speed and indicates nothing wrong 'locally' at least.

Perhaps the phrase "you get what you pay for" has more truth to it after all? :-\
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 14, 2022, 05:16:56 PM
Hmm . . . I see. Puzzling.  :-\
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on July 14, 2022, 06:45:50 PM
Sounds similar to what I had with Origin Broadband.  Something about their infrastructure meant I never got good single-threaded speeds the whole time I was with them, both on Digital Region and Openreach.  When I later re-activated that line with Plusnet it was a huge difference.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 15, 2022, 11:33:29 AM
Sounds similar to what I had with Origin Broadband.  Something about their infrastructure meant I never got good single-threaded speeds the whole time I was with them, both on Digital Region and Openreach.  When I later re-activated that line with Plusnet it was a huge difference.

Ouch, I hope I don't get a similar outcome with LF then! :fingers:

I've sent them another message earlier this morning with some more comparisons and speed tests. One being Cerberus Networks via Openreach FTTP and the other being Lightning Fibre. Ignoring the upload speed, as it's capped at a much lower value on Openreach FTTP, the difference between Cerberus Networks and Lightning Fibre's single connection/thread download speed is considerable.

(https://www.speedtest.net/result/13409460601.png)

(https://www.speedtest.net/result/13409489306.png)

-----

(https://www.thinkbroadband.com/_assets/speedtest/button/1657873193266896855.png)

(https://www.thinkbroadband.com/_assets/speedtest/button/1657874392413946455.png)

The only really nice thing about Lightning Fibre at the moment is the upload speed. Latency is also excellent but is virtually identical to Cerberus Networks. Multi connection/threaded download is almost identical.

When I have more news I'll share what I can. Hopefully some good news will follow soon.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on July 15, 2022, 01:22:56 PM
Question is, does TBB do a single-threaded upload test or is that multi-threaded?

Single threaded is even more important as downloads can usually be multi-threaded quite easily, but uploads cannot.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 15, 2022, 01:31:58 PM
Question is, does TBB do a single-threaded upload test or is that multi-threaded?

Single threaded is even more important as downloads can usually be multi-threaded quite easily, but uploads cannot.

While I'm not 100% certain, I believe upload is single threaded. I'll try to find out.

I've had a response from Lightning Fibre, unfortunately somewhat mixed. Maybe someone here can suggest something.

---

They thanked me for the information I've given them and said that it's indeed unusual.

They managed to recreate similar behaviour their end when running a single TCP stream.

They checked all network devices in the path and even tested over different backhaul links, but it all looks normal as far as they can tell.

On their laptop they saw similar speeds, between 3-400 Mbps on a single TCP stream. Running a single TCP stream over LAN with two other clients on the same LAN saw near gigabit speeds.

It was repeatable behaviour. They said given the network infrastructure is seemingly quite happy moving that traffic at that rate (and above), it all looks well otherwise. They believe that this is most likely a client issue at one or both ends. Their guess is that it relates to the TCP window sizing or similar given throughput otherwise is normal and single stream TCP tests are fine on some clients repeatably.

They aren't sure that there's anything they can alter to affect this and can't be of much more help but hope that their feedback helps give me more information on what they can see from their end.

Any suggestions?

I'm not sure what to say at this point. If Cerberus Networks works fine through the same networking equipment (router being a CCR2004-16G-2S+ by Mikrotik) and computer, just a different ONT and LAN cable, I don't understand how it can be a client side issue.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 15, 2022, 03:31:19 PM
Any suggestions?

No, nothing comes to mind.

Quote
If Cerberus Networks works fine through the same networking equipment (router being a CCR2004-16G-2S+ by Mikrotik) and computer, just a different ONT and LAN cable, I don't understand how it can be a client side issue.

s/and LAN cable/and Ethernet patch cable/  --  for clarity.

For completeness, it would be sensible to try a new Ethernet patch cable (used to connect the MikroTik router to the ONT of the service under test) and use it with both ONTs.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 15, 2022, 03:57:44 PM
No, nothing comes to mind.

s/and LAN cable/and Ethernet patch cable/  --  for clarity.

For completeness, it would be sensible to try a new Ethernet patch cable (used to connect the MikroTik router to the ONT of the service under test) and use it with both ONTs.

I thought about trying the same Ethernet patch cable that I use with the Openreach ONT, but I can't because the one that's connected to the Openreach ONT goes through a hole in the wall. It was cut and rejoined. The Lightning Fibre ONT is on the other side of the wall, it's not in the same room as the Openreach ONT. I can try a new cable though. I also tried a 'cable test' on RouterOS but it said 'link ok'. I have a testing gadget too which I can try but I have a feeling it would say that there's no problem with the cable going to the Lightning Fibre ONT.

I've replied to Lightning Fibre suggesting one other idea I can try but I highly doubt it will make a difference. However if it's fine to do so then it can't hurt to try. I know with Openreach's ONT you can power cycle it without any fuss, I imagine it's the same with Lightning Fibre's ONT but I've asked them just to make absolutely certain. It's the only other idea I have right now. In the reply I also mentioned how it was fine on day one and that I don't recall anything I've done here which could've caused this. If I did then surely the Cerberus Networks connection would also be slow (unless it's a dodgy cable connected to the Lightning Fibre ONT but I'll eliminate that possibility too).

EDIT: I've had a response, they don't recommend power cycling the ONT 'very often' because there have been issues where the '10G port' has been affected by 'constant power cycles'. Doing so the once should be fine though.

I only plan to do so once, just to see if it corrects this issue. I highly doubt it will though but it's worth a try.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 15, 2022, 04:18:55 PM
I thought about trying the same Ethernet patch cable that I use with the Openreach ONT, but I can't because the one that's connected to the Openreach ONT goes through a hole in the wall. It was cut and rejoined. The Lightning Fibre ONT is on the other side of the wall, it's not in the same room as the Openreach ONT.

Hmm . . . I think I understand. So just checking: The Openreach ONT is in room A, the Lightning Fibre ONT is in room B and your equipment rack (cupboard) is in room B?

Do you have an Ethernet patch cable that is long enough to (temporally) run via room A door, hallway, room B door? Scientifically, the "same cable experiment" really needs to be performed.

Power-cycling or powering off the ONT for anything between, say, 5 minutes and 23 hours should have no effect on either service.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 15, 2022, 04:37:23 PM
Hmm . . . I think I understand. So just checking: The Openreach ONT is in room A, the Lightning Fibre ONT is in room B and your equipment rack (cupboard) is in room B?

Do you have an Ethernet patch cable that is long enough to (temporally) run via room A door, hallway, room B door? Scientifically, the "same cable experiment" really needs to be performed.

Power-cycling or powering off the ONT for anything between, say, 5 minutes and 23 hours should have no effect on either service.

Yes (regarding room A and room B). I may have a long enough cable, I'll look through my cables and see what I have.

It's interesting how they said that 'constant' power cycling of the ONT has caused issues with the 10G port, though I don't know how often they consider 'constant' to be. I haven't power cycled the ONT ever but I have unplugged the ethernet patch cable from either my Mikrotik router or the Lightning Fibre (Zyxel) router a few times, mostly for diagnosing the slow speed issue though.

I'll post another update here in a bit when I have something to mention.

EDIT: For the curious, the ONT looks like this: https://www.adtran.com/solutions/by-segment/products/by-category/fiber-access/optical-network-terminals-ont/621.html

I see it has a standard gigabit ethernet port as well, but I'm uncertain if the service is configured to also work over that and whether it's even wise to consider seeing if anything's different on that port if it did work.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 15, 2022, 08:56:05 PM
Just an update to say that the cable made no difference. The power cycle didn't either, as expected.

After some consideration, I'm considering the option of re-contracting (and perhaps migrating) the Openreach FTTP connection to an ISP such as A&A. The Lightning Fibre connection will be kept, however...

As I use BGP and have my own block of IPv4 and IPv6 addresses I will also make use of these for the browsing the internet. I should be able to setup asymmetrical routing by having it so incoming traffic goes via Openreach FTTP (whichever ISP I end up choosing) while outgoing traffic goes via Lightning Fibre FTTP. This should be possible with two GRE tunnels, one for each connection. Before I re-contract/migrate the Openreach FTTP connection I will test this possibility with the current ISP (Cerberus Networks) to ensure it's possible, but in my mind I believe it should be.

It still works out cheaper than I'm paying Cerberus Networks at the moment (out of contract for a long time).

Who knows, Lightning Fibre may fix this issue one day in the future. The above seems like my best option I guess as I have a feeling it would be a challenge to get out of the contract early with LF without paying for the contract's term, even with the issue I'm having. Having both connections also provides some redundancy I guess. :)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 15, 2022, 09:19:22 PM
EDIT: For the curious, the ONT looks like this: https://www.adtran.com/solutions/by-segment/products/by-category/fiber-access/optical-network-terminals-ont/621.html

I see it has a standard gigabit ethernet port as well, but I'm uncertain if the service is configured to also work over that and whether it's even wise to consider seeing if anything's different on that port if it did work.

Thank you. I always appreciate sight of any relevant equipment.

Just an update to say that the cable made no difference. The power cycle didn't either, as expected.

So now all the hardware related experiments have been completed.

Quote
As I use BGP and have my own block of IPv4 and IPv6 addresses I will also make use of these for the browsing the internet. I should be able to setup asymmetrical routing by having it so incoming traffic goes via Openreach FTTP (whichever ISP I end up choosing) while outgoing traffic goes via Lightning Fibre FTTP. This should be possible with two GRE tunnels, one for each connection. Before I re-contract/migrate the Openreach FTTP connection I will test this possibility with the current ISP (Cerberus Networks) to ensure it's possible, but in my mind I believe it should be.

As you have explained it, it should be feasible.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on July 16, 2022, 12:57:45 AM
Its certainly an expensive solution, but given you already are using BGP it should be quite a nice solution.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 16, 2022, 01:03:37 AM
Done it! Seems to work fine :)

(https://www.thinkbroadband.com/_assets/speedtest/button/1657929372852067155.png)

(https://i.imgur.com/yxUGQxA.png)

I've masked out the ISP name in the speedtest.net result because the ISP name is effectively my 'real name' due to it being from one of my own IP addresses announced via BGP.

Download is going via Cerberus Networks FTTP currently (Openreach), but I will be migrating to A&A soon. Once I do the download speed should be much better. Cerberus Networks only has 2 x 1 gigabit ports on LONAP, not a 10 gigabit port at least, so I believe there's a little congestion going on there. Multiple connections/threads won't reach gigabit either, understandably.

Upload is going via Lightning Fibre FTTP.

As for pricing, I currently pay somewhere in the realm of £150/mo for Cerberus Networks. Even paying AAISP 5TB Home::1 1000/115 and Lightning Fibre 1000/1000 (at the deal pricing I have), it's still a bit cheaper per month.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Weaver on July 16, 2022, 01:33:42 AM
Iím pretty sure A&A would dig into this. They have their published "no b*llsh*t" policy and they can read a TCP protocol captured because thatís where the answer lies: the problem is why does TCP hate you so much and the answer is in: the precise timing of packets, packet loss, distribution of ACKs in time, per-connection field values, window sizes, buffer size, window scale, network pipe capacity, transit times so if someone looks at all that lot and has sufficient clue, then they can work out who is to blame within those two TCP instances.

Itís not good enough an ISP proving that their network has sufficient bandwidth by using a wise technique such as iperf or multiple TCP connections to saturate the link, because although those methods do accurately measure the network rather than the software implementation of TCP and o/s performance at the two ends, the behaviour of a single TCP connection is a Ďreal worldí scenario that deserves a realistic expectation of performance, itís not something wacky, highly unusable, or unrealistic.

I am possibly getting similar problems with the performance of upstream on my system with three internet access pipes, where there may or may not be multiple TCP connections in use in some scenarios. Many speed checkers, not just one, give disappointing values for upstream throughput, compared to the carefully calculated one for TCP PDU and TCP SDU (ie payload) total throughput. In the past the issue has varied markedly over time. It might be that the fact that there is one slow link out of the three is significant. But since there have been sizeable periods where full bandwidth has been seen, then I take it that that shows, as with you, that the network can handle the expected throughput which after all is extremely slow, weíre only talking about a few hundred kbps per link and 1.6 Mbps IP PDU throughput max; hardly challenging for multi-gigabit routers. Iím convinced the answer lies in TCP behaviour, but I donít have the concentration or the experience to read TCP protocol captures, although I can obtain them.

I canít remember; I may have talked to AA about this, but I havenít pestered them to do the tcpdump analysis thing for me, because I havenít had the energy to get into this, Iíll admit, and Iíve kept doing more digging and experiments myself. In my case not only is it AAs network but itís also their router, a Firebrick, so thatís another big reason why I can indeed legitimately ask for help, as it may be that the Firebrick is not doing the packet tx scheduling over the three unequal links in a TCP-friendly way. And thatís a whole other can of worms that you donít have. I could go back to them and get into it again but the truth is that Iíve been really unwell at times.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 16, 2022, 10:03:04 AM
Well, I've placed an order with A&A which should be completed around 02/08 (migration). The order system charged me the wrong setup fee but I've sent them an email about it so hopefully that will be corrected next week.

Iím pretty sure A&A would dig into this. They have their published "no b*llsh*t" policy and they can read a TCP protocol captured because thatís where the answer lies: the problem is why does TCP hate you so much and the answer is in: the precise timing of packets, packet loss, distribution of ACKs in time, per-connection field values, window sizes, buffer size, window scale, network pipe capacity, transit times so if someone looks at all that lot and has sufficient clue, then they can work out who is to blame within those two TCP instances.

Yeah if this was A&A's issue they would dig into it. I was honestly expecting a bit more from Lightning Fibre, suffice to say I'm a bit disappointed with them and I shall in the next couple of weeks leave an updated review stating what's happened as well as the currently disappointing outcome. While this is an annoying issue, which as more customers signup may become worse, it has been a fairly useful test run of their customer service. Granted this isn't an ordinary "every day" kind of problem they are likely tasked with though.

Itís not good enough an ISP proving that their network has sufficient bandwidth by using a wise technique such as iperf or multiple TCP connections to saturate the link, because although those methods do accurately measure the network rather than the software implementation of TCP and o/s performance at the two ends, the behaviour of a single TCP connection is a Ďreal worldí scenario that deserves a realistic expectation of performance, itís not something wacky, highly unusable, or unrealistic.

Indeed, I even proved to them that a 'local' connection between Lightning Fibre connections was performing fine and the problem is likely somewhere between their network and the internet. Instead they are content with the fact that 'some devices' are performing fine and that their network infrastructure is fine (at least locally anyway). They haven't spent much time investigating this matter from the sounds of it and are satisfied with also shifting the blame of the issue back to me and/or the servers I'm using to perform the speed tests (which ironically would also include their own Lightning Fibre server on Ookla I guess :D).

I am possibly getting similar problems with the performance of upstream on my system with three internet access pipes, where there may or may not be multiple TCP connections in use in some scenarios. Many speed checkers, not just one, give disappointing values for upstream throughput, compared to the carefully calculated one for TCP PDU and TCP SDU (ie payload) total throughput. In the past the issue has varied markedly over time.

Ouch, that's not good. I wish my issue showed some kind of variance but sadly it seems to be any time of day or night. The only significant variance I've had is when comparing day one of the service being live to any day thereafter. Great download speed on day one and considerably slower download speed from the following day onwards.

I canít remember; I may have talked to AA about this, but I havenít pestered them to do the tcpdump analysis thing for me, because I havenít had the energy to get into this, Iíll admit, and Iíve kept doing more digging and experiments myself.

Interestingly someone on Discord suggested I try looking at the 'sniffer' function in RouterOS and doing some comparisons to see if I can notice any anomalies which might explain what's going on. It's something I will try this morning, I'll just use RouterOS's own 'bandwidth test' feature this time as it gave similar results to iperf3 and other single TCP stream speed tests anyway.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 16, 2022, 11:56:14 AM
EDIT: Sadly this is normal behaviour from what I've been told. Another person who has an IPoE/DHCP type of connection is seeing something similar on a different ISP. Sadly it's unlikely to be the reason.

---

I have some more news to share, a possible discovery but I think I need the assistance of this forum in understanding whether this is correct or not.

I went ahead with using 'sniffer'. I compared both Cerberus Networks connection and Lightning Fibre connection for receiving packets using a single stream bandwidth test.

This is on the Cerberus Networks connection:
https://i.imgur.com/osZWx5Y.png

This is on the troublesome Lightning Fibre connection:
https://i.imgur.com/TeFOgz4.png

The packet size on the Lightning Fibre connection is inconsistent and going well above 1500, unlike Cerberus Networks. Of course, Cerberus Networks is PPPoE while Lightning Fibre is IPoE/DHCP so I'm not sure if that's why I'm seeing this massive difference.

I'd like to hear what anyone else here has to say about this, is this possibly normal behaviour for an IPoE/DHCP connection and not the reason I'm getting a slow throughput?
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: bogof on July 16, 2022, 12:52:36 PM
Could it be jumbo frames related?
What package are you going for with AAISP?  I will perhaps be shopping for a new provider soon given what I seem to be seeing with my Zen connection at the moment.  Be interesting to see what it looks like when you get it set up.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 16, 2022, 02:35:22 PM
Could it be jumbo frames related?
What package are you going for with AAISP?  I will perhaps be shopping for a new provider soon given what I seem to be seeing with my Zen connection at the moment.  Be interesting to see what it looks like when you get it set up.

That's what I'm thinking, but from what I heard this is possibly normal. Although someone pointed out to me that there's also potentially a pattern going on where as on their connection they don't see something like that. It's very puzzling to say the least.

As for the package, I've chosen Home::1 5TB (1000/115).

Zen... I used to be with them some years ago. In the early days of using Zen it was fantastic but going back to the time just before I left Zen and switched to AAISP I remember having some throughput issues with Zen which were never resolved. I wasn't their only customer suffering similar issues at the time either. Zen released me from the contract without me needing to pay an early termination charge after I left them a detailed and factual negative review on Trustpilot.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: bogof on July 16, 2022, 04:00:51 PM
That's what I'm thinking, but from what I heard this is possibly normal. Although someone pointed out to me that there's also potentially a pattern going on where as on their connection they don't see something like that. It's very puzzling to say the least.

As for the package, I've chosen Home::1 5TB (1000/115).
Thanks, that's the package I've been considering.  Our usage "only" gets up to around 2TB a month and that is with working from home and lots of streaming, so the 500/800GB limits would be too low but the 5TB tier would be fine.   I'll be watching with interest.

The only thing that really concerns me slightly with AAISP is that given they're a bit of a niche player, whether they have good links into the various content delivery peering systems that are around and improve our streaming experience.  For that reason (and, frankly, cost too) I'm quite tempted to give BT Business a go.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 16, 2022, 04:29:42 PM
To get some idea I guess you can take a look some resources such as these:

https://bgp.he.net/AS20712#_peers

https://bgp.he.net/AS20712#_graph4

https://www.peeringdb.com/net/2077

On their wiki they also mention something: https://support.aa.net.uk/Peering_and_Transit_-_Gamers

I believe AAISP only counts what you download towards your quota, upload is not counted. You also get the quota bonus each month, allowing you to rollover some unused quota from the previous month.

When I was with AAISP I don't recall having issues with throughput, no matter whether it was a single TCP connection or multiple TCP connections.

EDIT: I've done a BQM graph on TBB just in case that helps highlight something.

Today's snapshot:
(https://www.thinkbroadband.com/broadband/monitoring/quality/share/1ccbfd2d510189c55d7bb58795a8c5d6d08e7fbb-16-07-2022.png)

Ignore the packet loss around 6:30pm as that was me doing something.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: bogof on July 16, 2022, 06:36:13 PM
Thanks, really interesting, though very hard to interpret of course.  I can see from peeringdb that AAISP are peered to the big locations Netflix are at, for instance, but of course it's virtually impossible to correlate that back to expected performance!  I did note that AAISP's total peering appears to be 50G and peeringdb put their traffic levels at 10-20G bracket.  Whereas Zen have 320G of peering and are in the 300-500G traffic bracket.  Does that mean that AAISP have over a 2:1 ratio between their requirement and their peering, yet Zen are closer to 1:1?  Or is this data just not reliably useful? 
It's interesting to see LightningFibre only appear to have 20G of peering and don't disclose their traffic levels (which begs the question how much use are the published traffic levels anyway if they're volunteered).

Anyway, all very interesting.  I'm going to try and figure out if I can see any pattern between where my traffic is going and which speedtest servers are particularly poor.  To be carried on in my own thread, sorry for the slightly off-topic :)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 16, 2022, 06:59:24 PM
Thanks, really interesting, though very hard to interpret of course.  I can see from peeringdb that AAISP are peered to the big locations Netflix are at, for instance, but of course it's virtually impossible to correlate that back to expected performance!  I did note that AAISP's total peering appears to be 50G and peeringdb put their traffic levels at 10-20G bracket.  Whereas Zen have 320G of peering and are in the 300-500G traffic bracket.  Does that mean that AAISP have over a 2:1 ratio between their requirement and their peering, yet Zen are closer to 1:1?  Or is this data just not reliably useful? 
It's interesting to see LightningFibre only appear to have 20G of peering and don't disclose their traffic levels (which begs the question how much use are the published traffic levels anyway if they're volunteered).

Anyway, all very interesting.  I'm going to try and figure out if I can see any pattern between where my traffic is going and which speedtest servers are particularly poor.  To be carried on in my own thread, sorry for the slightly off-topic :)

I guess it depends how often an ISP updates their PeeringDB profile as well as if the data they've input (e.g. traffic level) is reasonably accurate. AAISP aim to not be the bottleneck whenever possible, so it wouldn't surprise me if there is roughly a 2:1 ratio generally. For Zen to roughly be 1:1, given some performance issues in the past and perhaps even currently, doesn't shock me.

An ISP can choose not to disclose certain information, although it might be counter-productive because as I understand it many other networks tend to use PeeringDB to learn more about another network. PeeringDB recommend disclosing as much information on the profile as possible.

Cerberus Networks has a worse capacity at the internet exchanges. They have 1 x 10G (possibly a recent addition, I don't recall it seeing it before but maybe I missed it) and 2 x 1G at LINX. They also only have 2 x 1G at LONAP.

---

No worries.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Alex Atkin UK on July 16, 2022, 07:12:18 PM
I think the key piece of information here is "we aim to not be the bottleneck" does not mean "we will have 1:1 ratio".

All it really means is they will monitor peak traffic and adjust accordingly so that contention effectively doesn't happen.

After all, having a 1:1 ratio wont do you any good if the route to the most used destination can only be reached down half the peers. ;)

That said, you also wouldn't technically be "the bottleneck" if half your customers DID have to go a convoluted route via your indirect peers due to the direct ones being overloaded.  So there's an awful lot of wiggle room.  Though I trust AAISP not to do that, I couldn't say the same of Zen.

Ultimately the bigger the ISP, the higher the contention ratio is going to be.  As you don't need a 1:1 ratio to run an effectively uncontended network in the real world, due to customer usage patterns.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Weaver on July 16, 2022, 08:33:25 PM
A note about the sniffer info. Packet size ~4k looks insane; it suggests either a bug in the tool somewhere, or else itís seeing irrelevant LAN-internal traffic mucking the results up. Even though there are mentions of Ďjumbo packetsí this only means Ďbaby jumboí in the case of WANs, ie <2000 bytes or more likely less than 1600 bytes including IP (and eg PPP) headers. The nodes in the middle of the internet, middle box routers, and the ISPs and carriers canít currently AFAIK handle 4k IPv6 or IPv4 packets never mind 9k. 9k is confined to the LANs that have all of: jumbo-capable switches, and NICs and operating systems in hosts. Maybe there are jumbo packets in the LAN though, confined to the LAN, no reason why not, and theyíre getting in the way of the results. Could be your o/sís are clever, detecting 9k MTU and taking full advantage of it on the LAN. But your gateway to the internet shouldnít have 9k MTU as itís misleading, IP MTU can only be 1500 (for IP PDUs) and a little more for PPP+PPPoE (my PPP packets to the internet are 1530 bytes max)
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 17, 2022, 04:40:18 PM
Good news, at least today... tomorrow, who knows? :D

Earlier today I asked a family member to do a single connection speed test on speedtest.net and see what their numbers are (they are also on Lightning Fibre). To my amazement they came back to me with nearly 1 gigabit down. After hearing that I thought I'd give mine another try and again to my amazement it's now nearly a gigabit down too!

(https://www.speedtest.net/result/13418372069.png)

Whether it stays this way... well, that's another matter. I hope so. I'll still go ahead with A&A but perhaps I can bond the connections with my usage of BGP, as long as the server at the other end which does the eBGP session can handle that amount of traffic that is. I believe it can.

Their upload was nearer gigabit too while mine isn't quite at that level but I'm not too fussed about that. It's still tons better than I can get via Openreach and it's probably something my end causing a slightly slower upload anyway. I'm just happy that I'm seeing near gigabit download again at last. I've made sure it's not just isolated to that speed test, I'm also seeing similar results with a few other servers and the TBB speed test too.

As I'm curious, I'm going to also see if there's something noticeably different on packet sniffer. I'll edit this post again shortly to mention what I found.

EDIT: The packet sizes are pretty much the same as before, when I was having the slow download speed. No noticeable difference in the packet sniffer output. So, whoever or whatever did something to fix the issue... I don't know what they did at the moment. I hope it doesn't go back to being slow tomorrow though!
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 17, 2022, 07:09:25 PM
That is good news.  :)

But without knowing what was causing the slowness . . .  :-\
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 17, 2022, 08:24:31 PM
Yeah I would like to know as well what has changed, unless today is just a one off and it returns again tomorrow or at some point next week.

I'll send a reply to Lightning Fibre in the morning to let them know that the problem has gone, assuming the problem hasn't returned in the morning that is :fingers:. They might be able to give me some insight into what's changed too, if so I'll post about it here.

After some further thought... I'm not absolutely certain if I will go ahead with A&A. If and that's a big if, the Lightning Fibre connection remains reasonably consistent for at least a week from now then I may cancel the migration from Cerberus Networks to AAISP.

As I understand it I have a fourteen day cooling off period if I wish to cancel the migration without penalty, at which point I could then cease with Cerberus Networks instead and just pay for Lightning Fibre (well, the first four months are free and then £34/mo). It's fairly important to consider with the way various costs such as energy have sky rocketed and may sky rocket further. I've just recently gone back on a fixed tariff which works out less than the next predicted increase, which is perhaps a small gamble but I'm confident that the recent prediction is either slightly conservative or accurate. I feel that the £85/mo (to AAISP) would be more useful in helping to pay for energy usage and other possible future uncertainties regarding the 'cost of living' crisis. If there's still some issues next week then I will likely allow the migration to go ahead, assuming that I'm right regarding the cooling off period.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 18, 2022, 10:21:07 AM
It's still running superb as of this morning, from many places in western EU I was able to generally download files at around 900 megabits or higher consistently.

I've informed Lightning Fibre of the good news and will see what they say, if they know of anything that was changed on their network over the weekend that might've been the cause of this issue originally. I'll either edit this post or post a new reply if they come back to me with anything useful.

I'm still connected to their ONT via a 1 gigabit port but I won't change back to 10 gigabit in case it breaks things again. When they offer a faster speed in the future and if I decide to upgrade to it then I'll do so at that point. It's working now so I feel it's best not to touch it in case it breaks again :D.

Assuming I'm right in my previous post, regarding the fourteen day cooling off period on the migration, then if Lightning Fibre remains fine for this week then I'll contact AAISP next week and ask them to cancel the migration and order. The only reason I did the migration was because Lightning Fibre's download speed was massively disappointing but now it's superb again and will hopefully stay reasonably like that for quite some time.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 18, 2022, 04:36:32 PM
EDIT: For the curious, the ONT looks like this: https://www.adtran.com/solutions/by-segment/products/by-category/fiber-access/optical-network-terminals-ont/621.html

I see it has a standard gigabit ethernet port as well, but I'm uncertain if the service is configured to also work over that and whether it's even wise to consider seeing if anything's different on that port if it did work.

Back on July 15th you posted the above which tells me that, up to that date, you were connecting to the Adtran SDX621 via its 10Gbps port and had never tried the 1Gbps port.

Subsequent edit: The above statement is correct.

I'm still connected to their ONT via a 1 gigabit port but I won't change back to 10 gigabit in case it breaks things again. When they offer a faster speed in the future and if I decide to upgrade to it then I'll do so at that point. It's working now so I feel it's best not to touch it in case it breaks again :D.

Today, July 18th, you disclose that you are connecting to the Adtran SDX621 via its 1Gbps port.

I've checked back through the posts and can't find when the port swap occurred . . . it could be that my reading is of fault. However, I now have to say that in terms of scientific experimentation the results are suspect . . . as more than one variable appears to have been changed.

[Subsequent edit: My last three sentences, above, are incorrect and, so, are struck out. See the following posts.]
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 18, 2022, 04:50:34 PM
Back on July 15th you posted the above which tells me that, up to that date, you were connecting to the Adtran SDX621 via its 10Gbps port and had never tried the 1Gbps port.

Today, July 18th, you disclose that you are connecting to the Adtran SDX621 via its 1Gbps port.

I've checked back through the posts and can't find when the port swap occurred . . . it could be that my reading is of fault. However, I now have to say that in terms of scientific experimentation the results are suspect . . . as more than one variable appears to have been changed.  :-\

Sorry, let me clarify. I mean I'm connecting to a 1 gigabit capable port on my Mikrotik router. I'm still connecting to the 10 gigabit port on the ONT. Hope that clarifies!

I did try the 1 gigabit port on the ONT but it was unusable.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 18, 2022, 04:59:59 PM
I did try the 1 gigabit port on the ONT but it was unusable.

Ah, so it is my mistake. I'll go and correct my post . . .

Thank you for the clarification.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: Ixel on July 21, 2022, 10:28:07 AM
I've had a reply from Lightning Fibre from the person who originally did various network related tests and looked things over on their end. Unfortunately they aren't aware of any changes on their network which would've resulted in this problem being resolved, so sadly it remains a mystery as to what the cause was as well as what the solution was.

So, if things remain stable by Monday of next week then I will contact AAISP and see if I can cancel my migration of my Openreach FTTP service to them. If that's possible then I will ask Cerberus Networks to cease the FTTP service. Installation (migration) on AAISP is due to be completed on August 2nd. I hope I can cancel the migration if Lightning Fibre remains stable, otherwise it's another £85/mo for 12 months which could be put to better use in the current climate.
Title: Re: Diary of My Experiences with Lightning Fibre (FTTP local ISP)
Post by: burakkucat on July 21, 2022, 04:34:14 PM
All understood. You will, I'm sure, post updates here, as and when.