Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[bogof]

Its certainly a long-term problem they need to solve as I was under the impression Zen were planning to ultimately cover all head-ends with their own infrastructure.

Indeed, and in initial conversations Zen told me they wanted to fix it and not migrate me back as ultimately everything would migrate to their network where possible.  But they really seemed to draw a blank in terms of things to fix it.  There was a replacement of some SFP modules at our exchange in some bit of gear, but other than that I'm not aware of what else they really tried.

Even though you might consider we are heading into peak time, single thread TBB downloads and Zen speed tests are up at full line rate at the moment, and long may it continue:

root@Home-Dream-Machine-SE:/ssd1/test# rm 512MB.zip ; time axel -U Chrome -n 1 -v -a -o 512MB.zip http://ipv4.download.thinkbroadband.com:81/512MB.zip
Initializing download: http://ipv4.download.thinkbroadband.com:81/512MB.zip
File size: 536870912 bytes
Opening output file 512MB.zip
Starting download

[100%] [...] [ 101.0MB/s] [00:00]

Downloaded 512.0 Megabyte in 5 seconds. (103460.45 KB/s)

root@Home-Dream-Machine-SE:/mnt/data/speedtest# ./speedtest -s 40788

   Speedtest by Ookla

     Server: Zen Internet - London (id = 40788)
        ISP: Zen Internet Ltd
    Latency:     6.41 ms   (0.08 ms jitter)
   Download:   914.37 Mbps (data used: 893.9 MB )
     Upload:   110.20 Mbps (data used: 49.7 MB )
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/3ed9415c-b22d-4589-9c85-26c77a2f56bc

Like a kid in a sweet shop again, rather than the sack of disappointment they delivered with the migration.  Line rate here, line rate there, line rate every &&$**$ where!

[bogof]

This chart from the Samknows web interface I found interesting.  RTP jitter down on the BTW connection is much better than any of the GEA gateway connections I've seen over the last month, while overall latency is slightly worse.

My own theory is that this high jitter causes TCP throttling mechanisms to back of the throughput through single connections.

You could kind of see this high jitter on the TBB BQM chart at the very top of this saga, post migration the BQM got much "furrier".

I'm quite impressed with the breadth of the analytics the Samknows box does, I will quite miss it when it goes back.  It's a real shame this kind of proactive monitoring isn't built into more routers on the market.  It would be great if in the event of come change you had evidence the ISP could trust of the performance in key aspects before, during and after some event.  As it is getting the issue recognised was way harder than you'd hope, even though it was pretty clear to me it was as a result of.something they had just.done.

[Chrysalis]

Its not clear what we looking at here, are these graphs for BTw or Zen backhaul?

Do you have grabs for both to show for comparison?

Most congestion algorithms use packet loss instead of RTT, but  the ones that do possibly will back off if jitter is a bit random.  No real idea as I have never used such a network condition, all my high speed devices are on stable jitter free datacentre connections.  You can emulate latency on dummynet to help test, but I dont think you can emulate jitter.   So could maybe write a script that adjusts the artificial latency every 20ms or something to try and emulate jitter.

Bear in mind 1.5ms of jitter wont do much, I am assuming we talking much higher levels of jitter.

[Alex Atkin UK]

Could the jitter be out-of-order packets?  That would certainly slow down TCP.

Just pondering if fragmentation could happen on Zens network and get recombined before it reaches the customer so you wouldn't be able to see it?

[bogof]

Its not clear what we looking at here, are these graphs for BTw or Zen backhaul?

Do you have grabs for both to show for comparison?

These are grabs of both simultaneously.  I was on Zen GEA for the last two months until 31st August.  The last stepchange on the graphs (ie the right hand side) is BTW, everything to the left of that stepchange in the land-of-rather-random is Zen GEA.  The various stepchanges within are where you can see how different the gateways are to each other.  You can see quite different jitter between two different London gateways on GEA, for instance (before August 24th and from August 24th to August 30th).  On August 30th it switched to a Manchester gateway for good measure just to make it look really poor before the migration back...!  Though actually not so bad for jitter.

I don't know about the significance of the jitter, but it is interesting that the throughput was better on the Zen GEA in the section where the jitter was lower from 24th August onwards, and much better once the jitter became almost insignificant after the migration back on 31st.  Certainly not cause and effect proof, but interesting data.

[Alex Atkin UK]

Incidentally I'd say no, it probably wouldn't be great to have this in routers as the tests its performing can potentially be quite invasive, at least the regular speed tests.

Nobody wants a random speed test performing in the middle of watching Netflix or worse, making a VoIP call, video conference, working from home or gaming.  Plus it would be a huge waste of backhaul bandwidth, how would you schedule it so you don't suddenly have 100s or 1000s of customers all maxing out the lines at the same time?  You'd jam up the network quite quickly.

[bogof]

Incidentally I'd say no, it probably wouldn't be great to have this in routers as the tests its performing can potentially be quite invasive, at least the regular speed tests.

Nobody wants a random speed test performing in the middle of watching Netflix or worse, making a VoIP call, video conference, working from home or gaming.  Plus it would be a huge waste of backhaul bandwidth, how would you schedule it so you don't suddenly have 100s or 1000s of customers all maxing out the lines at the same time?  You'd jam up the network quite quickly.

The samknows setup does have solutions in place for both those issues.  The test running is triggered centrally, so they control how many tests are running simultaneously.  Otherwise the tests would be meaningless as they'd easily eat up all the test server capacity.  And when a test is requested centrally, the Samknows agent has the option to abort the test if it detects significant other traffic.  This seems to work well.  On my unit there is a long list of tests the device refused to run or aborted because of user activity.  Of course over time for watching long term trends it doesn't matter.

Plus the schedule could be extremely sparse, unlike the one deployed at the moment in my unit for following a reported performance issue.   My ubiquiti triggers a speedtest once per day.  You don't even need to do them that often.

In short, I don't think any of the reasons you mention are reasons not to do it, I think they're just things to be aware of (that it looks like the Samknows team have already considered).

I understand there are already some routers that may have built in the Samknows tech (though I think this may only be the Realspeed portion for doing on demand tests from the router instead of from the browser).

[Chrysalis]

Ok in my opinion the jitter in itself is not slowing things directly, neither the increase in base latency when it goes up.

There is something odd in Zen's configuration.  If I was them I wouldnt be throwing of concerns about the jitter internally though as thats an indicator something isnt quite right.  These kind of issues could happen with packet ordering problems (are they bonding interfaces?) but complete guesswork as an outsider.

Its concerning for me because when CF ever get their act together and activate my area, I feel like Zen would be where I would go (AAISP annoyed me recently with a response about outages, apparently its normal for outages to happen on a nightly basis for an always on connection), however this one issue is putting me off.  No idea if there is an option for CF FTTP users to not use Zen's backhaul.

[Alex Atkin UK]

Would CF not use CF backhaul?

[bogof]

Would CF not use CF backhaul?

I think they're handing off at least some stuff through BTW or other providers links or facilities, as they are running fiber in Norwich into the BTW exchange.

[andrew-AAISP]

Sorry for jumping in :-)

...
Its concerning for me because when CF ever get their act together and activate my area, I feel like Zen would be where I would go (AAISP annoyed me recently with a response about outages, apparently its normal for outages to happen on a nightly basis for an always on connection), however this one issue is putting me off.  No idea if there is an option for CF FTTP users to not use Zen's backhaul.

This is a bit of a concern, as repeated nightly outages would be very unusual - I can take a look if you like? PM me (either via forum or on our IRC) if you like...

As for Zen's network, I'd not be surprised if they connect to CityFibre at multiple local POPs around the country and then run that traffic over their own national network to reach their core POPs. We're smaller, so rather than having multiple links we have a connection to CityFibre in London and CityFibre give us access to their (English) network via that link.

[XGS_Is_On]

Would CF not use CF backhaul?

Most of Zen's CF stuff uses the 'old' CF solution where the provider rents colocation in the CF fibre exchange and places their own kit in there similar to Openreach GEA. They take care of their own backhaul which may or may not involve leasing dark fibre from CF for part of it but then uses the provider's own network. The national access product where CF aggregate FEX together and present across NNIs is a relatively new thing and uses a combination of CF's own network and IIRC either Neos or Zayo to link the CF metro networks.

Apologies for the acronym soup.

EDIT: Remembering the national product is England-only potentially indicates use of Zayo. The company formerly known as SSE Telecoms and now Neos have fibre up to Aberdeen and Inverness, Zayo don't.

[Alex Atkin UK]

Ah, so presumably that is the reason why AAISP over CF are England only?

[Moderator edited to insert the two words in green, above.]

[j0hn]

Ah, so presumably that is the reason why AAISP are England only?

Exactly that.
Cityfibres "national" product is England only, currently. I'm sure that will change over time.

AFAIK AAISP don't have anywhere "unbundled" and rely on other carriers to get across the country.

Cityfibre only launched their national access product fairly recently. Before that you needed your own kit in each of their FEX's that you wanted to offer services in.
That's why you have so many different ISP's available in different areas on Cityfibre.

[Alex Atkin UK]

It certainly will be interesting to see if Zen GEA on CF has the same issues as OR.  If it doesn't, that would be extremely curious and give them some leverage to get OR to look into it deeper?