Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: Network problems 12/08/14  (Read 9760 times)

guest

  • Guest
Network problems 12/08/14
« on: August 12, 2014, 06:49:29 PM »

Some of you may have noticed some issues today which began around 9am BST.

Not going into great detail but the BGP FIB count went over 512k this morning. That's like the forwarding table which routers use to pass packets as fast as possible.

The problem is that a lot of routers either need f/w (IOS really) upgrades (expensive with Cisco) or you need to reduce the routes you (express) forward for. If you're a small company then reduce the routes, no big deal.

In laymans terms this is like the millenium bug for an older generation of routers.

Now no big deal unless you're a major carrier who has got this wrong.

That's what happened today so ebay & a load of other stuff was unavailable to VM, Sky, Zen and others. Edit - none of whom had the issue on their edge routers, it was all upstream.

Phones lit up all over the place :D

BT didn't have similar issues. I'm sure kitz can work out who it wasn't from that :)
« Last Edit: August 12, 2014, 06:58:01 PM by rizla »
Logged

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 33752
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: Network problems 12/08/14
« Reply #1 on: August 12, 2014, 07:04:20 PM »

Interesting information. Thank you.  :)

Now I know why I could not access eBay from my TalkTalk connection, earlier this morning.  ::)
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

sevenlayermuddle

  • Helpful
  • Addicted Kitizen
  • *
  • Posts: 5347
Re: Network problems 12/08/14
« Reply #2 on: August 12, 2014, 09:18:11 PM »

I did notice some weirdness today, browsing certain sites and specifically forums like 'stackoverflow'.

From time to time, a specific domain would go AWOL, with Safari claiming 'server not available' or similar, affecting multiple web pages of that domain, while the rest of the web was fine.   The way to get access back seemed to a full reset of Safari browser which might (not sure) force a fresh DNS lookup?   All would then be well for a while until, maybe an hour later, it all repeated.

I'd been blaming Demon, on the basis I always blame Demon.   Maybe I owe apologies? . :-[

Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32991
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Network problems 12/08/14
« Reply #3 on: August 12, 2014, 09:31:13 PM »

Yes I received notification from my hosts that parts of the Internet were unavailable for many UK ISPs yesterday & today.



Quote
The global routing table is a Ďmapí of each possible destination on the internet.  Every large network operator (such as ourselves, or the ISP you use to connect to the internet) holds a copy of this, or multiple copies in our case.  This is what enables every computer on the internet to reach every other computer.
 
Over the past two decades the routing table has been increasing in size, due to new ipv4 addresses being used and existing ipv4 address ranges being split (meaning that 2 consecutive ranges might have different paths).  Today it hit 512,000 routes.  This is a magic number as itís an inbuilt limit in many common routers and switches.
 
We had pre-empted this.  Most of our routers/switches already have a higher limit and we have recently spent £250,000 on network upgrades to improve the rest of our network. 


I believe some websites were affected, but afaik we werent... nor did I have any problems from Plusnet...  and a quick scan of the PN forums seems all is quiet. 
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32991
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Network problems 12/08/14
« Reply #4 on: August 12, 2014, 09:37:34 PM »

I find it strange that I cant find any media news about this.  You'd think it would be something that at least el reg would pick up on?
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

loonylion

  • Reg Member
  • ***
  • Posts: 723
Re: Network problems 12/08/14
« Reply #5 on: August 12, 2014, 10:20:41 PM »

Been on the net all day and haven't seen any evidence of this. BA and american airlines websites are crap and intermittently don't work, but nothing to do with routing tables.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32991
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Network problems 12/08/14
« Reply #6 on: August 13, 2014, 12:23:12 AM »

Its only just beginning to hit the media.  ZDnet has just published this

http://www.zdnet.com/internet-hiccups-today-youre-not-alone-heres-why-7000032566/
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

guest

  • Guest
Re: Network problems 12/08/14
« Reply #7 on: August 13, 2014, 02:44:10 PM »

I've found some of the coverage wonderfully amusing - like the muppet quoted on El Reg :

Bearer asks a critical question: "Why wasn't this in the press, like Y2K or IPv4?"

Ermmm it was. Repeatedly. People have been banging on about it for sodding YEARS & there's been dozens of warnings this year alone.

(If he means the mainstream media then he needs to get a clue very rapidly or get out of the tech business for good)

Some of you know I play Eve Online - they load balance between upstream suppliers (with some dreadful s/w but lets not go there) & yesterday about 30 seconds after the BGP FIB update 40% of their bandwidth decided to start flapping. After an hour of assuming someone there had a clue (dangerous I know) I decided to contact them* so they knew why this was happening. Four hours later they manage to get temporary routing & b/w in place - today they're changing h/w :D

If I hadn't been playing I probably wouldn't have noticed. I don't have access to an AS anymore but I could see routes flapping easy enough.

One of the other guys in my Eve corp lives in NZ & designs/builds VoIP systems, another is a chief network engineer with Telstra so we had a chat on TS instead of killing imaginary Icelandic spaceships :P One could connect, the other couldn't & after a brief derailment when someone said a router was dead at Telecity Meridian I twigged it after maybe 20 minutes. Still nothing on nanog or other regional mailing lists at this point.

Was very funny watching the carrier companies who employ monkeys falling over in a heap with routers flapping/crashing/dumping the entire routing table   >:D

*not customer service, one of the guys who deals with Tranquility (http://www.pcgamer.com/2013/06/15/eve-online/ or much older https://wiki.eveonline.com/en/wiki/Tranquility)  in London.
« Last Edit: August 13, 2014, 02:55:43 PM by rizla »
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32991
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Network problems 12/08/14
« Reply #8 on: August 13, 2014, 07:13:39 PM »

Its been interesting to observe and you really can imagine lots of Network Admins around the globe flapping like crazy.  I think a lot of them were aware it was going to happen... just not quite so soon.   

My webhosts held their hands up, and although they had just made a large investment in Juniper routers anticipating this was going to occur, they hadnt got around to installing them all and they still had a couple of Cisco routers in use.  Their reaction 'sorry for taking so long to find out the cause of the problem'.. ie 30mins.   All affected sites were fully back up and running within an hour and certainly by lunch time.   At which point everyone else was still struggling to find out the problem, because no one was talking to anyone else.   Still nothing from the ISPs and mainstream media.

From my understanding, there is a quick and dirty software adjustment that can be made on the Cisco routers, and although its a fix it is only temporary..  as those routers will now be working extra hard and we could continue to see increased latency along certain routes for some time to come.
A couple of my 'friends' cat related sites still broken or exceedingly slow.


I am still really amazed that theres nothing about this from the likes of BBC/Sky/CNN news.   You'd think they be jumping on an article saying "The Internet is Broken". 
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 40641
  • Penguins CAN fly
    • DSLstats
Re: Network problems 12/08/14
« Reply #9 on: August 13, 2014, 08:22:55 PM »

This sounds rather like the IPv4 issue - we've known about it for years, but hardly any ISPs seem to be taking it seriously yet.
Logged
  Eric

guest

  • Guest
Re: Network problems 12/08/14
« Reply #10 on: August 13, 2014, 08:34:13 PM »

You can (fairly easily) drop some routes you have minimal traffic for but it sort of defeats the purpose of an autonomous system if you potentially have to intervene with BGP updates.

Not an option for carriers.

Quite the contrary Roseway, its the carriers rather than the ISPs. There are a few carriers who are going to be paying out 9 figure numbers in compensation to some high profile clients (the triple 9 peeps).

As me & the other guy greeted each other on Eve this morning :

"How do you all like cloud computing now?"  :lol:
« Last Edit: August 13, 2014, 08:42:53 PM by rizla »
Logged

guest

  • Guest
Re: Network problems 12/08/14
« Reply #11 on: August 13, 2014, 08:41:42 PM »

Its been interesting to observe and you really can imagine lots of Network Admins around the globe flapping like crazy.  I think a lot of them were aware it was going to happen... just not quite so soon.

Not really kitz - there's been heads-up emails/promos/training from every (non-generic) manufacturer for all of this year at least.

Everyone knew it was going to happen in August or early Sept from May this year.

Beancounters are the problem & they can pay the breaches of service agreements  >:D

Beancounters on bonuses - what could possibly go wrong  ::)
Logged

broadstairs

  • Kitizen
  • ****
  • Posts: 3523
Re: Network problems 12/08/14
« Reply #12 on: August 14, 2014, 08:02:28 AM »

Just wondering if this problem might be the cause of my wife's laptop being unable to access the Microsoft update service to download the latest W8.1 updates. I've tried a few times and it just sits there without downloading anything. I am on TT btw.

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:Zyxel VMG3925

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 33752
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: Network problems 12/08/14
« Reply #13 on: August 14, 2014, 05:17:47 PM »

Yes, that could be the reason Stuart. That is exactly the type of response I saw when attempting to access eBay from my TT connection.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

loonylion

  • Reg Member
  • ***
  • Posts: 723
Re: Network problems 12/08/14
« Reply #14 on: August 14, 2014, 05:54:53 PM »

http://www.bbc.co.uk/news/technology-28786954

not quite what you were hoping for, but the BBC have caught on.
Logged
Pages: [1] 2