Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Author Topic: Massive Demon outage?  (Read 12445 times)

sevenlayermuddle

  • Helpful
  • Addicted Kitizen
  • *
  • Posts: 5369
Massive Demon outage?
« on: September 30, 2009, 08:57:29 PM »

Demon seem to be reaching for a new level of customer service today, with an outage that lasted for almost 8 hours, from 1pm til nearly 9pm.  I've no idea whether it affected all of Demon's customer, but their helpline was jammed solid all day.  When you selected the 'technical support' menu option it rang 5 times then cut you off, without even entering the queuing system, so it's fair to assume they had a widespread problem.

I love the following quotes from their service status pages, which appeared four hours or so into the outage...

Quote
17:16 Sep 30   hagueba   Some Broadband customers may currently be unable to connect. One of our Home Gateways has been rebooted in an attempt to restore service, however this was unsuccessful. Investigations are continuing into the root cause of this fault including assessment of further actions. We regret any inconvenience caused.
16:35 Sep 30   hagueba   Some Broadband customers may currently be unable to connect. One of our Home Gateways has been rebooted in an attempt to resolve suspected memory issues. There may be residual congestion as a result of this action and investigations are continuing into the root cause of this fault. We regret any inconvenience caused.

... in other words, "we tried switching it off on an again, and have no idea what to do next".  Sounds like even their own IT dept has to use the Indian help desk when problems need resolved?

7LM, not impressed  >:(

« Last Edit: September 30, 2009, 09:04:49 PM by sevenlayermuddle »
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Massive Demon outage?
« Reply #1 on: September 30, 2009, 11:01:19 PM »

>> we tried switching it off on an again, and have no idea what to do next

My thoughts exactly when I started to read the SS report.
Reboot..  oh that didnt work.. lets wait another hour and try again.  :'(

I know they use C&W for LLU for some accounts..  but if it caused a complete outage for so many.. it does make you wonder how many gateways they have!
Not good either coming so soon after this.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

sevenlayermuddle

  • Helpful
  • Addicted Kitizen
  • *
  • Posts: 5369
Re: Massive Demon outage?
« Reply #2 on: September 30, 2009, 11:29:58 PM »


Not good either coming so soon after this.

Crikey, I hadn't seen that, thanks for posting it. :o

I don't think I ever signed up to paperless billing as I've already had trouble with them setting up a continuous authority on my credit card the minute they got the number, so the only way I'll do business with them nowadays is payment by cheque through the post.
Logged

waltergmw

  • Kitizen
  • ****
  • Posts: 2776
Re: Massive Demon outage?
« Reply #3 on: October 01, 2009, 01:08:11 AM »

It seems there were a lot of gremlins about today.
The Virgin Media website and mail system was down from around 09:00 to about lunchtime today in Guildford at least.
Virgin did manage to put an appology page up instead of their web site.
The actual broadband service wasn't affected.

Kind regards,
Walter
Logged

HPsauce

  • Helpful
  • Kitizen
  • *
  • Posts: 2606
Re: Massive Demon outage?
« Reply #4 on: October 01, 2009, 08:21:52 AM »

Bethere had a big capacity problem too, due to a fibre break somewhere.
Maybe it was that that "killed" Demon?
Logged

pintosal

  • Reg Member
  • ***
  • Posts: 181
Re: Massive Demon outage?
« Reply #5 on: October 01, 2009, 08:50:08 AM »

Demon was my first broadband ISP around 10 years ago.
They were very good then, but now it sounds like they've joined the rest of the mediocrity.
Logged

waltergmw

  • Kitizen
  • ****
  • Posts: 2776
Re: Massive Demon outage?
« Reply #6 on: October 01, 2009, 09:19:15 AM »

See:-
https://www.blogger.com/comment.g?blogID=12990991&postID=1133301310748099069

which gives AAISP's take on the Be situation:-

AAISP said...

    Some more info and updates:

    A Global Crossing fibre issue has taken down eight 10Gb links on the O2/BE broadband core.

    This has been confirmed as a fibre break in the Leeds Huddersfield area. Engineers are currently on site re-splicing.

    (we're not currently seeing high latency, but BE expect it to kick in by mid afternoon if this issue hasn't been resolved by then.)

    The issue was raised at 06:25 today, no ETA on a fix has been issued yet.

    11:24 AM, September 30, 2009
Blogger AAISP said...

    Fibres have been re-spliced and this issue has been closed.

    2:13 PM, September 30, 2009
Logged

broadstairs

  • Kitizen
  • ****
  • Posts: 3697
Re: Massive Demon outage?
« Reply #7 on: October 01, 2009, 10:11:24 AM »

Glad I jumped ship a year ago, had been with Demon since we first got the web but they became too unreliable, too unresponsive and vastly too expensive. I now get BB and phone with unlimited land line calls world wide for £4 a month less than just my Demon BB.
Logged
ISP:Vodafone Router:Vodafone Wi-Fi hub FTTP

sevenlayermuddle

  • Helpful
  • Addicted Kitizen
  • *
  • Posts: 5369
Re: Massive Demon outage?
« Reply #8 on: October 01, 2009, 10:28:22 AM »

Couldn't agree more about how good Demon were ten years ago, but then they were a small privately owned company, and reputed to be 'the techies' ISP', as they seemed to pitch at people with good technical skills and accrodingly employed knowledgeable support staff.  However, the company's been sold/merged/taken over several times and there really seems to be no remnant of the original ISP left.

Leaving Demon, unfortunately, is a big upheaval as we run a popular (non-commercial) website, which orignally had a demon domain name.  We changed to a .com domain name seven or eight years ago, and the website's no longer even hosted by demon, but there's still an awful lot of links to that old demon site, which just redirects to the new .com.  And, of course, there's still the occasional (valuable) email that arrives addressed to the old demon email addresses.
Logged

GrumblingAgain

  • Just arrived
  • *
  • Posts: 2
Re: Massive Demon outage?
« Reply #9 on: October 01, 2009, 02:24:12 PM »

Hi

I was a customer badly affected by their outage yesterday and I was most unimpressed with their service (lack of). I was surprised by their service statement which said they rebooted and it didn't work. Scratch head what do we do now?

The reason for this post as I am no expert in these things, but I wonder if anyone would know from looking at this traceroute if it shows the actual cause of the problem or is it merely a symptom of some other problem?

I was tracing from my home (on VirginMedia) into our office where our office (on Demon) ....

  5    28 ms     9 ms    11 ms  bre-bb-b-as5-0.network.virginmedia.net [213.105.172.234]
  6    12 ms    11 ms    11 ms  telc-ic-1-as0-0.network.virginmedia.net [62.253.185.74]
  7    81 ms    16 ms    19 ms  ldn-b2-link.telia.net [213.248.100.97]
  8    16 ms    16 ms    17 ms  ldn-bb2-link.telia.net [80.91.249.177]
  9    17 ms    18 ms    17 ms  ldn-b4-link.telia.net [80.91.250.237]
 10    13 ms    11 ms    10 ms  thus-106917-ldn-b4.c.telia.net [213.248.100.162]
 11    11 ms    12 ms    18 ms  anchor-inside-4-g1-0-5.router.demon.net [193.195.25.65]
 12    15 ms    11 ms    12 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 13    11 ms    10 ms    11 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 14    16 ms    13 ms    13 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 15    10 ms    12 ms    12 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 16    13 ms    13 ms    12 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 17    10 ms    16 ms     9 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 18    17 ms    13 ms    12 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 19    12 ms    11 ms    16 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 20    43 ms     *       11 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 21    13 ms    11 ms    11 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 22    13 ms    11 ms    12 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 23    10 ms    17 ms    10 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 24    11 ms    12 ms    13 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 25    11 ms    11 ms    11 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 26    15 ms    12 ms    13 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 27    13 ms    11 ms    10 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 28    12 ms    11 ms    12 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]
 29    11 ms    10 ms    10 ms  anchor-inside-3-g5-0-1.router.demon.net [194.159.161.66]
 30    10 ms    15 ms    13 ms  anchor-access-3-g1-1.router.demon.net [194.159.161.65]

Trace complete.

Does the fact that it is clearly looping between  [194.159.161.66] and [194.159.161.65] indicate that one of those points was wrongly configured? or is that simply what happens when an onward link fails? I could see straight away where the problem might lie, I just wonder why it would take 8 hours to sort out?

Yes it was a double headache for me yesterday what with Virgin Media falling over the day before!

Thanks
Regards
GA
Logged

sevenlayermuddle

  • Helpful
  • Addicted Kitizen
  • *
  • Posts: 5369
Re: Massive Demon outage?
« Reply #10 on: October 01, 2009, 09:27:43 PM »

Hi there, and welcome to the forum!  And I must say, I'm relieved to find evidence that I didn't imagine the whole thing   :)

I'm afraid I'm no expert on IP routing, but my gut feeling is that you were definitely onto something.  From googling, I found this page which seems to suggest what the problem might have been (search the page for the word 'looping').

http://www.exit109.com/~jeremy/news/providers/traceroute.html#weirdness

Maybe the forum might have some IP gurus that would be able to pass further comment, and shed any light on whether it was reasonable that Demon took 8 hours to resolve it...?

7LM
Logged

waltergmw

  • Kitizen
  • ****
  • Posts: 2776
Re: Massive Demon outage?
« Reply #11 on: October 02, 2009, 08:34:46 AM »

Hi Everybody,

You might find some interesting reading on this subject in the article called "Facebook and BT and Virgin" at:-

http://5tth.blogspot.com/

Kind regards,
Walter
« Last Edit: October 02, 2009, 10:02:44 AM by waltergmw »
Logged

GrumblingAgain

  • Just arrived
  • *
  • Posts: 2
Re: Massive Demon outage?
« Reply #12 on: October 02, 2009, 08:50:12 AM »

Hi there, and welcome to the forum!  And I must say, I'm relieved to find evidence that I didn't imagine the whole thing   :)

I'm afraid I'm no expert on IP routing, but my gut feeling is that you were definitely onto something.  From googling, I found this page which seems to suggest what the problem might have been (search the page for the word 'looping').

http://www.exit109.com/~jeremy/news/providers/traceroute.html#weirdness

Maybe the forum might have some IP gurus that would be able to pass further comment, and shed any light on whether it was reasonable that Demon took 8 hours to resolve it...?

7LM

Hi

Thanks for that link. I can see now that my traceroute was just showing a common sympton of an onward route failing.  :blush: Interesting read though.

Cheers
GA
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Massive Demon outage?
« Reply #13 on: October 02, 2009, 10:39:45 AM »

>> I can see now that my traceroute was just showing a common sympton of an onward route failing.

Most likely.

Last week there was a total outage at Rapidswitch which took out access to many UK websites.  Although I dont use rapidswitch, my hosts use their services for DNS. Luckily for me my hosts had some resiliancy and were able to switch the routing...  but this is a tracert I saw just as they were switching to their backup DNS servers.

Quote
Tracing route to kitz.co.uk [91.198.165.131]
over a maximum of 30 hops:
 
  1    22 ms    99 ms    99 ms  Router [192.168.1.254]
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *       22 ms  10.1.2.158
  5    13 ms    13 ms    13 ms  hex89-linx.blueconnex.net [195.66.226.223]
  6    16 ms    16 ms    15 ms  euroconnex-1.br1.core.misp.co.uk [217.112.86.107
]
  7    15 ms    14 ms    15 ms  ge-1-377.captain.core.misp.co.uk [91.198.165.5]
 
  8     *       14 ms     *     thermalnine.footholds.net [91.198.165.131]
  9     *       14 ms     *     thermalnine.footholds.net [91.198.165.131]
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12    15 ms     *        *     thermalnine.footholds.net [91.198.165.131]
 13    14 ms     *        *     thermalnine.footholds.net [91.198.165.131]
 14    14 ms     *        *     thermalnine.footholds.net [91.198.165.131]
 15     *        *        *     Request timed out.
 16     *       14 ms    13 ms  thermalnine.footholds.net [91.198.165.131]

Notice the looping at hop 8 onwards.

------------

Another instance I recall with something looping.. is back in the very early days of adsl and getting my first router. 
The router interface for port forwarding wasn't at all user friendly and I couldnt find anything anywhere on how to do it for that particular router.
I'd been trying to set up an ftp server and it wasnt just a case of setting up one rule forwarding port 21 to 192.x.x.x ..  anyhow I thought I managed to get something working..  but it wasnt until later that I realised that although Id sucessfully managed to forward port 21...  somehow (dont ask how or what) I'd managed to set something up that sent all non FTP traffic directed at me from an outside source into a loop.

It was months later that someone mentioned that if you tried to ping my router, you'd see it looping for a couple of hops at itself.  I'm assuming that it was bouncing ICMP between the WAN and LAN but not getting any further because nothing had been specifically designated for that protocol.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: Massive Demon outage?
« Reply #14 on: October 05, 2009, 01:43:51 PM »

A long-time Demon user myself, I noticed that the Demon network (or BT-land-to-Demon) was broken between around 1pm Wed 30 Sep onwards and into the evening.

At two different sites, using ping and traceroute I could ping the first hop beyond my own router, but could get no further into the network. They fixed it by next morning.

With the exception of their DNS servers, which are another story, I've found Demon to be pretty reliable in terms of IP connectivity. I don't use their email or other services, so I couldn't say about that, and their performance of their DNS servers is highly questionable, but aside from that I in fact can't remember the last time I had an IP connectivity outage, as I am being careful to exclude BT-related and phoneline-related problems that are down to BT Wholesale.

I don't work for Demon :-) and would not pretend that they don't have their weaknesses. But I am a long time multi-site user, both of their home office and business grade ADSL services.
Logged
 

anything