Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: Many ISPs outages  (Read 4668 times)

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Many ISPs outages
« on: March 30, 2011, 06:33:36 PM »

Service: Broadband
Posted: Wed, Mar 30 2011 at 18:15:58
Subject: Broadband Connectivity Problems (66446) - NEW

At approximately 5.30pm a significant number of our broadband customers were disconnected from the Internet and many appear to be struggling to get back online.

This appears to be the result of a problem affecting parts of BT Wholesale's network and will be affecting other ISP's across the country.

Our network engineers are investigating and further details will be provided as soon as they become available.

Please accept our apologies for the inconvenience.

I know of problems on AAISP and PlusNet services off the THCN Cranleigh exchange starting at about 17:35.
Zen services remain working though.

Kind regards,
Walter
Logged

Azzaka

  • Reg Member
  • ***
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: Many ISPs outages
« Reply #1 on: March 30, 2011, 07:19:19 PM »

We are currently aware of an outage on our Slough Node. We have dropped quite a users and they are unable to reconnect at this time.

BTW have told us an engineer is enroute. Next update will be at 7:30 PM


Leo
Logged
I Sync', I Auth', therefore I am.
Online

Azzaka

  • Reg Member
  • ***
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: Many ISPs outages
« Reply #2 on: March 30, 2011, 07:58:44 PM »

We are currently aware of an outage on our Slough Node. We have dropped quite a users and they are unable to reconnect at this time.

BTW have told us the engineer is still working on this issue. We have been advised there is no ETA for completion of the work. As such please see the service alert for further information afetr 8pm.

Status Alert: http://status.zensupport.co.uk/index.php?serviceid=5...

Leo
Logged
I Sync', I Auth', therefore I am.
Online

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #3 on: March 30, 2011, 08:11:22 PM »

Here's PlusNet's notice. It will be interesting to see how long it takes to repair a fibre cable.

Kind regards,
Walter

Service: Broadband
Posted: Wed, Mar 30 2011 at 19:58:58
Subject: Broadband Connectivity Problems (66446) - UPDATE

This is an update to this evening's post about a major service outage that is currently affecting parts of BT Wholesale's network.

The problem surfaced at around 5.30pm this evening at which point a significant number of our broadband customers were disconnected from the Internet. At the time of writing these customers are still unable to get back online.

BT have attributed the problem to a fibre cut in the Slough area, however please be mindful that the loss of connectivity is being experienced by customers up and down the country. It is also affecting other ISPs as well as Plusnet.

We will continue to monitor the situation for updates and will post further information as soon as it becomes available.

The next Service Status update is due by 10.00pm this evening.
Logged

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #4 on: March 31, 2011, 06:57:26 AM »

Chaos still reigns. Below are two more reports. In Ewhurst the outages were on 20CN services as only Zen have been partially upgraded here.
Thankfully the services have been restored at some stage early this morning.
One might have hoped that all major services would have dual diverse routed links but perhaps this is not the case?

Service: Broadband
Posted: Thu, Mar 31 2011 at 00:25:27
Subject: Broadband Connectivity Problems (66446) - UPDATE

This is an update to this evening's posts about the BT Wholesale service outage that has left a significant number of our broadband customers without Internet connectivity.

We still have a substantial number of broadband subscribers who were disconnected around 5.30pm this evening and who are still unable to get back online.

Some ambiguity has emerged as to the true cause of the problem, which may not actually be due to a fibre break in Slough as previously reported.

One thing we are confident about though, is the fact that the problem is limited to a subset of customers provisioned on BT's IPStream Connect or WBC wholesale networks.

As we have recently moved a large number of subscribers across to these networks, this unfortunately amounts to quite a large volume of our customers.

BT are continuing to work on the problem and have advised that their next update will be circulated by 1.30am.

There is still no estimated fix time.

Service: Broadband
Posted: Thu, Mar 31 2011 at 03:40:20
Subject: Broadband Connectivity Problems (66446) - UPDATE

This is an update regarding the ongoing problems with BT wholesales service outage which continues to leave a large amount of our Broadband customers without Internet.

As earlier, we still have a number of subscribers whom, after losing connectivity at around 5.30pm yesterday evening, are regretfully still unable to connect.

Across the BT Wholesale network, it has been reported that a few lines are slowly coming back into connection, though we are still monitoring this. We have noticed that on our Graphing system, some lines seem to be connecting for a short amount of time before losing connectivity again.

Unfortunately we still do not have a estimated fix time, however as above, we hope that the amount of connections that we are seeing slowly regaining their service will continue to increase throughout the night.
« Last Edit: March 31, 2011, 07:05:53 AM by waltergmw »
Logged

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #5 on: March 31, 2011, 08:47:16 AM »

It seems as if the problem is fixed. I note that a Cisco Router at Slough was replaced. However it would be useful to have a little more clarity as to what the problem was.

Kind regards,
Walter

Service: Broadband
Posted: Thu, Mar 31 2011 at 08:20:20
Subject: Broadband Connectivity Problems (66446) - UPDATE

This is an update to last night's and this morning's posts about the BT Wholesale service outage that left a significant number of our broadband customers without Internet connectivity.

At approximately 3.15am this morning we saw a number of customers' circuits come back online. There was a further connectivity blip shortly after this, however that was short-lived and a lot of customers have been online since about 3.30am.

Those that were still unable to connect seem to have had their service restored from around 6.30am this morning.

BT have forwarded us confirmation that they now believe the problem to be fixed.

If you're still having difficulty connecting, then please try powering down your modem/router for 20 minutes before powering it back up and attempting to re-establish a connection.
Logged

Azzaka

  • Reg Member
  • ***
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: Many ISPs outages
« Reply #6 on: March 31, 2011, 11:55:54 AM »

We still have some customer offline and unable to connect.

The Cisco issue would ring true as some customer could reconnect when using 992.3 and not 992.5.

Leo
Logged
I Sync', I Auth', therefore I am.
Online

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #7 on: March 31, 2011, 12:22:06 PM »

Service: Broadband
Posted: Thu, Mar 31 2011 at 12:01:13
Subject: Broadband Connectivity Problems (66446) - RESOLVED

This is an update to this morning's posts about the BT Wholesale service outage that left a significant number of our broadband customers without Internet connectivity.

BT have now fixed the problem, so customers who lost connectivity should now be able to get back online.

If you're still having problems then we suggest you try powering down your modem/router for 20 minutes before powering it back up and attempting to re-establish a connection.

We have requested a detailed analysis from BT as to the events that led to the problem and the subsequent actions that were taken to fix it.

EDIT see also:-

http://www.thinkbroadband.com/news/i/4629.html

« Last Edit: March 31, 2011, 12:25:19 PM by waltergmw »
Logged

razpag

  • Reg Member
  • ***
  • Posts: 622
Re: Many ISPs outages
« Reply #8 on: March 31, 2011, 04:30:07 PM »

I get the feeling your disappointed the Fibre Cable was fixed so quickly Walter.  ;)

Just because BT/OR are a commercial giant, doesn't mean we all sit around with out feet up muttering "We'll get round to mending it later". Maybe in the old monopoly days of the GPO, but certainly not in present times.  ;D
Logged

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #9 on: March 31, 2011, 05:46:09 PM »

@ RP,

Not in the least ! I genuinely wanted to know how long it takes to repair a fibre cable - if indeed that was part of the problem as well as the Cisco Router.

I am quite well aware that you people are always hurtling about - sometimes because of your micro-managed existence.
One of your colleagues on loan here was tasked to install two lines in the dark at the end of November after he arrived at 15:55. Surprise surprise he didn't complete the task because nobody now seems to do any job estimation beforehand. The two lines were eventually completed in February (which did include two doses of bad weather). Another horrible broadband fault took 5 engineer visits before the last chap (VERY WELL DONE Matt from Haslemere) did some attritional engineering by swapping the pair over the last known very noisy overhead cable back through two DP joints.

You guys are mostly splendid chaps - it's your masters and bean counters that need educating !

Now I've lit the blue touchpaper !!!!

Kindest regards,
Walter
Logged

razpag

  • Reg Member
  • ***
  • Posts: 622
Re: Many ISPs outages
« Reply #10 on: March 31, 2011, 07:36:36 PM »

 ;D ;D ;D

I thank you for your continued support towards the engineering fraternity Walter. I couldn't possibly comment on your thoughts regarding leadership and accounting.  ;) ;) ;D
Logged

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #11 on: March 31, 2011, 11:48:52 PM »

Back to more serious matters again. Apparently PlusNet clients are still experiencing a significant number of problems particularly on 20 CN services. I am very surprised that it seems necessary to follow the procedure described below.
The two failures I'd observed in Ewhurst (One AAISP and the other Plusnet, both using 2Wire 2700HGV routers on BT Wholesale's 'Up to 8Mbps' IPStream Connect service recovered around 03:35 this morning and have remained stable since then. These both are routed through the Reading RAS.)

Service: Broadband
Posted: Thu, Mar 31 2011 at 21:40:31
Subject: Problems connecting following Major BT Service Outage IMT10397/11(66459) - NEW

This evening we've been receiving a lot of calls from customers who are struggling to get back online following the major BT service outage that was experienced early this morning/late last night.

Whilst we typically expect a certain degree of 'fall out' from a widespread problem like this, the volume of enquiries we're receiving is way beyond what we're used to. This is leading to increased wait times for customers trying to get in touch with us.

Those continuing to experience problems mainly seem to be provisioned on BT Wholesale's 'Up to 8Mbps' IPStream Connect service and we're investigating why this might be.

If you believe you're affected by this problem then the first thing you should do is power off your modem/router for 60 minutes before powering it back up and attempting to reconnect.

If that doesn't work then we're finding that most customers can get back online by accessing their router/modem settings and trying to manually reconnect from there.

This is how to do this using the Thomson 585v7/585v8 routers we supply:

- Open a web browser such as Firefox or Internet Explorer
- Browse to http://192.168.1.254
- If you are prompted for a password, enter either:

username: Administrator
password: <leave blank>

or:

username: admin
password: <the serial number form the base of the router> (minus the characters in brackets at the end)

(Please note that both the username and password are case sensitive)

- Under 'Broadband Connection', click the 'Internet' hyperlink
- Make sure your username and password are entered correctly and click 'Connect'
Logged

waltergmw

  • Content Team
  • Kitizen
  • *
  • Posts: 2772
Re: Many ISPs outages
« Reply #12 on: April 02, 2011, 09:39:45 AM »

The above message has now been repeated twice except that 65 minutes has become "just over an hour".
(There must be some significance in this time interval? Any ideas Leo ?)

Plusnet then say if that fails use the 585 V7 & V8 web interface to restart the connection.
I'd be unhappy at haveing to wait an hour so I'd probably try the restart procedure immediately.

Kind regards,
Walter
Logged

Azzaka

  • Reg Member
  • ***
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: Many ISPs outages
« Reply #13 on: April 04, 2011, 09:40:54 AM »

The above message has now been repeated twice except that 65 minutes has become "just over an hour".
(There must be some significance in this time interval? Any ideas Leo ?)

Plusnet then say if that fails use the 585 V7 & V8 web interface to restart the connection.
I'd be unhappy at haveing to wait an hour so I'd probably try the restart procedure immediately.

Kind regards,
Walter

We saw the same Walter,

We have asked our customers to retype the details and this gets them back online. Why? No idea at this point. I know we are investigating.

Leo
Logged
I Sync', I Auth', therefore I am.
Online

jeffbb

  • Content Team
  • Kitizen
  • *
  • Posts: 2329
Re: Many ISPs outages
« Reply #14 on: April 04, 2011, 02:03:00 PM »

Hi

Just a thought not all those disconnected have access ? so how does the info get to them ? :)

Regards Jeff
Logged
zen user
Pages: [1] 2
 

anything