Kitz Forum

Announcements => Site & Forum Discussion => Topic started by: kitz on September 15, 2009, 04:25:30 PM

Title: Site Down - 15th Sept 09
Post by: kitz on September 15, 2009, 04:25:30 PM
I'm aware that theres been problems with the site today.  :'(

At the moment I'm not sure exactly what the problem was other than it was the server.
I'll post more info when I get it.

Title: Re: Site Down - 15th Sept 09
Post by: roseway on September 15, 2009, 04:39:55 PM
It's good to be back. :)
Title: Re: Site Down - 15th Sept 09
Post by: waltergmw on September 15, 2009, 04:51:42 PM
Hi Kitz,

It's been slow on a few occasions recently. This is what my traceroute looks like.

 3  glfd-t3core-1a-ge-300-1636.network.virginmedia.net (62.254.207.85)  11.870 ms  10.055 ms  11.916 ms
 4  gfd-bb-a-ge-200-0.network.virginmedia.net (213.105.175.85)  156.189 ms  10.990 ms  15.074 ms
 5  bre-bb-b-ae2-0.network.virginmedia.net (212.43.163.90)  10.605 ms  12.646 ms  11.931 ms
 6  bre-bb-a-ae0-0.network.virginmedia.net (213.105.174.225)  29.963 ms  11.449 ms  11.897 ms
 7  tengigabitethernet8-3.ar4.ams2.gblx.net (64.213.176.61)  130.711 ms  107.658 ms  58.398 ms
 8  64.214.140.138 (64.214.140.138)  24.600 ms  27.278 ms  24.136 ms
 9  64.214.140.138 (64.214.140.138)  23.802 ms !X * *
10  * * *


Note I could still  access other parts of the site - it was only the forum that went into sulk mode.

Kind regards,
Walter
Title: Re: Site Down - 15th Sept 09
Post by: oldfogy on September 15, 2009, 04:52:09 PM
I'm back also.
Title: Re: Site Down - 15th Sept 09
Post by: tuftedduck on September 15, 2009, 04:58:15 PM
Good to be back.. :)

Unlike waltergmw I could not access any part of the site, as from about 0930 hours today.

edit to add......meant to say that at my first attempt to access twenty minutes or so ago,  I used the long standing linky in my bookmarks file but that took me to a blank white page.........I had to google Kitz, navigate back to the forum and then re-bookmark the page.
The old and the new links appear identical.....but the old one led nowhere.
I don't know if that has any significance or if anyone else had the same occurance..
Title: Re: Site Down - 15th Sept 09
Post by: camallison on September 15, 2009, 04:59:23 PM
For me, the whole site went into sulk mode - forum + very informative part.  I thought I had been sent to the sin bin!   :o
Title: Re: Site Down - 15th Sept 09
Post by: camallison on September 15, 2009, 05:00:39 PM
This was my traceroute since 09:30:

11 194.74.65.138    27ms   28ms   28ms  TTL:  0  (core2-10G0-5-0-0.ealing.ukcore.bt.net ok)
12 194.74.65.126    28ms   27ms   28ms  TTL:  0  (core2-pos5-0-0.telehouse.ukcore.bt.net ok)
13 195.99.125.86    86ms   28ms   28ms  TTL:  0  (No rDNS)
14 87.127.246.121   28ms   27ms   27ms  TTL:  0  (te4-4.telehouse-north.core.enta.net ok)
15 87.127.236.37    29ms   29ms   29ms  TTL:  0  (te5-3.telehouse-east2.core.enta.net ok)
16 87.127.236.97    30ms   31ms   30ms  TTL:  0  (te4-4.telehouse-east.core.enta.net ok)
17 78.33.51.146     26ms   28ms   27ms  TTL:  0  (78-33-51-146.static.enta.net probable bogus rDNS: No DNS)
18 91.198.165.4     28ms   28ms   29ms  TTL:  0  (br1.core.misp.co.uk probable bogus rDNS: No DNS)
19 91.198.165.5     29ms   29ms   28ms  TTL:  0  (ge-1-377.captain.core.misp.co.uk probable bogus rDNS: No DNS)
Title: Re: Site Down - 15th Sept 09
Post by: kitz on September 15, 2009, 08:58:11 PM
Quote
Please accept our sincere apologies for the disruption today which began at 9:30am. The server is now running normally with no data loss. Total downtime was approximately six hours.

Earlier this morning we detected suspicious activity under a user's account, and as a precaution immediately took the server offline to investigate. We took a copy of the server’s file system so that there could be no data loss. We take security very seriously, and did not want to underestimate any potential risk.

Our investigations found that software running on one user's account had been accessed by a third party and used to upload malicious files to the server. These files have now been removed and the server brought back online.

../snip/..


Please note that its note that when they say users account, its not related to users on this site.
A server will host many (100's) of different websites and it would appear it was one of those other sites that got breached due to them using unpatched/old software.

This site was unaffected, but as a precaution they took the whole server offline, just to make sure that no other sites could be/were infected.


--------------



Ive just also got off the phone to them as some of you may have earlier seen an error notification when trying to access the forum

Parse error: syntax error, unexpected T_ELSE in forum/Sources/Subs.php

This was a result that the server was previously running php4 & php5 and it has now been modified to only use php5. 
This change was scheduled to happen tonight as part of this work (http://forum.kitz.co.uk/index.php?topic=5657.0) but as a result of todays breach it was brought forward slightly whilst they were working on the server.
Title: Re: Site Down - 15th Sept 09
Post by: roseway on September 15, 2009, 09:02:08 PM
Thanks for updating us. What a frustrating day for you! :(
Title: Re: Site Down - 15th Sept 09
Post by: kitz on September 15, 2009, 09:05:32 PM
Re the down time.

The whole site would have been completely offline for a couple of hours and all services suspended (including mail).

Some parts/services were brought back up before others,  eg mail, then html.
The forum took a while longer to come back up as it uses php /sql .
There may have been parts of the site such as those pages that just use html rather than php that came back up first whilst the script elements were reconfigured.
Title: Re: Site Down - 15th Sept 09
Post by: kitz on September 15, 2009, 09:11:52 PM
>> Thanks for updating us. What a frustrating day for you!

lol.. not really. - I couldnt work on the site & I had no email access.

So...... I sat and played a game this afternoon rather than doing anything site related.
 :lol: :lol:

OK.... there were a couple of times when I had to phone up about reconfiguring some things as the site began to come back up, but aside from that there wasnt anything I could do.. and I just reported and let them do the work. :)