Kitz Forum
Announcements => Site Announcements => Topic started by: Astral on June 25, 2007, 09:29:53 AM
-
There were 44 visitors about 10 minutes ago, now it's up to 75! :)
-
102 and rising. What's going on?
-
missed it - although I see it got up to 109.
Just had a look at the logs and I cant really see anything particulary unusual.
atm theres a batch of yahoo slurp bots spidering the forum, but users were at 22 when I started to check and spread is about right mix.
[edit now down to 12]
Since I can only really check the "current" viewers doesnt really give much away im afraid.
-
>>>> a batch of yahoo slurp bots spidering the forum
Isn't the English language wonderful? :lol:
-
heh - it is when you look at it like that :lol:
-
I see we're up to 117 today. Would it be those bootilicious yahoo spiders again?
-
damn - missed it again :(
goes to see if I can see anything in the logs
-
They're back with their mates!
-
wth!!!
Most Online Ever: 189 (June 27, 2007, 11:04:55 PM)
Missed it again though :(
Its got to be spider bots for them to all arrive at once like that.
afaik, the forums are already pretty well googled, so if its the Yahoo slurp bots catching up indexing the site or what I dunno.
But that also seems like one hell of a lot of bots to arrive all at once.
-
Your name was lit up when I posted They must lie in wait until you sign out. :)
-
Does it only happen late at night when I'm at my slumbers?
-
I literally must have just gone afk when it all happened. Will have to try be around tonight at about 11ish to see if it happens then.
-
This is ridiculous - theres 130 of them on now and it is Yahoo slurp bots.
Cant believe that in some cases they are all crawling the same threads!
-
Good heavens, yes, I see them.
-
Seems like I'm not the only one experiencing this problem. I did some googling earlier and theres a heck of a lot of sites out there complaining this month about the behaviour of Yahoo Slurp bots.
I saw someone say they'd increased his bandwidth by about 13GB
Theyve gone crazy!
I could try blocking them all [seen a few sites actually resort to that], but not sure if I really want to do this just yet.
Will keep an eye on things and see if someone comes up with something in the meantime - or hopefully yahoo sort their act out.
-
Looks like the bots have come for another slurp!
-
Unless anyone has seen otherwise - it seems to have been back to normal over the past few days?
-
I haven't seen anything recently. I don't know if this is relevant, but I came upon this comment from the admin of another forum having similar problems:
"One thing I did was to block Yahoo's stupid search bot. It was clogging up the site and ignoring the robots.txt file."
-
Thanks eric.
I had actually considered the robots.txt file if things continued. Before chris went to france he also passed me a linky about a SMF mod that could perhaps be implemented which stops bots being listed in your stats. Unfort I havent had chance yet to do anything further on that one, but its something I may well due in the future when I have chance to look at it properly.
From what I have seen things do seem to have settled down a lot and wonder if the bots were just having a mad month? (or couple of days whilst it did a pile of updating?)
-
Yes, it seems to have gone quiet, so maybe it was just an updating splurge.
-
There is one site I go on, they get loads every day !.
-
It was silly when you saw it happen.. I managed to catch them at it and view the forum stat logs at the same time.
They'd all arrive at once and there would be about 12 bots looking at exactly the same thread.. they'd hang around for about half an hour - again many of them all indexing the same threads. I cant understand why it took about 180 bots to crawl a site all at once.
crazy - a couple of bots should easily be able to crawl most sites within half an hour.
-
When I looked at who was on-line earlier today, I thought, my god you've been out touting for new members LOL.
And the numbers were the sudden influx of new members. :D
Sorry I shouldn't laugh because I don't know what strain it puts onto your site.
-
No the sites fine.. when I was looking for hosting earlier in the year I was looking at medium sized packages, but bearing in mind how much traffic had grown over the previous 6 months, with a little help I decided that going for a business package would be a good idea.
I'm glad I did now because the sites been slashdotted and survived :)
I can imagine a fair few website holders not being too happy though, if youre on a normal hosting package then something like this could take you over your limits.
The main site gets the vast majority of the hits but they seemed to leave that as normal and only target the forums. This is how much bandwidth they used on the forums alone
- Inktomi Slurp 819.25 MB
Compare that to a couple of other random stats (bearing in mind google is normally the biggy for most sites)
Googlebot 275.58 MB
Unknown robot (identified by 'crawl') 59.50 MB
Alexa (IA Archiver) 22.53 MB
Unknown robot (identified by 'robot') 5.04 MB
AskJeeves 3.13 MB
Unknown robot (identified by 'spider') 443.80 KB
Lycos 118.11 KB