Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 [2]

Author Topic: Sudden rush  (Read 10979 times)

Astral

  • Addicted Kitizen
  • *****
  • Posts: 6864
Re: Sudden rush
« Reply #15 on: June 29, 2007, 09:27:33 AM »

Looks like the bots have come for another slurp!
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Sudden rush
« Reply #16 on: July 02, 2007, 08:49:43 PM »

Unless anyone has seen otherwise - it seems to have been back to normal over the past few days?
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43467
  • Penguins CAN fly
    • DSLstats
Re: Sudden rush
« Reply #17 on: July 05, 2007, 07:53:10 AM »

I haven't seen anything recently. I don't know if this is relevant, but I came upon this comment from the admin of another forum having similar problems:

"One thing I did was to block Yahoo's stupid search bot. It  was clogging up the site and ignoring the robots.txt file."
Logged
  Eric

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Sudden rush
« Reply #18 on: July 05, 2007, 01:30:13 PM »

Thanks eric.

I had actually considered the robots.txt file if things continued.  Before chris went to france he also passed me a linky about a SMF mod that could perhaps be implemented which stops bots being listed in your stats.  Unfort I havent had chance yet to do anything further on that one, but its something I may well due in the future when I have chance to look at it properly.

From what I have seen things do seem to have settled down a lot and wonder if the bots were just having a mad month? (or couple of days whilst it did a pile of updating?)
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43467
  • Penguins CAN fly
    • DSLstats
Re: Sudden rush
« Reply #19 on: July 05, 2007, 03:59:01 PM »

Yes, it seems to have gone quiet, so maybe it was just an updating splurge.
Logged
  Eric

tickmike

  • Kitizen
  • ****
  • Posts: 3640
  • Yes Another Penguin !. :)
Re: Sudden rush
« Reply #20 on: July 05, 2007, 10:05:42 PM »

There is one site I go on, they get loads every day  !.
Logged
I have a set of 6 fixed IP's From  Eclipse  isp.BT ADSL2(G992.3) line>HG612 as a Modem, Bridge, WAN Not Bound to LAN1 or 2 + Also have FTTP (G.984) No One isp Fixed IP >Dual WAN pfSense (Hardware Firewall and routing).> Two WAN's, Ethernet LAN, DMZ LAN, Zyxel GS1100-24 Switch.

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Sudden rush
« Reply #21 on: July 06, 2007, 12:35:06 AM »

It was silly when you saw it happen..   I managed to catch them at it and view the forum stat logs at the same time.

They'd all arrive at once and there would be about 12 bots looking at exactly the same thread..  they'd hang around for about half an hour - again many of them all indexing the same threads.  I cant understand why it took about 180 bots to crawl a site all at once.

crazy - a couple of bots should easily be able to crawl most sites within half an hour.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

oldfogy

  • Helpful
  • Kitizen
  • *
  • Posts: 3568
  • If it ain't broke....... I'll soon fix it.
Re: Sudden rush
« Reply #22 on: July 06, 2007, 12:56:58 AM »

When I looked at who was on-line earlier today, I thought, my god you've been out touting for new members LOL.
And the numbers were the sudden influx of new members.  :D
Sorry I shouldn't laugh because I don't know what strain it puts onto your site.
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33879
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: Sudden rush
« Reply #23 on: July 06, 2007, 01:34:24 AM »

No the sites fine..  when I was looking for hosting earlier in the year I was looking at medium sized packages, but bearing in mind how much traffic had grown over the previous 6 months, with a little help I decided that going for a business package would be a good idea.

I'm glad I did now because the sites been slashdotted and survived :)

I can imagine a fair few website holders not being too happy though, if youre on a normal hosting package then something like this could take you over your limits.

The main site gets the vast majority of the hits but they seemed to leave that as normal and only target the forums. This is how much bandwidth they used on the forums alone

- Inktomi Slurp   819.25 MB

Compare that to a couple of other random stats (bearing in mind google is normally the biggy for most sites)

Googlebot 275.58 MB
Unknown robot (identified by 'crawl') 59.50 MB
Alexa (IA Archiver) 22.53 MB
Unknown robot (identified by 'robot') 5.04 MB
AskJeeves 3.13 MB
Unknown robot (identified by 'spider') 443.80 KB
Lycos  118.11 KB
 
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker
Pages: 1 [2]