Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Author Topic: One website crawler using excessive bandwidth  (Read 3820 times)

broadstairs

  • Kitizen
  • ****
  • Posts: 2792
One website crawler using excessive bandwidth
« on: May 25, 2013, 08:46:26 AM »

This morning I discovered my website has used over 12gb of my monthly bandwidth allowance, I have a 20gb a month allowance and it usually eats up somehwre between 2-4gb.

Looking at my statistics I have discovered that one website crawler has used over 10gb of my bandwidth this month, it is called BSpider. I use the .htaccess file to try to stop unwanted crawlers only allowing the good guys to index my website so I've added this one now.

I am wondering if anyone else here has had a problem with this crawler?

Stuart
Logged
ISP:TalkTalk Connection:FTTC Cab:ECI Router:ZyXEL VMG3925-B10B

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 30478
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: One website crawler using excessive bandwidth
« Reply #1 on: May 25, 2013, 09:39:06 PM »

12Gb is hellish excessive!  Ive never had a crawler use absolutely anywhere near that.   
I wouldnt be too pleased if I were you so dont blame you for adding them to your htaccess file. :shoot:

Ive not heard of it before, so Ive just looked at my stats.
Whoever it is, seems to have left me  unscathed with 577k and only 39 visits in total for the month.
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

Azzaka

  • ISP Rep
  • Reg Member
  • *
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: One website crawler using excessive bandwidth
« Reply #2 on: May 23, 2014, 07:22:30 AM »

http://www.botreports.com/user-agent/bspider10-libwww-perl040.shtml

Shows it's not very nice at all.

One way to stop it, well to divert and then if you like, hit it back: http://n0where.net/automated-target-acquisition/
Logged
I Sync', I Auth', therefore I am.
Online