Internet > Web Hosting & Web Design

One website crawler using excessive bandwidth

(1/1)

broadstairs:
This morning I discovered my website has used over 12gb of my monthly bandwidth allowance, I have a 20gb a month allowance and it usually eats up somehwre between 2-4gb.

Looking at my statistics I have discovered that one website crawler has used over 10gb of my bandwidth this month, it is called BSpider. I use the .htaccess file to try to stop unwanted crawlers only allowing the good guys to index my website so I've added this one now.

I am wondering if anyone else here has had a problem with this crawler?

Stuart

kitz:
12Gb is hellish excessive!  Ive never had a crawler use absolutely anywhere near that.   
I wouldnt be too pleased if I were you so dont blame you for adding them to your htaccess file. :shoot:

Ive not heard of it before, so Ive just looked at my stats.
Whoever it is, seems to have left me  unscathed with 577k and only 39 visits in total for the month.

Azzaka:
http://www.botreports.com/user-agent/bspider10-libwww-perl040.shtml

Shows it's not very nice at all.

One way to stop it, well to divert and then if you like, hit it back: http://n0where.net/automated-target-acquisition/

Navigation

[0] Message Index