Kitz Forum
Internet => Web Hosting & Web Design => Topic started by: broadstairs on May 25, 2013, 08:46:26 AM
-
This morning I discovered my website has used over 12gb of my monthly bandwidth allowance, I have a 20gb a month allowance and it usually eats up somehwre between 2-4gb.
Looking at my statistics I have discovered that one website crawler has used over 10gb of my bandwidth this month, it is called BSpider. I use the .htaccess file to try to stop unwanted crawlers only allowing the good guys to index my website so I've added this one now.
I am wondering if anyone else here has had a problem with this crawler?
Stuart
-
12Gb is hellish excessive! Ive never had a crawler use absolutely anywhere near that.
I wouldnt be too pleased if I were you so dont blame you for adding them to your htaccess file. :shoot:
Ive not heard of it before, so Ive just looked at my stats.
Whoever it is, seems to have left me unscathed with 577k and only 39 visits in total for the month.
-
http://www.botreports.com/user-agent/bspider10-libwww-perl040.shtml
Shows it's not very nice at all.
One way to stop it, well to divert and then if you like, hit it back: http://n0where.net/automated-target-acquisition/