This morning I discovered my website has used over 12gb of my monthly bandwidth allowance, I have a 20gb a month allowance and it usually eats up somehwre between 2-4gb.
Looking at my statistics I have discovered that one website crawler has used over 10gb of my bandwidth this month, it is called BSpider. I use the .htaccess file to try to stop unwanted crawlers only allowing the good guys to index my website so I've added this one now.
I am wondering if anyone else here has had a problem with this crawler?
Stuart