Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
   Compare ISP   Rate your ISP
   Glossary   Glossary
Please login or register.

Login with username, password and session length
Advanced search  


Author Topic: One website crawler using excessive bandwidth  (Read 5525 times)


  • Kitizen
  • ****
  • Posts: 3624
One website crawler using excessive bandwidth
« on: May 25, 2013, 08:46:26 AM »

This morning I discovered my website has used over 12gb of my monthly bandwidth allowance, I have a 20gb a month allowance and it usually eats up somehwre between 2-4gb.

Looking at my statistics I have discovered that one website crawler has used over 10gb of my bandwidth this month, it is called BSpider. I use the .htaccess file to try to stop unwanted crawlers only allowing the good guys to index my website so I've added this one now.

I am wondering if anyone else here has had a problem with this crawler?

ISP:BT Router:BT Smart Hub 2 Connection:FTTP


  • Administrator
  • Senior Kitizen
  • *
  • Posts: 33164
  • Trinity: Most guys do.
Re: One website crawler using excessive bandwidth
« Reply #1 on: May 25, 2013, 09:39:06 PM »

12Gb is hellish excessive!  Ive never had a crawler use absolutely anywhere near that.   
I wouldnt be too pleased if I were you so dont blame you for adding them to your htaccess file. :shoot:

Ive not heard of it before, so Ive just looked at my stats.
Whoever it is, seems to have left me  unscathed with 577k and only 39 visits in total for the month.
Please do not PM me with queries for broadband help as I may not be able to respond.
How to get your router line stats :: ADSL Exchange Checker


  • Reg Member
  • ***
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: One website crawler using excessive bandwidth
« Reply #2 on: May 23, 2014, 07:22:30 AM »

Shows it's not very nice at all.

One way to stop it, well to divert and then if you like, hit it back:
I Sync', I Auth', therefore I am.