[Catalog-sig] why is the wiki being hit so hard?
"Martin v. Löwis"
martin at v.loewis.de
Sat Aug 4 09:42:45 CEST 2007
> If they do not respect them, then you can use this program:
> http://danielwebb.us/software/bot-trap/ to catch them.
> If you are doing this, Martin, use the German version instead:
> because it has a few useful additions. I forget what now.
> Most scrapers, these days, respect robots.txt which will make this
> program useless for catching them. But some days you can get lucky.
That would also be an idea. I'll see how the throttling works out;
if it fails (either because it still gets overloaded - which shouldn't
happen - or because legitimate users complain), I'll try that one.
More information about the Catalog-SIG