[Tutor] checking for robots on a python script based site

[Kent Johnson]

Tsila Hassine wrote:
> Hello All!
> 
> I have a site that takes some input from the user and then produces a 
> dynamic page based on the input. In the last months bots (not only 
> search engine ones) have been hitting the site pretty heavily, so my sys 
> admin disabled it. As far as I know there are 2 options to fight this :
> 1: write a robots.txt file - but then I'd have to rely on the bots' good 
> will.

Seems to me this is worth doing, at least you will screen out the white 
hat robots.

Kent