If there is anyone here with an interest in web-spiders,
it would be nice if someone else could take a look at
which addresses the risk of false positives with the robots.txt parser.
Back to the thread
Back to the list