If there is anyone here with an interest in web-spiders, it would be nice if someone else could take a look at http://bugs.python.org/issue21469 which addresses the risk of false positives with the robots.txt parser. Raymond