12 May
2014
12 May
'14
2:15 a.m.
If there is anyone here with an interest in web-spiders, it would be nice if someone else could take a look at
http://bugs.python.org/issue21469
which addresses the risk of false positives with the robots.txt parser.
Raymond