RobotRules ---------- Parses robots.txt files. NB: This is a straight conversion of the perl module WWW::RobotRules This module parses a robots.txt file as specified in http://info.webcrawler.com/mak/projects/robots/norobots.html URL: http://nano.com/~vivake/python/ Download: http://nano.com/~vivake/python/RobotRules.py License: Python Style Categories: Web Vivake Gupta (vivake@nano.com) http://nano.com/~vivake/ -- <a href="http://nano.com/~vivake/python/">RobotRules</a> -- Parses robots.txt files.