[Python-bugs-list] [ python-Bugs-690214 ] robotparser only applies first applicable rule
SourceForge.net
noreply@sourceforge.net
Thu, 20 Feb 2003 10:55:14 -0800
Bugs item #690214, was opened at 2003-02-20 13:55
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=690214&group_id=5470
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Mark Pilgrim (f8dy)
Assigned to: Nobody/Anonymous (nobody)
Summary: robotparser only applies first applicable rule
Initial Comment:
robotparser robotparser.py::RobotFileParser::can_fetch
currently returns the result of the first applicable rule. It
should loop through all rules looking for anything that
disallows access. For example, if your first rule applies
to 'wget' and 'python' and disallows access to /dir1/, and
your second rule is a 'python' rule that disallows access
to /dir2/, robotparser will falsely claim that python is
allowed to access /dir2/.
Patch against current source attached.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=690214&group_id=5470