[New-bugs-announce] [issue4108] robotparser.py fail when more than one User-Agent: * is present
taskinoor hasan sajid
report at bugs.python.org
Sun Oct 12 15:41:32 CEST 2008
New submission from taskinoor hasan sajid <taskinoor at gmail.com>:
Check the robots.txt file from mathworld.
It contains 2 User-Agent: * lines.
"These name tokens are used in User-agent lines in /robots.txt to
identify to which specific robots the record applies. The robot
must obey the first record in /robots.txt that contains a User-
Agent line whose value contains the name token of the robot as a
substring. The name comparisons are case-insensitive. If no such
record exists, it should obey the first record with a User-agent
line with a "*" value, if present. If no record satisfied either
condition, or no records are present at all, access is unlimited."
But it seems that our robotparser is obeying the 2nd one. the problem
occures because robotparser assumes that no robots.txt will contain two
* user-agent. it should not have two two such line, but in reality many
site may have two.
So i have changed robotparser.py as follow:
def _add_entry(self, entry):
if "*" in entry.useragents:
# the default entry is considered last
if self.default_entry == None: # this check is added
self.default_entry = entry
And at the end of parse(self, lines) method
self._add_entry(entry) # necessary if there is no new line
at end and last User-Agent is *
components: Library (Lib)
title: robotparser.py fail when more than one User-Agent: * is present
versions: Python 2.5
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce