[issue13281] Make robotparser.RobotFileParser ignore blank lines

[Ezio Melotti]

Ezio Melotti <ezio.melotti at gmail.com> added the comment:

If it's added it should be a versionchanged, not a versionadded.
I'm also not entirely sure this should be considered a new feature and don't see the point of having a strict mode.  IMHO robotparser should honor what the robots.txt files say, and not doing so because there's an extra blank line doesn't strike me as a useful behavior.  Of course it shouldn't parse every kind of broken syntax, but the OP pointed to two fairly popular websites that use blank lines and that seems to indicate that blank lines are generally allowed.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13281>
_______________________________________