[issue6325] robotparser doesn't handle URL's with query strings

[Senthil Kumaran]

Senthil Kumaran <orsenthil at gmail.com> added the comment:

I modified the patch slightly (so that it takes care of path, query, params and fragments).

Fixed in r83209,r83210 and r83211.

I also think that we need to move the robotparser to allow regexs in the allow and disallow patterns. ( Shall open an issue in the tracker, if it is not already present).

----------
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6325>
_______________________________________