[New-bugs-announce] [issue21475] Support the Sitemap and Crawl-delay extensions in robotparser

[Raymond Hettinger]

New submission from Raymond Hettinger:

Resources:

* http://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions

* https://support.google.com/webmasters/answer/183669?hl=en

* https://github.com/seomoz/reppy

----------
components: Library (Lib)
keywords: easy
messages: 218308
nosy: rhettinger
priority: normal
severity: normal
status: open
title: Support the Sitemap and Crawl-delay extensions in robotparser
type: enhancement
versions: Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21475>
_______________________________________