[New-bugs-announce] [issue41803] Robots.txt

Skyreka Agence Web report at bugs.python.org
Thu Sep 17 07:50:11 EDT 2020


New submission from Skyreka Agence Web <admin at skyreka.com>:

I can't  documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.skyreka.com/robots.txt, I believe that this is how it's implemented now in most parsers ?

----------
messages: 377055
nosy: admin2
priority: normal
severity: normal
status: open
title: Robots.txt

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41803>
_______________________________________


More information about the New-bugs-announce mailing list