[New-bugs-announce] [issue41803] Robots.txt
Skyreka Agence Web
report at bugs.python.org
Thu Sep 17 07:50:11 EDT 2020
New submission from Skyreka Agence Web <admin at skyreka.com>:
I can't documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.skyreka.com/robots.txt, I believe that this is how it's implemented now in most parsers ?
----------
messages: 377055
nosy: admin2
priority: normal
severity: normal
status: open
title: Robots.txt
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41803>
_______________________________________
More information about the New-bugs-announce
mailing list