[New-bugs-announce] [issue38067] Add headers parameter on RobotFileParser

Emmanuel Arias report at bugs.python.org
Mon Sep 9 08:28:17 EDT 2019

New submission from Emmanuel Arias <emmanuelarias30 at gmail.com>:

Hello I open an issue from a PR that I think need discussion:

I write here the @fuwaraido's PR description:

> I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.

components: Library (Lib)
messages: 351440
nosy: eamanu
priority: normal
pull_requests: 15423
severity: normal
status: open
title: Add headers parameter on RobotFileParser
type: enhancement
versions: Python 3.9

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list