[New-bugs-announce] [issue7152] urllib2.build_opener() skips ProxyHandler

[Barry A. Warsaw]

New submission from Barry A. Warsaw <barry at python.org>:

Try this:

>>> from urllib2 import build_opener
>>> build_opener().handlers

In Python 2.4, you will see ProxyHandler as the first handler, but this
handler is missing from the list in Python 2.5, 2.6, and 2.7, despite this
text in the documentation:

    urllib2.build_opener([handler, ...])

    Return an OpenerDirector instance, which chains the handlers in the
order
    given. handlers can be either instances of BaseHandler, or subclasses of
    BaseHandler (in which case it must be possible to call the constructor
    without any parameters). Instances of the following classes will be in
    front of the handlers, unless the handlers contain them, instances
of them
    or subclasses of them: ProxyHandler, UnknownHandler, HTTPHandler,
    HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler,
    HTTPErrorProcessor.

In fact, there is no way to add a ProxyHandler at all using the public API.
This is because the following code was added to Python 2.5, purportedly as a
fix for bug 972322:

    http://bugs.python.org/issue972322

# urllib2.py:307

            if meth in ["redirect_request", "do_open", "proxy_open"]:
                # oops, coincidental match
                continue

Because of this, the following are not a workarounds:

>>> opener.add_handler(ProxyHandler)
>>> build_opener(ProxyHandler())

In fact, as near as I can tell, the only way to get a ProxyHandler in
there is
to do an end-run around .add_handler():

>>> proxy_handler = ProxyHandler()
>>> opener.handlers.insert(0, proxy_handler)
>>> proxy_handler.add_parent(opener)

I'm actually quite shocked this has never been reported before.

ISTM that the right fix is what was originally suggested in bug 972322:

    http://bugs.python.org/msg46172

"The alternative would be to rename do_open and proxy_open, and leave the
redirect_request case unchanged (see below for why)."

The intent of this patch could not have been to completely prevent
ProxyHandler from being included in the list of handlers, otherwise why keep
ProxyHandler at all?  If that was the case, then the documentation for
urllib2
is broken, and it should have described this change as occurring in Python
2.5.

----------
components: Library (Lib)
messages: 94144
nosy: barry
priority: high
severity: normal
stage: needs patch
status: open
title: urllib2.build_opener() skips ProxyHandler
type: behavior
versions: Python 2.5, Python 2.6, Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7152>
_______________________________________