[New-bugs-announce] [issue24333] urllib2 HTTPS connection over a digest auth enabled proxy gives 407

Chi Hsuan Yen report at bugs.python.org
Sat May 30 20:40:30 CEST 2015

New submission from Chi Hsuan Yen:

This is originally my question at stackoverflow.com. (http://stackoverflow.com/q/30511341/3786245) I think it's a bug, so I posted it here.

I'm trying to fetch HTTPS pages through a proxy with digest authentication. Here are my codes:

import urllib.request

class SimplePasswordManager(object):
    def __init__(self, username, password):
        self.username = username
        self.password = password

    def add_password(self, realm, uri, user, passwd):

    def find_user_password(self, realm, authuri):
        return self.username, self.password

proxy_handler = urllib.request.ProxyHandler({
    'http': '<proxy server ip>',
    'https': '<proxy server ip>',
password_mgr = SimplePasswordManager('<username>', '<password>')
proxy_auth_handler = urllib.request.ProxyDigestAuthHandler(passwd=password_mgr)
opener = urllib.request.build_opener(proxy_auth_handler, proxy_handler)
req = opener.open('http://httpbin.org/ip')
req = opener.open('https://httpbin.org/ip')

And the results:

  "origin": "<proxy server ip>"

Traceback (most recent call last):
  File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.4/http/client.py", line 1088, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1126, in _send_request
  File "/usr/lib/python3.4/http/client.py", line 1084, in endheaders
  File "/usr/lib/python3.4/http/client.py", line 922, in _send_output
  File "/usr/lib/python3.4/http/client.py", line 857, in send
  File "/usr/lib/python3.4/http/client.py", line 1223, in connect
  File "/usr/lib/python3.4/http/client.py", line 837, in connect
  File "/usr/lib/python3.4/http/client.py", line 820, in _tunnel
OSError: Tunnel connection failed: 407 Proxy Authentication Required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "proxy_test.py", line 25, in <module>
    req = opener.open('https://httpbin.org/ip')
  File "/usr/lib/python3.4/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 481, in _open
    '_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1225, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.4/urllib/request.py", line 1184, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error Tunnel connection failed: 407 Proxy Authentication Required>

Seems HTTP connection works while HTTPS not. I think it's a bug in urllib. For HTTPS connections, HTTPConnection.connect() calls HTTPConnection._tunnel(), and the latter function throws an OSError for 407 when sending a CONNECT request. There's no chance for OpenerDirector.open() to call HTTPErrorProcessor.http_response(). As a result, 407 errors are not handled correctly in ProxyDigestAuthHandler.http_error_407().

Finally, is there a workaround before this is fixed?

components: Library (Lib)
messages: 244483
nosy: Chi Hsuan Yen
priority: normal
severity: normal
status: open
title: urllib2 HTTPS connection over a digest auth enabled proxy gives 407
type: behavior
versions: Python 2.7, Python 3.2, Python 3.4, Python 3.6

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list