[New-bugs-announce] [issue5625] test_urllib2 fails - urlopen error file not on local host

Ned Deily report at bugs.python.org
Tue Mar 31 17:48:08 CEST 2009

New submission from Ned Deily <nad at acm.org>:

[NOTE: applies to 2.x urllib2 and similar code in merged 3.x urllib]

test_urllib2 can fail because urllib2.FileHandler assumes incorrectly
that the local host has only a single IP address.  It is not uncommon
to have host IP configurations where a host has more than one network
interface and the same IP host name is associated with each address.

Both the urllib module and test_urllib2 use
to find "the" host IP address.  But, as can be seen here, 
consecutive calls may produce different addresses depending on the
network configuration and underlying os implementation:

Python 2.6.1 (r261:67515, Dec 17 2008, 23:27:50) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.gethostbyname(socket.gethostname())
>>> socket.gethostbyname(socket.gethostname())
>>> socket.gethostbyname(socket.gethostname())

This leads to predictable test failures when the calls in test_urllib2
and urllib2.FileHandler return different addresses:

test test_urllib2 failed -- Traceback (most recent call last):
st_urllib2.py", line 621, in test_file
    r = h.file_open(Request(url))
.py", line 1229, in file_open
    return self.open_local_file(req)
.py", line 1266, in open_local_file
    raise URLError('file not on local host')
URLError: <urlopen error file not on local host>

The simplest way to avoid the test failure is to modify
urllib2.FileHandler to use socket.gethostbyname_ex which returns all
of the IPv4 addresses associated with a hostname:
>>> socket.gethostbyname_ex(socket.gethostname())
('myhost.net', [], ['', ''])

Attached patches for 2.x urllib2 and 3.x urllib do that.  Note that 
there remain other issues in this area:
- when urllib2 is enhanced to support IPv6, code is needed to return
  all of the host's IPv6 addresses as well (-> adding a note to open
- the merged 3.0 urlib has two nearly identical functions named
  open_local_file, one each from 2.x urllib.URLopener and
  urllib2.FileHandler, and both use similarly flawed
  socket.gethostbyname(socket.gethostname()) tests but the tests for
  local vs remote file URLs is somewhat different in each.
  (The patches here do not attempt to address this other than to add
   a comment.)

components: Library (Lib)
files: patch-nad0017-trunk-26.txt
messages: 84806
nosy: nad
severity: normal
status: open
title: test_urllib2 fails - urlopen error file not on local host
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1
Added file: http://bugs.python.org/file13514/patch-nad0017-trunk-26.txt

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list