[Python-bugs-list] [ python-Bugs-503031 ] urllib.py: open_http() host problem

[noreply@sourceforge.net]

Bugs item #503031, was opened at 2002-01-13 10:09
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=503031&group_id=5470

Category: Python Library
Group: Python 2.2
Status: Open
Resolution: None
Priority: 1
Submitted By: Jason Cowley (sachmoz)
Assigned to: Guido van Rossum (gvanrossum)
Summary: urllib.py: open_http() host problem

Initial Comment:
While trying to use the httplib.py urlopen() function, 
as follows:

doc = urlopen("http://www.python.org").read()
print doc

I was receiving the following trace:

Traceback (most recent call last):
  File "C:/Documents and 
Settings/Administrator/Desktop/jason/grabpage.py", 
line 3, in ?
    doc = urlopen("http://www.python.org").read()
  File "C:\Python22\lib\urllib.py", line 73, in urlopen
    return _urlopener.open(url)
  File "C:\Python22\lib\urllib.py", line 178, in open
    return getattr(self, name)(url)
  File "C:\Python22\lib\urllib.py", line 283, in 
open_http
    h = httplib.HTTP(host)
  File "C:\Python22\lib\httplib.py", line 688, in 
__init__
    self._setup(self._connection_class(host, port))
  File "C:\Python22\lib\httplib.py", line 343, in 
__init__
    self._set_hostport(host, port)
  File "C:\Python22\lib\httplib.py", line 349, in 
_set_hostport
    port = int(host[i+1:])
ValueError: invalid literal for int(): 

I managed to track the problem down to the function 
open_http() in urllib.py. The value of the 'host' 
variable contained the string 'http:' rather 
than 'www.python.org', when a call is made as follows:

httplib.HTTP(host)

Line 272 of urllib.py should be setting the 
variable 'host' to the value of 'realhost' but the 
statement is never executed. The function 'proxy_bypas
()' doesn't appear to do anything but return 0.

I fixed it for my own purposes by adding a statement:

host = realhost


----------------------------------------------------------------------

>Comment By: Jason Cowley (sachmoz)
Date: 2002-01-15 10:17

Message:
Logged In: YES 
user_id=426262

The actual settings in the registry look slightly different:

http=http://www-cache.freeserve.com:8080;ftp=http://www-
cache.freeserve.com:8080

Notice the '=' signs.

These settings have been set automatically by Freeserve, 
and so there are perhaps millions of people in the UK with 
the same registry settings (and therefore the same problem).

I have mailed Freeserve to ask them to confirm if the 
settings are correct.


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-01-15 09:46

Message:
Logged In: YES 
user_id=6380

If that's really what getproxies_registry() prints, then
look again at the URL in the dict for key 'http'. It says

'http://http://www-cache.freeserve.com:8080'

In other words a double http:// prefix!!!

If you fix the registry the problem will go away.

I don't think this is a problem with urllib.py.


----------------------------------------------------------------------

Comment By: Jason Cowley (sachmoz)
Date: 2002-01-14 05:04

Message:
Logged In: YES 
user_id=426262

I hope this is what you need:

>>> print getproxies_environment()
{}
>>> print getproxies_registry()
{'ftp': 'ftp://http://www-
cache.freeserve.com:8080', 'http': 'http://http://www-
cache.freeserve.com:8080'}
>>>


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-01-13 20:46

Message:
Logged In: YES 
user_id=6380

Hm, you can only ever end up in that code block if you have
some kind of proxy settings active. On Windows, those are in
the registry, even if you think they are not.

Your fix is clearly not right -- but in order to find out
what is right, I need your proxy settings.

----------------------------------------------------------------------

Comment By: Jason Cowley (sachmoz)
Date: 2002-01-13 14:35

Message:
Logged In: YES 
user_id=426262

I am not using a proxy, but I have a dial-up connection to 
an ISP and I am using Windows 2000.

The Python version info is:

Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit 
(Intel)] on win32

Here is the modification I made to urllib.py:

272: if proxy_bypass(realhost):
273:   host = realhost # this line was not being executed
274: host = realhost   # I added this to fix urlopen()

Without this line I added, the following statement was
being executed 9-10 lines below, with 'http:' as the value
of host:

h = httplib.HTTP(host)

Which later caused the problem when _set_hostport in 
httplib.py tries to convert an empty string to an int on 
line 349:
 
port = int(host[i+1:])

I have attached my copy of "urllib.py".


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-01-13 13:33

Message:
Logged In: YES 
user_id=6380

I cannot reproduce this.

What are your proxy settings?


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=503031&group_id=5470