[Distutils] setuptools process_url doesn't handle URLErrors gracefullly

[Jim Fulton]

This is mostly for Phillip, or anyone else who might have a way to  
guess how the impossible might happen, possibly including experts in  
the way that urllib2 works. :)

On May 11, 2007, at 4:21 AM, Christian Theune wrote:
> I'm fighting against a  problem where the DNS name of the homepage  
> of a
> package that is listed on the cheeseshop isn't accessible any more. In
> this special case the DNS name for this thing is gone (the package is
> `rwproperty` the missing DNS entry is z3lab.org).
>
> In this case easy_install dies with the following error, although the
> eggs are listed on the cheeseshop page and the listed homepage is  
> pretty
> irrelevant:
>
> Searching for rwproperty
> Reading http://cheeseshop.python.org/pypi/rwproperty/
> Reading
> http://www.z3lab.org/sections/blogs/philipp-weitershausen/ 
> 2006_05_29_pycon-06-lightning-talk
> Traceback (most recent call last):
>   File "/usr/bin/easy_install", line 7, in ?
>     sys.exit(
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/command/easy_install.py", line 1670, in main
>     with_ei_usage(lambda:
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/command/easy_install.py", line 1659, in with_ei_usage
>     return f()
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/command/easy_install.py", line 1674, in <lambda>
>     distclass=DistributionWithoutHelpCommands, **kw
>   File "/usr/lib/python2.4/distutils/core.py", line 149, in setup
>     dist.run_commands()
>   File "/usr/lib/python2.4/distutils/dist.py", line 946, in  
> run_commands
>     self.run_command(cmd)
>   File "/usr/lib/python2.4/distutils/dist.py", line 966, in  
> run_command
>     cmd_obj.run()
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/command/easy_install.py", line 211, in run
>     self.easy_install(spec, not self.no_deps)
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/command/easy_install.py", line 432, in easy_install
>     dist = self.package_index.fetch_distribution(
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 462, in fetch_distribution
>     self.find_packages(requirement)
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 303, in find_packages
>     self.scan_url(self.index_url + requirement.unsafe_name+'/')
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 610, in scan_url
>     self.process_url(url, True)
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 201, in process_url
>     page = self.process_index(url, page)
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 278, in process_index
>     self.scan_url(new_url)
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 610, in scan_url
>     self.process_url(url, True)
>   File
> "/usr/lib/python2.4/site-packages/setuptools-0.6c5-py2.4.egg/ 
> setuptools/package_index.py", line 191, in process_url
>     self.fetched_urls[url] = self.fetched_urls[f.url] = True
> AttributeError: URLError instance has no attribute 'url'

This particular traceback is very very weird.  The line before the  
error line computes f by calling open_url:

         f = self.open_url(url)

open_url is:

     def open_url(self, url):
         if url.startswith('file:'):
             return local_open(url)
         try:
             return open_with_auth(url)
         except urllib2.HTTPError, v:
             return v
         except urllib2.URLError, v:
             raise DistutilsError("Download error: %s" % v.reason)

open_with_auth returns the result of calling urllib2.urlopen.  If  
urlopen raised URLError, it would have been caught by  open_url and  
re-raised as a DistutilsError.  The only way I can see open_url  
*returning* a URLError is if something in urllib2 returned it and I  
don't see that anywhere. I suppose that something could be catching  
URLError and returning it, but I don't see any evidence of that  
either.  I also suppose that there could be some weird urllib2 plugin  
that is causing this behavior.

The original server problem that spurred this report has been fixed.   
When I try to reproduce this myself, using a faux index server and  
project, I get the DistutilsError I would expect to be raised by  
open_url.  Also, buildbot logs we have from the original server  
problem on our servers also show the   expected DistutilsError.

I'm going to work on a fix assuming that open_url is raising a  
DistutilsError in this situation.  I'm most puzzled and troubled by  
Christian's traceback.  My fix won't fix his symptom, and I don't  
want to try to hack in a fix for his symptom if I can't reproduce or  
test it. If anyone else has any insights, they'd be welcome. :)

My last hope is that the test I write for the fix will fail on  
Christian's machine.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org