[Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)

[zooko]

On Mar 19, 2008, at 3:23 PM, Guido van Rossum wrote:

> If other people want to chime in please do so; if this is just a
> dialog between Phillip and me I might incorrectly assume
> that nobody besides Phillip really cares.

I really care.  I've used setuptools, easy_install, eggs, and  
pkg_resources extensively for the past year or so (and contributed a  
few small patches).  There have been plenty of problems, but I find  
them to be overall useful tools.

It is a great boon to a programming community to lower the costs of  
re-using other people's code.  The Python community will benefit  
greatly once a way to do that becomes widely enough accepted to reach  
a tipping point and become ubiquitous.  Setuptools is already the de  
facto standard, but it hasn't become ubiquitous, possibly in part  
because of "egg hatred", about which more below.

I've interviewed several successful Python hackers who "hate eggs" in  
order to understand what they hate about them, and I've taken notes  
from some of these interviews.  (The list includes MvL, whose name  
was invoked earlier in this thread.)

After filtering out yer basic complaining about bugs (which  
complaints are of course legitimate, but which don't indict  
setuptools as worse than other software of comparable scope and  
maturity), their objections seem to fall into two categories:

1.  "The very notion of package dependency resolution and  
programmable or command-line installation of packages at the language  
level is a bad notion."

This can't really be the case.  If the existence of such  
functionality at the programming language level were an inherently  
bad notion, then we would be hearing some complaints from the Ruby  
folks, where the Gems system is standard and ubiquitous.  We hear no  
complaints -- only murmurs of satisfaction.  One person recently  
reported to me that while there are more packages in Python, he finds  
himself re-using other people's code more often when he works in  
Ruby, because almost all Ruby software is Gemified, but only a  
fraction of Python software is Eggified.

Often this complaint comes with the idea that eggs conflict with  
their system-level package management tools.  (These are usually  
Debian/Ubuntu users.)

Note that Ruby software is not too hard to include in operating  
system packaging schemes -- my Ubuntu Hardy apt-cache shows plenty of  
Ruby software.  A sufficiently mature and widely supported setuptools  
could actually make it easier to integrate Python software into  
Debian -- see stdeb [1].

2.  "Setuptools/eggs give me grief."

What can really be the case is that setuptools causes a host of  
small, unnecessary problems for people who prefer to do things  
differently than PJE does.  Personally, I prefer to use GNU stow, and  
setuptools causes unnecessary, but avoidable, problems for me.  Many  
people object (rightly enough) to a "./setup.py install"  
automatically fetching new software over the Internet by default.   
The fact that easy_install creates a site.py that changes the  
semantics of PYTHONPATH is probably the most widely and deservedly  
hated example of this kind of thing [2].  I could go on with a few  
other common technical complaints of this kind.

These type-2 problems can be fixed by changing setuptools or they can  
be grudgingly accepted by users, while retaining compatibility with  
the large and growing ecosystem of eggy software.  Certainly fixing  
setuptools to play better with others is a more likely path to  
success than setting out to invent a non-egg-compatible alternative.   
Such a project might never be implemented well enough to serve, and  
if it were it would probably never overtake eggs's lead in the Python  
ecosystem, and if it did it would probably not turn out to be a  
better tool.

So, since you asked for my chime, I advise you to publically bless  
eggs, setuptools, and easy_install as plausible future standards and  
solicit patches which address the complaints.  For that matter,  
soliciting specific complaints would be a good start.  I've done so  
in private many times with only partial success as to the "specific"  
part.  One promising approach is to request objections in the form of  
automated tests that setuptools fails, e.g. [3].

Regards,

Zooko O'Whielacronx

[1] http://stdeb.python-hosting.com/
[2] http://www.rittau.org/blog/20070726-02
    And no, PJE's suggested "trivial fix" does not satisfy the  
objectors, as it can't support the use case of "cd somepkg ; python ./ 
setup.py install ; cd .. ; python -c 'import somepkg'".
[3] http://twistedmatrix.com/trac/ticket/2308#comment:5