[Python-Dev] PEP 365 (Adding the pkg_resources module)

[glyph at divmod.com]

On 09:33 am, p.f.moore at gmail.com wrote:
>I'll chime in here, too. I really want to like
>setuptools/easy_install, but I don't. I'll try to be specific in my
>reasons, in the hope that they can be addressed. I know some of these
>are "known about", but one of my meta-dislikes of setuptools is that
>known issues never seem to get addressed (I know, patches accepted,
>but I haven't got the time either...)

I agree with almost everything that Paul says, and he put it quite well, 
so I'll spare the "me too", but I do have some additional gripes to add.

setuptools (or, for that matter, distutils) has no business installing 
stuff in the system directory on a Linux box with a package manager. 
The *major* benefit I can see to a tool like easy_install is providing a 
feature that system packagers do not: allowing developers to quickly 
pull down all their dependencies into a *user directory* without 
worrying about system administration.  However, not only does setuptools 
not do this on its own, it actively fights me when I try to do it.

Admittedly, my user directory is a little messed up.  Combinator, the 
Divmod path management / developer deployment tool, does some 
inadvisable things to attempt to trick distutils into doing local 
installation.  However, setuptools does have some pretty clear bugs in 
this area; for example, it hard-codes a copy of a list that's present in 
site.py to try to figure out where .pth files will be honored, rather 
than looking at what's actually on sys.path.  Every time I've tried to 
install a package for development using setuptools - and I am speaking 
across a wide range of versions here, so this isn't a specific bug 
report - it's either emitted a completely inscrutable traceback or 
printed an error message telling me that it couldn't or wouldn't install 
to the provided location.
>But if these problems are solved, then I have no problem with seeing
>the features of setuptools added to the standard library - resource
>APIs, plugin/entry point APIs, ways to create executable scripts, and
>such things *should* be standardised. Dependency resolution and
>automatic installation isn't something I like (probably because as a
>Windows user I've never used such a system, so I mistrust it) but if
>it works *with* the system and not against it, I don't mind.

This is more of a vague impression than a specific technical criticism, 
but it really seems like the implementation of these features face a lot 
of unnecessary coupling in setuptools.  Twisted (Hey, did you guys know 
I work on Twisted?  It seems I hardly ever mention it!), for example, 
implements resource APIs (twisted.python.modules), plugins 
(twisted.plugin, which is a bit like some of the uses of entrypoints), 
and the zip-file agnosticism of both (twisted.python.zipstream) without 
needing any packaging metadata or ini files.  It just introspects the 
Python path and adds a little frosting to importers.

I could be wrong about setuptools' actual design; this could be a 
documentation or UI issue, because I haven't read the code.  But when 
interacting with it as a user and perusing its API, it definitely seems 
as though things are woven too tightly together, and the whole thing is 
very centered around the concept of a "build", i.e. running some kind of 
compilation or deployment step with a setup.py.  One of my favorite 
things about python is that stuff just runs without needing that 
normally.  I know that "setup.py develop" is supposed to avoid that - 
but even that itself is a deployment step which generates some metadata. 
With the setuptools-free libraries I use, it's just check out, then run; 
no other intermediary step.  I'm spoiled, of course, having apt to do 
the package-management work for me on the majority of my dependencies, 
and combinator mostly handling the rest.

easy_install also definitely has problems with security.  It 
automatically downloads links from plain-HTTP connections, some of them, 
I believe, from publicly editable wiki pages, and installs them with no 
verification of signatures (as root!  because how else are you going to 
get them to the only supported installation directory!).  I believe that 
this is possibly the easiest issue to fix though, and I hope that my 
information here is already out of date.  I realize that people are 
already doing this (insecure installation) with their web browsers, but 
there are tons of UI cues in a web browser looking at a link on a wiki 
page which you don't get from an automated command-line tool.

As others have said, I wanted to like setuptools.  I wanted to believe 
we could be saved from the unfortunate issues with distutils.  But the 
extremely high degree of frustration I've encountered every time I've 
tried to use it, its general inscrutability, its awful UI ("python -c 
"import setuptools; execfile('setup.py')" develop", seriously?  you 
couldn't write a command-line tool to make that look like 'setuptool 
develop' or something?) and now the availability of my own libraries 
which do the things in setuptools that interest me, have served to 
strongly discourage me from trying to closely inspect or fix it.  I just 
kind of vaguely hope that it will be overhauled if it's ever really 
considered for inclusion in the standard library and I try not to think 
too hard about it.  I'm not actively opposing it, for those who want to 
use it - c.f. http://twistedmatrix.com/trac/ticket/1286 - but it 
definitely doesn't work for me.

Just for the record: I wrote my own zip-file-friendly resource-loading 
library after trying to use setuptools.  I had to get some code working 
on an embedded device, and it needed to load Twisted plugins (which 
predates setuptools by a long while, I believe - or at least I didn't 
know about them at the time).  setuptools somehow simultaneously broke 
the path requirements of the twisted plugin system and blew my memory 
budget.  I attempted to investigate but didn't get far - it was quite a 
lot easier to just write some libraries that performed one task at a 
time rather than trying to manage the whole deployment.