[Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)

[Phillip J. Eby]

At 12:49 AM 4/19/2006 +0200, Martin v. Löwis wrote:
>Fredrik Lundh wrote:
> > it's still listed under "possible additions" in the release PEP, and 
> there don't
> > seem to be a PEP or any other easily located document that explains exactly
> > how it works, what's required from library developers, how it affects 
> existing
> > toolchains, etc.  is this really ready for inclusion ?  does anyone but 
> Phillip
> > understand how it works ?
>
>I don't understand it.

Have you read the manuals?


>  My concern is that it appears to involve a lot of
>magic.

Please define "magic".  Better, please point to the API functions or 
classes, or the setup commands documented in the manual, to show me what 
these things are that appear to be magic.

There do exist implementation details that are not included in user 
documentation, but this is going to be true of any system.  These details 
are sometimes necessarily complex due to distutils limitations, behavioral 
quirks of deployed packages using distutils, and the sometimes baroque 
variations in sys.path resolution across platforms, Python versions, and 
invocation methods.

That these details are not discussed at length in the user documentation is 
because they are boring hideous things that I wish I didn't have to know 
myself, and that few people using setuptools would ever want to know.

They *do*, however, get discussed (or at least explained by me) at length 
on the distutils-sig.  I explain the quirks and the tradeoffs for each bit 
of implementation in detail there, and have been doing so for nearly a year 
now.  Anybody who wants to know what's going on has had plenty of 
opportunity to learn.


>This magic might do the "right thing" in many cases, and it might
>indeed help the user that the magic is present, yet I still do believe
>that magic is inherently evil: explicit is better than implicit.

Are documented defaults "implicit" or "magic"?

To the extent that there is anything that may be called "magic" in 
setuptools, it exists only because the necessary infrastructure isn't 
already present in Python, or because it was required to work around the 
quirks of other systems.  Setuptools chooses to "work at all costs" because 
backward-compatibility skirts the chicken-and-egg problems that would 
otherwise exist.

If setuptools *weren't* so highly backward-compatible, its use would never 
have caught on enough to allow this discussion to be taking place in the 
first place.  :)

The primary place where it *isn't* backward compatible is if somebody does 
"setup.py install" *without* using --root, *and* they care about the exact 
file layout of the result, *and* they are installing a package that 
explicitly uses setuptools... in which case their complaint is with the 
author of the package, if the author didn't explain that they used 
setuptools or point them to relevant documentation.

Setuptools' manual prominently explains what developers should tell their 
users, if they use setuptools in their setup.py:

"""To keep these users happy, you should review the following topics in 
your project's installation instructions, if they are relevant to your 
project and your target audience isn't already familiar with setuptools and 
easy_install."""

http://peak.telecommunity.com/DevCenter/setuptools#what-your-users-should-know

With respect to you and MAL, I think that some of your judgments regarding 
setuptools may have perhaps been largely formed at a time last year when, 
among other things:

* That documentation section I just referenced didn't exist yet

* Many common installation scenarios (e.g. custom PYTHONPATH) didn't "just 
work" without special setup steps

* --single-version-externally-managed was barely a proposal, and it wasn't 
automatically activated when --root was used

These are significant changes that are directly relevant to the objections 
that you and he raised (regarding startup time, path length, tools 
compatibility, etc.), and which I added because of those objections.  I 
think that they are appropriate responses to the issues raised, in that 
they allow the audience that cares about them (you, MAL, and system 
packagers in general) to avoid the problems that those features' absence 
caused.

It would probably helpful if you would both be as specific as possible in 
your objections so that they can be addressed individually.  If you don't 
want setuptools in 2.5, let's be clear either on the specific 
objections.  If the objection is to the very *idea* of setuptools, that's 
fine too, as long as we're clear that's what the objection is.

So I would ask that you please make a list of what you would have to see 
changed in setuptools before you would consider it viable for stdlib 
inclusion, or simply admit that there are no changes that would satisfy 
you, or that you don't know enough about it to say, or that you'd like it 
to be kicked back to distutils-sig for more discussion ad infinitum, or 
whatever your actual objections are.

Then I will make my responses to your proposals, and then Guido can have 
his say based on the cons from you and the pro's from me.  If he says no 
for 2.5, that's okay by me.  I didn't propose its inclusion, the users (and 
Guido) did.  But now that I've busted butt to get it ready in time, I'd 
prefer that any withdrawal decision be based on actual facts, rather than 
FUD, hand-waving, and vague innuendo.  Meanwhile, this discussion has used 
up the time that I otherwise would have spent writing 2.5 documentation 
today (e.g., for the pkgutil additions).