[Python-Dev] Status of packaging in 3.3

[PJ Eby]

On Wed, Jun 20, 2012 at 11:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Right - clearly enumerating the features that draw people to use
> setuptools over just using distutils should be a key element in any
> PEP for 3.4
>
> I honestly think a big part of why packaging ended up being incomplete
> for 3.3 is that we still don't have a clearly documented answer to two
> critical questions:
> 1. Why do people choose setuptools over distutils?

Some of the reasons:

* Dependencies
* Namespace packages
* Less boilerplate in setup.py (revision control, data files support,
find_packages(), etc.)
* Entry points system for creating extensible applications and frameworks
that need runtime plugin discovery
* Command-line script wrappers
* Binary plugin installation system for apps (i.e. dump eggs in a directory
and let pkg_resources figure out what to put on sys.path)
* "Test" command
* Easy distribution of (and runtime access to) static data resources

Of these, automatic dependency resolution with as close to 100% backward
compatibility for installing other projects on PyPI was almost certainly
the #1 factor driving setuptools' initial adoption.  The 20% that drives
the 80%, as it were.  The rest are the 80% that brings in the remaining 20%.

>
> 2. What's wrong with setuptools that meant the idea of including it
> directly in the stdlib was ultimately dropped and eventually replaced
> with the goal of incorporating distutils2?

Based on the feedback from Python-Dev, I withdrew setuptools from 2.5
because of what I considered valid concerns raised regarding:

1. Lack of available persons besides myself familiar with the code base and
design
2. Lack of design documents to remedy #1
3. Lack of unified end-user documentation

And there was no time for me to fix all of that before 2.5 came out,
although I did throw together the EggFormats documentation.  After that,
the time window where I was being paid (by OSAF) for setuptools
improvements came to an end, and other projects started taking precedence.

Since then, setuptools *itself* has become stable legacy code in much the
same way that the distutils has: pip, buildout, and virtualenv all built on
top of it, as it built on top of the distutils.  Problem #3 remains, but at
least now there are other people working on the codebase.

>   If the end goal is "the bulk of the setuptools feature set
> without the problematic features and default behaviours that make
> system administrators break out the torches and pitchforks", then we
> should *write that down* (and spell out the implications) rather than
> assuming that everyone knows the purpose of the exercise.

That's why I brought this up.  ISTM that far too much of the knowledge of
what those use cases and implications are, has been either buried in my
head or spread out among diverse user communities in the past.

Luckily, a lot of people from those communities are now getting
considerably more involved in this effort.  At the time of, say, the 2.5
setuptools question, there wasn't anybody around but me who was able to
argue the "why eggs are good and useful" side of the discussion, for
example.

(If you look back to the early days of setuptools, I often asked on
distutils-sig for people who could help assemble specs for various
things...  which I ended up just deciding for myself, because nobody was
there to comment on them.  It took *years* of setuptools actually being in
the field and used before enough people knew enough to *want* to take part
in the design discussions.  The versioning and metadata PEPs were things I
asked about many years prior, but nobody knew what they wanted yet, or even
knew yet why they should care.)

Similarly, in the years since then, MvL -- who originally argued against
all things setuptools at 2.5 time -- actually proposed the original
namespace package PEP.

So I don't think it's unfair to say that, seven years ago, the ideas in
setuptools were still a few years ahead of their "time".  Today, console
script generation, virtual environments, namespace packages, entry point
discovery, setup.py-driven testing tools, static file inclusion, etc. are
closer to "of course we should have that/everybody uses that" features,
rather than esoteric oddities.

That being said, setuptools *itself* is not such a good thing.  It was
originally a *private* add-on to distutils (like numpy's distutils
extensions) and a prototyping sandbox for additions to the distutils.
(E.g. setuptools features were added to distutils in 2.4 and 2.5.)  I
honestly didn't think at the time that I was writing those features (or
even the egg stuff), that the *long term* goal would be for those things to
be maintained in a separate package.

Instead, I (rather optimistically) assumed that the value of the approaches
would be self-evident, and copied the way the other setuptools features
were.  (To this day, there are an odd variety of other little experimental
"future distutils enhancements" still living in the setuptools code base,
like support for building shared libraries to be used in common between
multiple C extensions.)

By the way, for an overview of setuptools' components and use cases, and
what happened with 2.5, see here:

  http://mail.python.org/pipermail/python-dev/2006-April/064145.html

The plan I proposed was to phase out setuptools and merge its functionality
into distutils for 2.6, but as I mentioned above, my available bandwidth to
work on the project essentially vanished shortly after the above post;
setuptools was pretty much "good enough" for OSAF's needs at the time, and
they had other development priorities for my time.

So, if we are to draw any lesson from the past, it would seem to be, "make
sure that the people who'll be doing the work are actually going to be
available through to the next Python version".

After all, if they are not, it may not much matter whether the code is in
the stdlib or not.  ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120621/59143ba5/attachment.html>