On Wed, Jun 20, 2012 at 11:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
>
> Right - clearly enumerating the features that draw people to use
> setuptools over just using distutils should be a key element in any
> PEP for 3.4
>
> I honestly think a big part of why packaging ended up being incomplete
> for 3.3 is that we still don't have a clearly documented answer to two
> critical questions:
> 1. Why do people choose setuptools over distutils?

Some of the reasons:

* Dependencies
* Namespace packages
* Less boilerplate in setup.py (revision control, data files support, find_packages(), etc.)
* Entry points system for creating extensible applications and frameworks that need runtime plugin discovery
* Command-line script wrappers
* Binary plugin installation system for apps (i.e. dump eggs in a directory and let pkg_resources figure out what to put on sys.path)
* "Test" command
* Easy distribution of (and runtime access to) static data resources

Of these, automatic dependency resolution with as close to 100% backward compatibility for installing other projects on PyPI was almost certainly the #1 factor driving setuptools' initial adoption.  The 20% that drives the 80%, as it were.  The rest are the 80% that brings in the remaining 20%.
 
>
> 2. What's wrong with setuptools that meant the idea of including it
> directly in the stdlib was ultimately dropped and eventually replaced
> with the goal of incorporating distutils2?

Based on the feedback from Python-Dev, I withdrew setuptools from 2.5 because of what I considered valid concerns raised regarding:

1. Lack of available persons besides myself familiar with the code base and design
2. Lack of design documents to remedy #1
3. Lack of unified end-user documentation

And there was no time for me to fix all of that before 2.5 came out, although I did throw together the EggFormats documentation.  After that, the time window where I was being paid (by OSAF) for setuptools improvements came to an end, and other projects started taking precedence.

Since then, setuptools *itself* has become stable legacy code in much the same way that the distutils has: pip, buildout, and virtualenv all built on top of it, as it built on top of the distutils.  Problem #3 remains, but at least now there are other people working on the codebase.

>   If the end goal is "the bulk of the setuptools feature set
> without the problematic features and default behaviours that make
> system administrators break out the torches and pitchforks", then we
> should *write that down* (and spell out the implications) rather than
> assuming that everyone knows the purpose of the exercise.

That's why I brought this up.  ISTM that far too much of the knowledge of what those use cases and implications are, has been either buried in my head or spread out among diverse user communities in the past.

Luckily, a lot of people from those communities are now getting considerably more involved in this effort.  At the time of, say, the 2.5 setuptools question, there wasn't anybody around but me who was able to argue the "why eggs are good and useful" side of the discussion, for example.

(If you look back to the early days of setuptools, I often asked on distutils-sig for people who could help assemble specs for various things...  which I ended up just deciding for myself, because nobody was there to comment on them.  It took *years* of setuptools actually being in the field and used before enough people knew enough to *want* to take part in the design discussions.  The versioning and metadata PEPs were things I asked about many years prior, but nobody knew what they wanted yet, or even knew yet why they should care.)

Similarly, in the years since then, MvL -- who originally argued against all things setuptools at 2.5 time -- actually proposed the original namespace package PEP.

So I don't think it's unfair to say that, seven years ago, the ideas in setuptools were still a few years ahead of their "time".  Today, console script generation, virtual environments, namespace packages, entry point discovery, setup.py-driven testing tools, static file inclusion, etc. are closer to "of course we should have that/everybody uses that" features, rather than esoteric oddities.

That being said, setuptools *itself* is not such a good thing.  It was originally a *private* add-on to distutils (like numpy's distutils extensions) and a prototyping sandbox for additions to the distutils.  (E.g. setuptools features were added to distutils in 2.4 and 2.5.)  I honestly didn't think at the time that I was writing those features (or even the egg stuff), that the *long term* goal would be for those things to be maintained in a separate package.

Instead, I (rather optimistically) assumed that the value of the approaches would be self-evident, and copied the way the other setuptools features were.  (To this day, there are an odd variety of other little experimental "future distutils enhancements" still living in the setuptools code base, like support for building shared libraries to be used in common between multiple C extensions.)

By the way, for an overview of setuptools' components and use cases, and what happened with 2.5, see here:

  http://mail.python.org/pipermail/python-dev/2006-April/064145.html

The plan I proposed was to phase out setuptools and merge its functionality into distutils for 2.6, but as I mentioned above, my available bandwidth to work on the project essentially vanished shortly after the above post; setuptools was pretty much "good enough" for OSAF's needs at the time, and they had other development priorities for my time.

So, if we are to draw any lesson from the past, it would seem to be, "make sure that the people who'll be doing the work are actually going to be available through to the next Python version".

After all, if they are not, it may not much matter whether the code is in the stdlib or not.  ;-)