[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)
nad at acm.org
Thu Oct 8 23:28:47 CEST 2009
<94bdd2610910080131j323b98d9i871bce43465f237a at mail.gmail.com>,
Tarek Ziadé <ziade.tarek at gmail.com> wrote:
> Here's a quick summary of the main things that are going to happen in
> Distutils, and Distribute, and a few words on virtualenv and pip.
> (there is much much more work going on, but I don't want to drown
> people with details)
Thanks for the summary! Unfortunately, as the saying goes, the devil
is in the details. Here are a few comments.
> = Distribute =
> I won't explain here again why we have forked, I think it's obvious to
> anyone here now. I'll rather explain what
> we are planning in Distribute and how it will interact with Distutils.
> Distribute has two branches:
> - 0.6.x : provides a Setuptools-0.6c9 compatible version
> - 0.7.x : will provide a refactoring
> == 0.6.x ==
> Not "much" is going to happen here, we want this branch to be helpful
> to the community *today* by addressing the 40-or-so bugs
> that were found in Setuptools and never fixed. This is eventually
> happen soon because its development is
> fast : there are up to 5 commiters that are working on it very often
> (and the number grows weekly.)
> The biggest issue with this branch is that it is providing the same
> packages and modules setuptools does, and this
> requires some bootstrapping work where we make sure once Distribute is
> installed, all Distribution that requires Setuptools
> will continue to work. This is done by faking the metadata of
> Setuptools 0.6c9. That's the only way we found to do this.
> There's one major thing though: thanks to the work of Lennart, Alex,
> Martin, this branch supports Python 3,
> which is great to have to speed up Py3 adoption.
> The goal of the 0.6.x is to remove as much bugs as we can, and try if
> possible to remove the patches done
> on Distutils. We will support 0.6.x maintenance for years and we will
> promote its usage everywhere instead of
This is a key and, I think, difficult issue that's going to need a lot
of thought and work to be successful. setuptools / easy_install filled
several unmet needs of both package developers and end users and became
a de-facto standard because there was essentially no competition. So,
in a sense, there was never a migration to setuptools from something.
But now, especially given GvR's blessing of Distribute, there needs to
be a migration from setuptools to Distribute, although neither are part
of the standard library. There are no references to setuptools or
easy_install in any of the standard Python documentation.
So how do end users know about easy_install? Their first encounter
is/was probably through some recipe on a web site, most likely the
installation instructions for some package they want to use. And I
suspect pretty much all of those installation instructions direct the
user to one of two places, either the PyPi page for setuptools or the
peakcommunity wiki page:
Right now those are essentially the ultimate contact points for users.
A general Google seach for easy_install gets 357,000 hits; one
restricted to pypi.python.org gets 2,500. I don't know how many of the
latter are active packages on PyPi but certainly there must be hundreds
that currently use setuptools and document the use of easy_install for
their users. Those packages' documentation often directs users to
download the ez_install script to bootstrap setuptools; I believe there
are some packages that download it automatically, if necessary.
So, without some other intervention, users will continue to follow the
instructions with those hundreds of packages (or on those thousands of
websites) and manually or automatically try to install setuptools.
Which may or may not work - as witnessed with the release of 2.6.3.
Brett and others suggested including some sort of warnings in the python
release documentation. Is there consensus that should be done?
Starting with 2.6.4? Seems like that would be up to Barry and Guido to
Ultimately, to successfully migrate to Distribute is going to require a
good understanding of the use cases for easy_install (primarily those of
end users) and for the setuptools API (those of package developers).
Those use cases should be documented somehow, either in the roadmap or a
reference elsewhere. This is especially true if your intentions are to
deprecate the easy_install command in Distribute 0.7 and if APIs are
going to change for package developers. As Antoine points out, pip is
not equivalent to easy_install. BTW, keep in mind that today people
also use easy_install to manage the installation of packages that do not
It will also be important to understand and address the major
distribution channels for Python installations, many of which already
include setuptools in their distributions. There are the multi-platform
distributors, like python.org (which does not currently distribute
setuptools), ActiveState, and Enthought. You've already mentioned some
of the Unix-y O/S ones: Debian, Fedora, et al. On the OS X side,
there's Apple and the major third-party providers of open-source
packages, MacPorts and Fink - they all distribute setuptools. I don't
have personal experience with the Windows world to know what all the
channels are there. Are there other operating systems who package
Python and possibly setuptools? They, one way or another, need to know
and have some sort of plan going forward for their distributions
regarding setuptools and Distribute.
And, of course, the stickiest issues in all this are not technical, but
rather social. For most of this, the most important part will likely be
giving a clear direction to all of the major groups involved: end users,
package developers, distributors. How to decide on and then effectively
communicate that direction is not at all trivial, I think.
Thanks again for your hard work on this, Tarek!
nad at acm.org
More information about the Python-Dev