[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

[Ned Deily]

In article 
<94bdd2610910080131j323b98d9i871bce43465f237a at mail.gmail.com>,
 Tarek Ziadé <ziade.tarek at gmail.com> wrote:
> Here's a quick summary of the main things that are going to happen in
> Distutils, and Distribute, and a few words on virtualenv and pip.
> (there is much much more work going on, but I don't want to drown
> people with details)

Thanks for the summary!   Unfortunately, as the saying goes, the devil 
is in the details.  Here are a few comments.

> = Distribute =
> 
> I won't explain here again why we have forked, I think it's obvious to
> anyone here now. I'll rather explain what
> we are planning in Distribute and how it will interact with Distutils.
> 
> Distribute has two branches:
> 
> - 0.6.x : provides a Setuptools-0.6c9 compatible version
> - 0.7.x : will provide a refactoring
> 
> == 0.6.x ==
> 
> Not "much" is going to happen here, we want this branch to be helpful
> to the community *today* by addressing the 40-or-so bugs
> that were found in Setuptools and never fixed. This is eventually
> happen soon because its development is
> fast : there are up to 5 commiters that are working on it very often
> (and the number grows weekly.)
> 
> The biggest issue with this branch is that it is providing the same
> packages and modules setuptools does, and this
> requires some bootstrapping work where we make sure once Distribute is
> installed, all Distribution that requires Setuptools
> will continue to work. This is done by faking the metadata of
> Setuptools 0.6c9. That's the only way we found to do this.
> 
> There's one major thing though: thanks to the work of Lennart, Alex,
> Martin, this branch supports Python 3,
> which is great to have to speed up Py3 adoption.
> 
> The goal of the 0.6.x is to remove as much bugs as we can, and try if
> possible to remove the patches done
> on Distutils. We will support 0.6.x maintenance for years and we will
> promote its usage everywhere instead of
> Setuptools.

This is a key and, I think, difficult issue that's going to need a lot 
of thought and work to be successful.  setuptools / easy_install filled 
several unmet needs of both package developers and end users and became 
a de-facto standard because there was essentially no competition.  So, 
in a sense, there was never a migration to setuptools from something.  
But now, especially given GvR's blessing of Distribute, there needs to 
be a migration from setuptools to Distribute, although neither are part 
of the standard library.  There are no references to setuptools or 
easy_install in any of the standard Python documentation.

So how do end users know about easy_install?  Their first encounter 
is/was probably through some recipe on a web site, most likely the 
installation instructions for some package they want to use.  And I 
suspect pretty much all of those installation instructions direct the 
user to one of two places, either the PyPi page for setuptools or the 
peakcommunity wiki page:

   http://pypi.python.org/pypi/setuptools
   http://peak.telecommunity.com/DevCenter/EasyInstall

Right now those are essentially the ultimate contact points for users.  
A general Google seach for easy_install gets 357,000 hits; one 
restricted to pypi.python.org gets 2,500.   I don't know how many of the 
latter are active packages on PyPi but certainly there must be hundreds 
that currently use setuptools and document the use of easy_install for 
their users.  Those packages' documentation often directs users to 
download the ez_install script to bootstrap setuptools; I believe there 
are some packages that download it automatically, if necessary.

So, without some other intervention, users will continue to follow the 
instructions with those hundreds of packages (or on those thousands of 
websites) and manually or automatically try to install setuptools.  
Which may or may not work - as witnessed with the release of 2.6.3.

Brett and others suggested including some sort of warnings in the python 
release documentation.  Is there consensus that should be done?  
Starting with 2.6.4?   Seems like that would be up to Barry and Guido to 
pronounce.

Ultimately, to successfully migrate to Distribute is going to require a 
good understanding of the use cases for easy_install (primarily those of 
end users) and for the setuptools API (those of package developers).  
Those use cases should be documented somehow, either in the roadmap or a 
reference elsewhere.  This is especially true if your intentions are to 
deprecate the easy_install command in Distribute 0.7 and if APIs are 
going to change for package developers.  As Antoine points out, pip is 
not equivalent to easy_install.  BTW, keep in mind that today people 
also use easy_install to manage the installation of packages that do not 
use setuptools.

It will also be important to understand and address the major 
distribution channels for Python installations, many of which already 
include setuptools in their distributions.  There are the multi-platform 
distributors, like python.org (which does not currently distribute 
setuptools), ActiveState, and Enthought.  You've already mentioned some 
of the Unix-y O/S ones: Debian, Fedora, et al.  On the OS X side, 
there's Apple and the major third-party providers of open-source 
packages, MacPorts and Fink - they all distribute setuptools.  I don't 
have personal experience with the Windows world to know what all the 
channels are there.  Are there other operating systems who package 
Python and possibly setuptools?  They, one way or another, need to know 
and have some sort of plan going forward for their distributions 
regarding setuptools and Distribute.

And, of course, the stickiest issues in all this are not technical, but 
rather social.  For most of this, the most important part will likely be 
giving a clear direction to all of the major groups involved: end users, 
package developers, distributors.  How to decide on and then effectively 
communicate that direction is not at all trivial, I think.

Thanks again for your hard work on this, Tarek!

-- 
 Ned Deily,
 nad at acm.org