[Distutils] [Fwd: Re: Annoucing distribute project]

Guido van Rossum guido at python.org
Fri Sep 26 18:55:28 CEST 2008

On Fri, Sep 26, 2008 at 8:47 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 03:24 PM 9/25/2008 -0700, Guido van Rossum wrote:
>> > Right now there's a momentum in the community, including framework
>> > gurus,
>> > that are
>> > willing to work on a new distutils package. They are not core developers
>> > but
>> > they are really
>> > good in distribution matters. Even Phillip Eby said that starting a new
>> > distutils could be a good
>> > pick in this thread earlier.
>> I wasn't there. I'd like to refer to a post by Joel Spolsky about the
>> problem with total rewrites:
>> http://www.joelonsoftware.com/articles/fog0000000069.html
> The economic factors are a bit different, here.  Joel himself has previously
> pointed out that where Netscape failed, Mozilla won - i.e., the economics of
> open source can mean that it's sometimes easier to get volunteers for a new
> project than for fixing an old one, or at least for a project where dropping
> backward compatibility is allowed (e.g. Py3K).

Yeah, but *is* dropping backward compatibility an option here?

> In the case of the distutils, the people who are capable of extending it are
> tired of doing so, and the people who have the energy and time are very
> unlikely to be able to work on it much without breaking something
> significant.

The same might be happen to distutils 2 in a few years. You are going
to have to plan making it more maintainable. It might also help if
there was a team of people working on the replacement instead of a
single one -- reduces the risk of that one person getting tired or
engaged or whatever.

> Distutils is also far too flexible in some areas to be able to
> improve much while maintaining 100% backward compatibility -- it doesn't
> enforce One Obvious Way To Do It.

That's a good criticism, and can be dealt with through careful deprecation.

> What's more, there are very few people who've even said they like the
> distutils API or think it's a good fit for the application area.  And,
> frankly, the domain knowledge embedded in the distutils are of fairly
> limited scope and kind:
> * Extension building, compile/link options and defines
> * Wildly-differing installation path schemes across platforms
> * Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi

So maybe this part is worth salvaging?

> Most other things the distutils do are either well-specified, obsolete (e.g.
> its internal logging and option parsing libs), or probably not worth keeping
> (e.g. bdist_dumb).

It's fine to gradually replace these with better stuff. Deprecated
backward compatible APIs can be offered for a few Python releases.

>> > Maybe a "distutils 2" project could start outside Python, using
>> > distutils
>> > and setuptools code as
>> > legacy infrastructure, and taking back pieces of it,
>> >
>> > Then it could be re-integrated in Python as a replacement for distutils
>> > when
>> > it is mature ?
>> Only if much effort and study went into the planning of this
>> re-integration.
> That's why at least some of the discussion has been around requirements
> gathering and PEP-writing as a first step.

That's good to hear.

> My own inclination is that a scalable future for distutils means an improved
> sdist format, the end of setup.py as an command-line interface, and
> community-maintained platform-specific installation tools that process
> source or binary distributions.  Most complaints about distutils (and
> setuptools, for that matter) are focused on installation policy&preference
> issues.  Making it possible and practical for a variety of tools to flourish
> around a standardized format (ala WSGI) seems like the way to go.

Given the success of WSGI (which I use every day) this sounds like a
very good plan!

> Notice that the existence of eggs has already allowed buildout, virtualenv,
> and pyinstall to appear.  But eggs don't handle installing tests or
> documentation, and they have to be prebuilt by platform.  An improved sdist
> format, on the other hand, with standardized layout and metadata would
> address all of those issues.

Beware however of too large a scope -- the project may crumble under
its ambition, or be mired in border conflicts with
language-independent distribution management tools like RPM or the
Debian/Ubuntu tools.

> The tools for building this format and APIs for inspecting it would be
> candidates for the stdlib, in much the same way that wsgiref was - the spec
> is/should be stable, and those parts that are compiler/install-layout
> specific will need to be maintained in the stdlib anyway, for Python's own
> build infrastructure.

Good thinking. This is also similar to the conclusion we came to with
respect to NumPy -- NumPy itself is a separate package and will always
remain so, but core Python contains some things that make this

> In that sense, "distutils 2" would not be so much a rewrite of the
> distutils, as the separation of them into tools for distributing, and tools
> for installing, where some of the tools for installing may be
> community-maintained.
> That's the general idea, anyway.

Sounds good.

--Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Distutils-SIG mailing list