On Fri, Sep 26, 2008 at 8:47 AM, Phillip J. Eby
At 03:24 PM 9/25/2008 -0700, Guido van Rossum wrote:
Right now there's a momentum in the community, including framework gurus, that are willing to work on a new distutils package. They are not core developers but they are really good in distribution matters. Even Phillip Eby said that starting a new distutils could be a good pick in this thread earlier.
I wasn't there. I'd like to refer to a post by Joel Spolsky about the problem with total rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html
The economic factors are a bit different, here. Joel himself has previously pointed out that where Netscape failed, Mozilla won - i.e., the economics of open source can mean that it's sometimes easier to get volunteers for a new project than for fixing an old one, or at least for a project where dropping backward compatibility is allowed (e.g. Py3K).
Yeah, but *is* dropping backward compatibility an option here?
In the case of the distutils, the people who are capable of extending it are tired of doing so, and the people who have the energy and time are very unlikely to be able to work on it much without breaking something significant.
The same might be happen to distutils 2 in a few years. You are going to have to plan making it more maintainable. It might also help if there was a team of people working on the replacement instead of a single one -- reduces the risk of that one person getting tired or engaged or whatever.
Distutils is also far too flexible in some areas to be able to improve much while maintaining 100% backward compatibility -- it doesn't enforce One Obvious Way To Do It.
That's a good criticism, and can be dealt with through careful deprecation.
What's more, there are very few people who've even said they like the distutils API or think it's a good fit for the application area. And, frankly, the domain knowledge embedded in the distutils are of fairly limited scope and kind:
* Extension building, compile/link options and defines * Wildly-differing installation path schemes across platforms * Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi
So maybe this part is worth salvaging?
Most other things the distutils do are either well-specified, obsolete (e.g. its internal logging and option parsing libs), or probably not worth keeping (e.g. bdist_dumb).
It's fine to gradually replace these with better stuff. Deprecated backward compatible APIs can be offered for a few Python releases.
Maybe a "distutils 2" project could start outside Python, using distutils and setuptools code as legacy infrastructure, and taking back pieces of it,
Then it could be re-integrated in Python as a replacement for distutils when it is mature ?
Only if much effort and study went into the planning of this re-integration.
That's why at least some of the discussion has been around requirements gathering and PEP-writing as a first step.
That's good to hear.
My own inclination is that a scalable future for distutils means an improved sdist format, the end of setup.py as an command-line interface, and community-maintained platform-specific installation tools that process source or binary distributions. Most complaints about distutils (and setuptools, for that matter) are focused on installation policy&preference issues. Making it possible and practical for a variety of tools to flourish around a standardized format (ala WSGI) seems like the way to go.
Given the success of WSGI (which I use every day) this sounds like a very good plan!
Notice that the existence of eggs has already allowed buildout, virtualenv, and pyinstall to appear. But eggs don't handle installing tests or documentation, and they have to be prebuilt by platform. An improved sdist format, on the other hand, with standardized layout and metadata would address all of those issues.
Beware however of too large a scope -- the project may crumble under its ambition, or be mired in border conflicts with language-independent distribution management tools like RPM or the Debian/Ubuntu tools.
The tools for building this format and APIs for inspecting it would be candidates for the stdlib, in much the same way that wsgiref was - the spec is/should be stable, and those parts that are compiler/install-layout specific will need to be maintained in the stdlib anyway, for Python's own build infrastructure.
Good thinking. This is also similar to the conclusion we came to with respect to NumPy -- NumPy itself is a separate package and will always remain so, but core Python contains some things that make this possible.
In that sense, "distutils 2" would not be so much a rewrite of the distutils, as the separation of them into tools for distributing, and tools for installing, where some of the tools for installing may be community-maintained.
That's the general idea, anyway.
Sounds good. -- --Guido van Rossum (home page: http://www.python.org/~guido/)