On Thu, Sep 25, 2008 at 1:59 PM, Tarek Ziadé wrote:

On Thu, Sep 25, 2008 at 7:36 PM, Guido van Rossum wrote:

...

I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years. There's also the issue that it's not enough to roll out the replacement with a new version of Python -- it would leave everyone using the old distutils in the lurch. It's a prime example of the network effect: until everyone owns one, nobody will want to own one. The best strategy I can think of would be to gradually rewrite the existing distutils, remaining compatible but also offering new APIs, and 5 years hence maybe the installed base will be close enough to 100% that developers writing setup scripts will start using the new features. That's how AJAX conquered the world.

It looks like you are describing setuptools in some ways: it was written on the top of distutils and provided enhancements and new features.

The mistake made was that setuptools didn't make it into the core distro. That in turn was because of concerns over its maintenance once incorporated into the core. (There's no need to go into more details.)

But in the meantime distutils itself did not evolve.

Because nobody offered to evolve it gradually (setuptools was an attempt at a quantum leap, which I was trying to explain is the wrong approach).

As far as I know, within the last year, one of the only new feature in distutils was the one I have proposed (multiple servers support in .pypirc). But it was really hard to get my change make it in because most Python core developers have a lack of interest for distutils. It took me 8 months iirc.

So you need to work yourself up to becoming a core developer. That's not impossible -- it just means contributing to other areas besides distutils.

Furthermore there are many things to change in this module to make it use modern Python before starting to add new features. (like logging usage but this is just an example)

The best thing I can recommend is to start by contributing relatively small cleanups, that are easy to review and apply.

My point is that I am afraid not much will happen if something doesn't start outside Python core.

My worry is that if something develops outside the core without careful thought about migration and compatibility it will be impossible to adapt into the core. A metaphor: You're not building a brand new bridge where there was no bridge before. You're trying to upgrade an existing bridge that people use for their daily commute. You can't even close a single lane during rush hour -- you have to plan carefully how the new bridge can grow organically right next to the old one and how traffic can be kept flowing continuously. Yes, it is much more expensive than just building a new bridge somewhere else. But it is the only way to do it. So you have to figure out how to do it.

Right now there's a momentum in the community, including framework gurus, that are willing to work on a new distutils package. They are not core developers but they are really good in distribution matters. Even Phillip Eby said that starting a new distutils could be a good pick in this thread earlier.

I wasn't there. I'd like to refer to a post by Joel Spolsky about the problem with total rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html

Maybe a "distutils 2" project could start outside Python, using distutils and setuptools code as legacy infrastructure, and taking back pieces of it,

Then it could be re-integrated in Python as a replacement for distutils when it is mature ?

Only if much effort and study went into the planning of this re-integration. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Uwe Schmitt wrote:

Hi,

is anybody aware of numscons ? http://conference.scipy.org/static/wiki/numscons.pdf

At least I am, since I am the main - and sole author at this point - of numscons :) Numscons was created to solve a specific problem: distutils is a nightmare for extensions which need to control the compilation process in a fine-grained manner. But numpy/scipy is quite special, and numscons would be overkill for most python packages. And it does not solve at all the problems people are talking about here (that is mostly deployment issues). The reasons why numscons was created are the same as why people want to get rid of distutils, though, that is distutils is very inflexible if you have special needs (and numpy/scipy needs are special; how many python packages need fortran ?), which is why I think it is somewhat relevant to this discussion. Numscons does show some points which are worth being pointed out IMHO: - it is possible to add a new command which handle a pretty big portion of distutils (the arcane part Guido was talking about I believe) in a sane manner, and it could be done gradually (scons is called from distutils, and scons is a distutils command: you do python setup.py scons to build your extensions, which means it is transparent for users at least). - It took me a few weeks to get simple python extensions built on most platforms (of which a big portion was to understand distutils and numpy.distutils). Of course, it would take more time to deal with all platforms python supports (more exactly all the corner cases), but I don't think it is that difficult. - A "miniscons" would be needed, but there are already many projects which do that, indicating that it should be relatively easy. The first rake prototype was 100 lines I believe; vellum is 300 lines. Something like vellum would be enough for most needs I believe. Of course, doing it for python itself means some issues which were non-existent in numpy's case become big problems (bootstrapping issues, for once). cheers, David

Guido van Rossum wrote:

Can you think of a specific way to evolve distutils to provide the API you would like to see, without (for the time being) losing backwards compatibility?

Here is what I did with numscons: numscons is called through distutils scons command, which is called during the build dance. By default, the scons command does nothing, but you add scons scripts through a distutils command, and in that case only scons does something. Those hooks do not break anything in numpy.distutils. The best proof is that the default build system is still numpy.distutils, and numscons is not in numpy sources (only the hooks are). Some people build with numscons, other - the majority - use the current system. Same source tree, same setup.py files (except for C extensions of course; they have to be synchronised, but the common part between the two build could easily be factorized in a common_setup I think). That's the idea I was suggesting and may be applicable to python as a whole: - design a small self contained build system, which does nothing but is extensible (there is a lot of prior art: vellum, rake/rant, etc...), and "hook" it into distutils through a new distutils command. People smarter than me will enjoy this part, and will hopefully come up with something nice. - The new command would not nothing if you don't use the new features: you don't break any existing setup.py - Another thing would be to gradually move all the configuration from code into a separate module in a sane way: putting the flags in config files, not in the code, make an API to discover and reuse those settings, etc... Basically, all the arcane stuff. This would be independant of distutils, but would be callable from distutils (by the new command and the current distutils). A good way to design the API here would be to make sure it work in distutils and in e.g. scons (so a python extension builder in scons could be easily done importing this module only: all the compiler specific flags, etc...). It would be a distutils.sysconfig on steroids, which works for every platform - including windows with VS. So that every other python build tool, being vellum, scons, whatever, could access those. I believe this can be made general enough to be usable by almost any tool. At least I started something toward this direction for scons, and if it works for scons, it should work for a lot of tools. That would enable refactoring something like 1/3 of current distutils code, and provide a better build tool for people who use the new 'stuff'. I realize this is really vague at that point, but up to today, I never thought about what could be applied to python in general from my work numscons. If you think the above makes any sense, I will think more about this, cheers, David