Re: [Distutils] [Fwd: Re: Annoucing distribute project]

I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years. There's also the issue that it's not enough to roll out the replacement with a new version of Python -- it would leave everyone using the old distutils in the lurch. It's a prime example of the network effect: until everyone owns one, nobody will want to own one. The best strategy I can think of would be to gradually rewrite the existing distutils, remaining compatible but also offering new APIs, and 5 years hence maybe the installed base will be close enough to 100% that developers writing setup scripts will start using the new features. That's how AJAX conquered the world. --Guido On Thu, Sep 25, 2008 at 10:18 AM, Blake Winton <bwinton@latte.ca> wrote:
Just as a question, is there _any_ chance of a new project to replace distutils being accepted into Python?
Thanks, Blake.
-------- Original Message -------- Subject: Re: [Distutils] Annoucing distribute project Date: Thu, 25 Sep 2008 12:55:36 -0400 From: Jim Fulton <jim@zope.com> To: Andreas Jung <lists@zopyx.com> CC: distutils-sig@python.org References: <94bdd2610809240432i307ed937rbfb1fd063cec915b@mail.gmail.com> <20080924173708.13DA93A4128@sparrow.telecommunity.com> <94bdd2610809241244p13ad74dbkc13ee5b3ae3e14c8@mail.gmail.com> <48DAA90C.20504@free.fr> <20080924213342.612CC3A4127@sparrow.telecommunity.com> <94bdd2610809241456o20dade8bl4b46f3e0ed7aeb90@mail.gmail.com> <20080924223457.2BD7E3A4127@sparrow.telecommunity.com> <94bdd2610809242252j131d9fbcyd3e629667b455d29@mail.gmail.com> <20080925163034.208613A4072@sparrow.telecommunity.com> <C0E161B65AAB12A92255EE01@suxmac.local>
On Sep 25, 2008, at 12:46 PM, Andreas Jung wrote:
--On 25. September 2008 12:31:40 -0400 "Phillip J. Eby" <pje@telecommunity.com > wrote:
Personally, I would rather see not a "fork", but development of an entirely new disutils replacement, and then not worry at all about backward compatibility at the setup.py level. Apart from the pkg_resources module, I would treat setuptools and distutils as legacy infrastructure, and explore better ways to build and manage distributions (including eggs).
We can not restart with something completely different or new. Such an approach would be totally unacceptable due to the huge amount of existing modules having switched to setuptools. Setuptools or an improved codefork must be the tools for managing distibutions in Python.
I would be willing to start with something new if and only if it was adopted as part of Python.
Jim
-- Jim Fulton Zope Corporation
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Sep 25, 2008 at 7:36 PM, Guido van Rossum <guido@python.org> wrote:
I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years. There's also the issue that it's not enough to roll out the replacement with a new version of Python -- it would leave everyone using the old distutils in the lurch. It's a prime example of the network effect: until everyone owns one, nobody will want to own one. The best strategy I can think of would be to gradually rewrite the existing distutils, remaining compatible but also offering new APIs, and 5 years hence maybe the installed base will be close enough to 100% that developers writing setup scripts will start using the new features. That's how AJAX conquered the world.
It looks like you are describing setuptools in some ways: it was written on the top of distutils and provided enhancements and new features. But in the meantime distutils itself did not evolve. As far as I know, within the last year, one of the only new feature in distutils was the one I have proposed (multiple servers support in .pypirc). But it was really hard to get my change make it in because most Python core developers have a lack of interest for distutils. It took me 8 months iirc. Furthermore there are many things to change in this module to make it use modern Python before starting to add new features. (like logging usage but this is just an example) My point is that I am afraid not much will happen if something doesn't start outside Python core. Right now there's a momentum in the community, including framework gurus, that are willing to work on a new distutils package. They are not core developers but they are really good in distribution matters. Even Phillip Eby said that starting a new distutils could be a good pick in this thread earlier. Maybe a "distutils 2" project could start outside Python, using distutils and setuptools code as legacy infrastructure, and taking back pieces of it, Then it could be re-integrated in Python as a replacement for distutils when it is mature ? My 2 cents :) Tarek

On Thu, Sep 25, 2008 at 1:59 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Thu, Sep 25, 2008 at 7:36 PM, Guido van Rossum <guido@python.org> wrote:
I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years. There's also the issue that it's not enough to roll out the replacement with a new version of Python -- it would leave everyone using the old distutils in the lurch. It's a prime example of the network effect: until everyone owns one, nobody will want to own one. The best strategy I can think of would be to gradually rewrite the existing distutils, remaining compatible but also offering new APIs, and 5 years hence maybe the installed base will be close enough to 100% that developers writing setup scripts will start using the new features. That's how AJAX conquered the world.
It looks like you are describing setuptools in some ways: it was written on the top of distutils and provided enhancements and new features.
The mistake made was that setuptools didn't make it into the core distro. That in turn was because of concerns over its maintenance once incorporated into the core. (There's no need to go into more details.)
But in the meantime distutils itself did not evolve.
Because nobody offered to evolve it gradually (setuptools was an attempt at a quantum leap, which I was trying to explain is the wrong approach).
As far as I know, within the last year, one of the only new feature in distutils was the one I have proposed (multiple servers support in .pypirc). But it was really hard to get my change make it in because most Python core developers have a lack of interest for distutils. It took me 8 months iirc.
So you need to work yourself up to becoming a core developer. That's not impossible -- it just means contributing to other areas besides distutils.
Furthermore there are many things to change in this module to make it use modern Python before starting to add new features. (like logging usage but this is just an example)
The best thing I can recommend is to start by contributing relatively small cleanups, that are easy to review and apply.
My point is that I am afraid not much will happen if something doesn't start outside Python core.
My worry is that if something develops outside the core without careful thought about migration and compatibility it will be impossible to adapt into the core. A metaphor: You're not building a brand new bridge where there was no bridge before. You're trying to upgrade an existing bridge that people use for their daily commute. You can't even close a single lane during rush hour -- you have to plan carefully how the new bridge can grow organically right next to the old one and how traffic can be kept flowing continuously. Yes, it is much more expensive than just building a new bridge somewhere else. But it is the only way to do it. So you have to figure out how to do it.
Right now there's a momentum in the community, including framework gurus, that are willing to work on a new distutils package. They are not core developers but they are really good in distribution matters. Even Phillip Eby said that starting a new distutils could be a good pick in this thread earlier.
I wasn't there. I'd like to refer to a post by Joel Spolsky about the problem with total rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html
Maybe a "distutils 2" project could start outside Python, using distutils and setuptools code as legacy infrastructure, and taking back pieces of it,
Then it could be re-integrated in Python as a replacement for distutils when it is mature ?
Only if much effort and study went into the planning of this re-integration. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
You're not building a brand new bridge where there was no bridge before. You're trying to upgrade an existing bridge that people use for their daily commute.
But that hasn't been decided. Whether to upgrade the existing bridge or build a new one is precisely what we're debating. And it seems to me that the easiest way to avoid disrupting existing traffic *is* to build a new bridge alongside and then scrap the old one. If the existing bridge isn't too badly run down and is basically still the kind of bridge you want, upgrading it may make sense. But that doesn't seem to be the case with the distutils bridge. -- Greg

Guido van Rossum wrote:
I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years.
Well, yes and no. Yes, there is some arcane knowledge in distutils, but the information is already there. Incidentally, a problem with distutils is that it is very difficult to reuse this knowlege in a programmatic way. For example, my first 'real' contact with distutils was to try extending it to build ctypes extensions (that is pure C dynamically loaded libraries loadable from ctypes). It should have been easy, since from a compile/build POV, that's the same than a C python extension. Unfortunately, the options are buried into the code, and worse, their location depend on the used platform/compiler (for example, distutils.sysconfig is not usable with Visual Studio). At the end, I gave up, doing it in a cross-platform way without copying all the knowledge from distutils was too difficult (for me at least). I was fed up with distutils complexity in numpy/scipy, so I started using scons within distutils to build all compiled code in numpy/scipy. I spent more time dealing with distutils idiosyncraties than writing the core new system. I am certainly not advocating using scons, but the fact is that in a couple of months, I, a not that knowledgable person about python and cross-platforms issues at that point, manage to build numpy and scipy without using distutils *at all* for C/C++/Fortran extensions. And numpy is certainly far from trivial to build (numpy distutils extensions is ~ 10 kloc), and deal with quite a variety of compilers and platforms. Maybe there is something to be learnt here: we could have a new small subproject to deal with building extensions, and plug it into distutils (e.g. have an extremely simplified scons; scons is overkill for the vast majority of python packages, and certainly not integrable in python core; several small projects in that space already exist). Something like rake; the arcane knowledge (for compiled extensions at least) could be moved gradually from distutils to this submodule. Would that fit into your vision or not ? cheers, David

Hi, is anybody aware of numscons ? http://conference.scipy.org/static/wiki/numscons.pdf I thinke they have some good ideas, maybe those ideas help. Greetings, Uwe -- Dr. rer. nat. Uwe Schmitt F&E Mathematik mineway GmbH Science Park 2 D-66123 Saarbrücken Telefon: +49 (0)681 8390 5334 Telefax: +49 (0)681 830 4376 uschmitt@mineway.de www.mineway.de Geschäftsführung: Dr.-Ing. Mathias Bauer Amtsgericht Saarbrücken HRB 12339

Uwe Schmitt wrote:
Hi,
is anybody aware of numscons ? http://conference.scipy.org/static/wiki/numscons.pdf
At least I am, since I am the main - and sole author at this point - of numscons :) Numscons was created to solve a specific problem: distutils is a nightmare for extensions which need to control the compilation process in a fine-grained manner. But numpy/scipy is quite special, and numscons would be overkill for most python packages. And it does not solve at all the problems people are talking about here (that is mostly deployment issues). The reasons why numscons was created are the same as why people want to get rid of distutils, though, that is distutils is very inflexible if you have special needs (and numpy/scipy needs are special; how many python packages need fortran ?), which is why I think it is somewhat relevant to this discussion. Numscons does show some points which are worth being pointed out IMHO: - it is possible to add a new command which handle a pretty big portion of distutils (the arcane part Guido was talking about I believe) in a sane manner, and it could be done gradually (scons is called from distutils, and scons is a distutils command: you do python setup.py scons to build your extensions, which means it is transparent for users at least). - It took me a few weeks to get simple python extensions built on most platforms (of which a big portion was to understand distutils and numpy.distutils). Of course, it would take more time to deal with all platforms python supports (more exactly all the corner cases), but I don't think it is that difficult. - A "miniscons" would be needed, but there are already many projects which do that, indicating that it should be relatively easy. The first rake prototype was 100 lines I believe; vellum is 300 lines. Something like vellum would be enough for most needs I believe. Of course, doing it for python itself means some issues which were non-existent in numpy's case become big problems (bootstrapping issues, for once). cheers, David

On Thu, Sep 25, 2008 at 7:53 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Guido van Rossum wrote:
I don't think anyone loves distutils, or even likes it. Unfortunately replacing it will be very tough, because distutils has arcane but important knowledge built into it about many platforms, and a new from-scratch system is unlikely to reach that level of coverage for years.
Well, yes and no. Yes, there is some arcane knowledge in distutils, but the information is already there. Incidentally, a problem with distutils is that it is very difficult to reuse this knowlege in a programmatic way. For example, my first 'real' contact with distutils was to try extending it to build ctypes extensions (that is pure C dynamically loaded libraries loadable from ctypes). It should have been easy, since from a compile/build POV, that's the same than a C python extension. Unfortunately, the options are buried into the code, and worse, their location depend on the used platform/compiler (for example, distutils.sysconfig is not usable with Visual Studio). At the end, I gave up, doing it in a cross-platform way without copying all the knowledge from distutils was too difficult (for me at least).
I was fed up with distutils complexity in numpy/scipy, so I started using scons within distutils to build all compiled code in numpy/scipy. I spent more time dealing with distutils idiosyncraties than writing the core new system. I am certainly not advocating using scons, but the fact is that in a couple of months, I, a not that knowledgable person about python and cross-platforms issues at that point, manage to build numpy and scipy without using distutils *at all* for C/C++/Fortran extensions. And numpy is certainly far from trivial to build (numpy distutils extensions is ~ 10 kloc), and deal with quite a variety of compilers and platforms.
Maybe there is something to be learnt here: we could have a new small subproject to deal with building extensions, and plug it into distutils (e.g. have an extremely simplified scons; scons is overkill for the vast majority of python packages, and certainly not integrable in python core; several small projects in that space already exist). Something like rake; the arcane knowledge (for compiled extensions at least) could be moved gradually from distutils to this submodule. Would that fit into your vision or not ?
Can you think of a specific way to evolve distutils to provide the API you would like to see, without (for the time being) losing backwards compatibility? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Can you think of a specific way to evolve distutils to provide the API you would like to see, without (for the time being) losing backwards compatibility?
Here is what I did with numscons: numscons is called through distutils scons command, which is called during the build dance. By default, the scons command does nothing, but you add scons scripts through a distutils command, and in that case only scons does something. Those hooks do not break anything in numpy.distutils. The best proof is that the default build system is still numpy.distutils, and numscons is not in numpy sources (only the hooks are). Some people build with numscons, other - the majority - use the current system. Same source tree, same setup.py files (except for C extensions of course; they have to be synchronised, but the common part between the two build could easily be factorized in a common_setup I think). That's the idea I was suggesting and may be applicable to python as a whole: - design a small self contained build system, which does nothing but is extensible (there is a lot of prior art: vellum, rake/rant, etc...), and "hook" it into distutils through a new distutils command. People smarter than me will enjoy this part, and will hopefully come up with something nice. - The new command would not nothing if you don't use the new features: you don't break any existing setup.py - Another thing would be to gradually move all the configuration from code into a separate module in a sane way: putting the flags in config files, not in the code, make an API to discover and reuse those settings, etc... Basically, all the arcane stuff. This would be independant of distutils, but would be callable from distutils (by the new command and the current distutils). A good way to design the API here would be to make sure it work in distutils and in e.g. scons (so a python extension builder in scons could be easily done importing this module only: all the compiler specific flags, etc...). It would be a distutils.sysconfig on steroids, which works for every platform - including windows with VS. So that every other python build tool, being vellum, scons, whatever, could access those. I believe this can be made general enough to be usable by almost any tool. At least I started something toward this direction for scons, and if it works for scons, it should work for a lot of tools. That would enable refactoring something like 1/3 of current distutils code, and provide a better build tool for people who use the new 'stuff'. I realize this is really vague at that point, but up to today, I never thought about what could be applied to python in general from my work numscons. If you think the above makes any sense, I will think more about this, cheers, David
participants (5)
-
David Cournapeau
-
Greg Ewing
-
Guido van Rossum
-
Tarek Ziadé
-
Uwe Schmitt