[Distutils] People want CPAN :-)

[David Cournapeau]

Tarek Ziadé wrote:
> On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau <david at ar.media.kyoto-u.ac.jp>
> [..]
>>  - if you extend an existing command, you have to take care whether you run
>>   under setuptools or distutils (and soon distribute will make this worse).
>
> No, because unlike setuptools, we want to remove the patching that is done on
> the Command class and on the DIstribution class, and make Distribute a good
> Distutils citizen in that matter.

I don't think that's possible, not without major changes of the
distutils design at least. Right now, in numpy.distutils, we extend the
Distribution class to hold a few additional data. We inherit from
setuptools or distutils depending on whether setuptools has been
imported. The fundamental problem is not so much that setuptools does
monkey-patching, but that you need to inherit to extend. Once you have
several possible packages which inherit independently from those
classes, you have to special case each of them if you want to support
all of them.

>
>
>
> This is going to be changed because I am currently refactoring the
> build_ext command
> so such methods are in a base "compiler" class and eventually in util.py

Refactoring cannot help, unless you fundamentally change how it works. I
gave an example of a simple feature which is hard to do with current
distutils: obtaining the install prefix early in the build scheme. When
you do:

python setup.py install --prefix=foo

the build_* commands don't know prefix, only install knows it. In numpy,
I added various hack each more ugly than the other to fake running the
install command before running the build commands to obtain this
information. I would be surprised if subsequent refactor of distutils
will not break it.

> I agree the code is not modern, but things are getting changed through
> small steps.
>
> Although, I don't buy the "strange errors" part and things getting swallowed :)

I don't understand what's there to buy. Several people reported
distutils errors without any backtrace, though a fair shared of those
were caused by our own extensions.

Concerning changing into small steps: I understand that changing
gradually is often better than starting from scratch, but my own
experience with numpy.distutils and distutils convinced me that there is
not much to save from distutils code. I consider the API, the UI and the
implementation deeply flawed. For me, an  "improved backward compatible"
distutils is an oxymoron. More practically, I think the recent
distutils-related changes in 2.6.3 and 2.6.4  will keep happening. In
numpy.distutils, we depend a lot on internal details of distutils,
because we need things not available in the API, and I don't think we
are alone.

I would mention that numpy is one of the package which has the most
invested in distutils in terms of code (sloccount tells me that
numpy.distutils is 10315 LOC, whereas python 2.6.2 distutils is 10648
LOC), so I am certainly aware of the cost of breaking backward
compatibility.

> My opinion is that you've build something else when Distutils was not
> evolving anymore.

It has nothing to do with distutils evolving. We have had our own
extensions for years (I am a relatively newcommer, but some
numpy.distutils code goes back to 2001), so we could have done pretty
much what we wanted. We now have two build systems: one based on
distutils, one based on scons. Every-time I added a build feature to
numpy, it took me much more time to add it to distutils than to the
scons build. The code needed to support scons build is ~ 2000 lines of
code, is more robust, handle dependencies automatically, support
reliable parallel builds, and is more maintainable.

cheers,

David