Tarek Ziadé wrote:
On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> [..]
- if you extend an existing command, you have to take care whether you run under setuptools or distutils (and soon distribute will make this worse).
No, because unlike setuptools, we want to remove the patching that is done on the Command class and on the DIstribution class, and make Distribute a good Distutils citizen in that matter.
I don't think that's possible, not without major changes of the distutils design at least. Right now, in numpy.distutils, we extend the Distribution class to hold a few additional data. We inherit from setuptools or distutils depending on whether setuptools has been imported. The fundamental problem is not so much that setuptools does monkey-patching, but that you need to inherit to extend. Once you have several possible packages which inherit independently from those classes, you have to special case each of them if you want to support all of them.
This is going to be changed because I am currently refactoring the build_ext command so such methods are in a base "compiler" class and eventually in util.py
Refactoring cannot help, unless you fundamentally change how it works. I gave an example of a simple feature which is hard to do with current distutils: obtaining the install prefix early in the build scheme. When you do: python setup.py install --prefix=foo the build_* commands don't know prefix, only install knows it. In numpy, I added various hack each more ugly than the other to fake running the install command before running the build commands to obtain this information. I would be surprised if subsequent refactor of distutils will not break it.
I agree the code is not modern, but things are getting changed through small steps.
Although, I don't buy the "strange errors" part and things getting swallowed :)
I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. Concerning changing into small steps: I understand that changing gradually is often better than starting from scratch, but my own experience with numpy.distutils and distutils convinced me that there is not much to save from distutils code. I consider the API, the UI and the implementation deeply flawed. For me, an "improved backward compatible" distutils is an oxymoron. More practically, I think the recent distutils-related changes in 2.6.3 and 2.6.4 will keep happening. In numpy.distutils, we depend a lot on internal details of distutils, because we need things not available in the API, and I don't think we are alone. I would mention that numpy is one of the package which has the most invested in distutils in terms of code (sloccount tells me that numpy.distutils is 10315 LOC, whereas python 2.6.2 distutils is 10648 LOC), so I am certainly aware of the cost of breaking backward compatibility.
My opinion is that you've build something else when Distutils was not evolving anymore.
It has nothing to do with distutils evolving. We have had our own extensions for years (I am a relatively newcommer, but some numpy.distutils code goes back to 2001), so we could have done pretty much what we wanted. We now have two build systems: one based on distutils, one based on scons. Every-time I added a build feature to numpy, it took me much more time to add it to distutils than to the scons build. The code needed to support scons build is ~ 2000 lines of code, is more robust, handle dependencies automatically, support reliable parallel builds, and is more maintainable. cheers, David