Stefan van der Walt wrote:
> Hi David
>
> On Thu, Jan 24, 2008 at 12:34:56AM +0900, David Cournapeau wrote:
>>     I've just released the 0.3.0 release of numscons, an alternative 
>> build system for numpy. The tarballs are available on launchpad.
>>
>> https://launchpad.net/numpy.scons.support/0.3/0.3.0
>>
>> To use it, you need to get the build_with_scons numpy branch: see 
>> http://projects.scipy.org/scipy/numpy/wiki/NumpyScons for more
>> details.
>
> Fantastic job, thank you for investing so much time in improving the
> build system.  I noticed that you were trying to get scipy building --
> how is that progressing? 
Not much, but there is not much left to do: I don't remember exactly, 
but umfpack was still problematic; but except that, scipy should be able 
to build with numscons once I change the imports (to cope with the split 
between scons support and numpy.distutils).

Scipy is much easier to build than numpy, because except f2py/fortran 
related problems, everything needed in scipy is needed by numpy. And the 
really hard job was to understand distutils anyway :)
>  Scons should make building on non-*nix
> platforms a lot easier.
Not so much, because scons does not know how to build python extensions 
(I had to develop my own builder, which fortunately should end up in 
scons at some point). Since I started this work, my perception on the 
advantages have changed:
 - real dependencies handling. I did not think it would be so 
interesting, but while using it, I got used to not removing the build 
directory when I change one configuration.
 - ability to use different compiler options for different extensions.
 - easy customization of compiler flags (interesting for debugging, or 
find the best optimization flags).
 - checks are logged in config.log files
 - and of course, the ability to build shared library (c-types 
extensions), static libraries, etc... which is one of the reason I 
started this.

There are also other points which are less obvious, or for which I am 
too biased:
 - the system is simpler: the codebase is 3 times smaller (2500 vs 8000 
lines) and there is a clear distinction between configuration and code 
(flags are not buried in the code anymore, they are defined in separate 
.cfg files). Also, even if scons is 'unpythonic' for some aspects, the 
codebase is so much better than distutils in the stdlib. Everytime I 
touch stdlib distutils, I fear that it will break something in another 
unrelated place, or that I did not call some functions in the right, 
platform specific order; I don't have this problem with scons.
 - the SConscripts also are simpler to read than setup.py I think 
(specially for numpy.core and other modules with complicated 
configurations). They are longer, but there is less magic, and the 
intent is more obvious I think. This is really important, because I 
think I spent more than 50 % of my time on understanding distutils for 
this. I hope that nobody will have to do that anymore.

cheers,

David

dmitrey wrote:

Hi all, I don't know much about what are these scons are, if it's something essential (as it seems to be from amount of mailing list traffic) why can't it be just merged to numpy, w/o making any additional branches?

It's a very large, still experimental change to the entire build infrastructure. It requires branches for us to evaluate. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

dmitrey wrote:

Hi all, I don't know much about what are these scons are, if it's something essential (as it seems to be from amount of mailing list traffic) why can't it be just merged to numpy, w/o making any additional branches? scons is a build system, like make. The difference being that sconscripts (makefiles for scons) are written in python, and well supported on windows.

The standard way to build python extension is to use distutils, but distutils itself is difficult to extend (the code is basically spaghetti, not maintained and not documented), and to be fair, numpy's needs go way beyond the usual python extension: we want to control optimization flags, we need fortran, we depend on a lot of external libraries we need to check, etc... numpy uses distutils extension (numpy.distutils), to support fortran and other goodies. But because it is based on distutils, it has inherited some of the distutils' problems; in particular, it is not possible to build dynamically loaded libraries (necessary for ctypes-based extensions), it is difficult to check for additional libraries, etc... So instead, I plug scons in distutils, so that all C code in numpy is built through sconscripts. But the build system arguably being one of the most crucial part of numpy, it is necessary to do changes incrementally in branches (the code changes are big: several thousand of python code, which is difficult to test by nature). cheers, David

Neal Becker wrote:

Is numscons specific to numpy/scipy, or is it for building arbitrary python extensions (replacing distutils?). I'm hoping for the latter. Actually, another way to answer your question: I am working on a patch such as the part of numscons which takes care of building python extensions in a cross platform way can be included in scons itself (I already have contribute quite a few patches/fix for scons since I started this, since I would rather not depend on in-house changes to scons).

cheers, David