[Numpy-discussion] For review: first milestone of scons support in numpy

[Pearu Peterson]

David Cournapeau wrote:
> Pearu Peterson wrote:
>> 2) It seems that scons does not interfare with numpy.distutils much.
>> If this is true and numpy/scipy builds will not break when scons is
>> not installed then I think you could continue the scons support
>> development in trunk.
> It won't break if scons is not installed because scons sources are 
> copied into the branch. Scons developers explicitely support this:
> 
> http://www.scons.org/faq.php#SS_3_3
> 
> (AFAIK, it does not pose any problem license-wise, since scons is new 
> BSD license; it adds ~350 kb of compressed source code to numpy).

I think this is good.
Does scons require python-dev? If not then this will solve one of the
frequent issues that new users may experience: not installed distutils.

> Now, concerning migrating all compiled extensions to scons, I would 
> prefer avoiding doing it in the trunk; but I heard horror stories about 
> subversion and merging, so maybe staying outside the trunk is too risky 
> ?

I think numpy is quite stable now that it's safe to develop in a branch
(if trunk is very actively developed then merging branches
can be a nightmare). However, IMHO using a branch makes other
developers to stay aside from branch development and in time it is
more and more difficult to merge.

> Also, when I said that I don't see big problems to replace distutils 
> for compiled extensions, I lied by over-simplification. If scons is used 
> for compiled extension, with the current design, distutils will call 
> scons for each package. Calling scons is expensive (importing many 
> modules, etc...: this easily takes around one second for each non 
> trivial Sconscript), and also, because each of them is independent, we 
> may check several times for the same thing (fortran compiler, etc...), 
> which would really add up to the build time.
 >
> I see two approaches here:
>     - do not care about it because numpy is unlikely to become really bigger
>     - considering that numpy is really one big package (contrary to 
> scipy which, at least in my understanding, is gearing toward less 
> inter-package dependencies ?), we should only have one big sconscript 
> for configuration (checking blas/lapack, compilers, etc...), and use 
> scons recursively. In this case, it should not be much slower than the 
> current system.

Note that building a subpackage in subpackage directory must be 
supported. So a big sconscript may not be an option.

The third approach would be to cache those checks that are called
frequently to, say, $HOME/.numpy/scons.

numpy.distutils setup.py script gathers the information from
subpackage setup.py scripts recursively and then passes all the
information to one setup function call.
I think setupscons.py should do the same. If scons does not support
recursive reading of scons scripts then the corresponding feature
should be implemented, I guess it would not be difficult.

>> 4) though, we cannot remove numpy.distutils for backward compatibility
>> with software using numpy.distutils. However, it should be possible
>> to enhance Configuration class to generate the corresponding SConstruct
>> scripts. It will save some work when converting configuration() 
>> functions to SConstruct scripts.
> Could you elaborate on this point ? I am not sure what you mean by 
> generating Configuration class ?

I meant that Configuration class could have a method, say 
toscons(<filename>),  that will generate SConstruct script
from the information that Configuration instance holds. I thought that 
this would just ease creating SConstruct scripts from
existing setup.py files.

Pearu