[Numpy-discussion] Bitting the bullet: using scons to build extensions inside distutils ?

[David Cournapeau]

Matthew Brett wrote:
> Hi,
>
>>    Starting thinking over the whole distutils thing, I was thinking
>> what people would think about using scons inside distutils to build
>> extension.
>
> In general this seems like an excellent idea.  If we can contribute
> what we need to scons, that would greatly ease the burden of
> maintenance, and benefit both projects.  The key problem will be
> support.  At the moment Pearu maintains and owns numpy.distutils.
> Will we have the same level of commitment and support for this
> alternative do you think?
I have started to ask some questions related to fortran to the scons ML. 
At least one guy reports using scons for complex fortran builds (with 
pre processing, modules, etc...). There are many tools already available 
for scons. Taking the sources, here is the function which defines the 
default tools for supported platforms:

 if str(platform) == 'win32':
        "prefer Microsoft tools on Windows"
        linkers = ['mslink', 'gnulink', 'ilink', 'linkloc', 'ilink32' ]
        c_compilers = ['msvc', 'mingw', 'gcc', 'intelc', 'icl', 'icc', 
'cc', 'bcc32' ]
        cxx_compilers = ['msvc', 'intelc', 'icc', 'g++', 'c++', 'bcc32' ]
        assemblers = ['masm', 'nasm', 'gas', '386asm' ]
        fortran_compilers = ['g77', 'ifl', 'cvf', 'f95', 'f90', 'fortran']
        ars = ['mslib', 'ar', 'tlib']
    elif str(platform) == 'os2':
        "prefer IBM tools on OS/2"
        linkers = ['ilink', 'gnulink', 'mslink']
        c_compilers = ['icc', 'gcc', 'msvc', 'cc']
        cxx_compilers = ['icc', 'g++', 'msvc', 'c++']
        assemblers = ['nasm', 'masm', 'gas']
        fortran_compilers = ['ifl', 'g77']
        ars = ['ar', 'mslib']
    elif str(platform) == 'irix':
        "prefer MIPSPro on IRIX"
        linkers = ['sgilink', 'gnulink']
        c_compilers = ['sgicc', 'gcc', 'cc']
        cxx_compilers = ['sgic++', 'g++', 'c++']
        assemblers = ['as', 'gas']
        fortran_compilers = ['f95', 'f90', 'f77', 'g77', 'fortran']
        ars = ['sgiar']
    elif str(platform) == 'sunos':
        "prefer Forte tools on SunOS"
        linkers = ['sunlink', 'gnulink']
        c_compilers = ['suncc', 'gcc', 'cc']
        cxx_compilers = ['sunc++', 'g++', 'c++']
        assemblers = ['as', 'gas']
        fortran_compilers = ['f95', 'f90', 'f77', 'g77', 'fortran']
        ars = ['sunar']
    elif str(platform) == 'hpux':
        "prefer aCC tools on HP-UX"
        linkers = ['hplink', 'gnulink']
        c_compilers = ['hpcc', 'gcc', 'cc']
        cxx_compilers = ['hpc++', 'g++', 'c++']
        assemblers = ['as', 'gas']
        fortran_compilers = ['f95', 'f90', 'f77', 'g77', 'fortran']
        ars = ['ar']
    elif str(platform) == 'aix':
        "prefer AIX Visual Age tools on AIX"
        linkers = ['aixlink', 'gnulink']
        c_compilers = ['aixcc', 'gcc', 'cc']
        cxx_compilers = ['aixc++', 'g++', 'c++']
        assemblers = ['as', 'gas']
        fortran_compilers = ['f95', 'f90', 'aixf77', 'g77', 'fortran']
        ars = ['ar']
    elif str(platform) == 'darwin':
        "prefer GNU tools on Mac OS X, except for some linkers and IBM 
tools"
        linkers = ['applelink', 'gnulink']
        c_compilers = ['gcc', 'cc']
        cxx_compilers = ['g++', 'c++']
        assemblers = ['as']
        fortran_compilers = ['f95', 'f90', 'g77']
        ars = ['ar']
    else:
        "prefer GNU tools on all other platforms"
        linkers = ['gnulink', 'mslink', 'ilink']
        c_compilers = ['gcc', 'msvc', 'intelc', 'icc', 'cc']
        cxx_compilers = ['g++', 'msvc', 'intelc', 'icc', 'c++']
        assemblers = ['gas', 'nasm', 'masm']

I don't see important platforms missing: all commercial Unices which 
matter are there with their default compiler, the "big 3" are there too 
(Mac Os X, Windows, Linux/*BSD). On all those platforms, scons knows how 
to build static and shared libraries, support rpath on the combinations 
platform/tools which support it, etc... And adding new tools is much 
easier than with distutils, I think.

Support-wise, scons is used by many project, both open source and 
commercial. Although not extremely knowledgeable about it, I have done 
non trivial things with it (including the equivalent of autoconf macro 
to look for BLAS/LAPACK on many platforms, cross compilation, convertion 
of some projects from autotools to scons), so I think I know where its 
default are (e.g. it is terrible for deployment, compared to autotools; 
as we would drive scons from distutils, this does not matter, though).
>
> How easy would it be to throw up a prototype for the rest of us to
> look at and get a feel for what the benefits would be?
I don't intend to do everything at once. I was thinking about first 
getting a new command scons for numpy.distutils: distutils would simply 
launch scons with all the necessary arguments (compilers and so on); 
this would make compiling ctypes extension possible at last, without 
touching much of the code.

I have already asked the openalea people if I could borrow some code 
from them under acceptable license for us; if they accept, it should 
take only a few days before having something to show.

cheers,

David