[Numpy-discussion] Waf or scons/numscons for a C/Fortran/Cython/Python project -- what's your recommendation?

[Kurt Smith]

On Sat, Jan 16, 2010 at 10:04 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Sun, Jan 17, 2010 at 4:12 AM, Kurt Smith <kwmsmith at gmail.com> wrote:
>> My questions here concern those familiar with configure/build/install
>> systems such as distutils, setuptools, scons/numscons or waf
>> (particularly David Cournapeau).
>>
>> I'm creating a tool known as 'fwrap' that has a component that needs
>> to do essentially what f2py does now -- take fortran source code and
>> compile it into a python extension module.  It uses Cython to create
>> the extension module, and the current configure/build/install system
>> is a very kludgy monkeypatched Cython.distutils and numpy.distutils
>> setup.py script.  The setup.py script works for testing on my system
>> here, but for going prime time, I dread using it.  David has made his
>> critiques of distutils known for scientific software, and I agree.
>> What's the best alternative?
>
> The best alternative in the short term is no alternative: making sure
> everything you need is incorporated in numpy.distutils. Otherwise, you
> will have to recreate everything that distutils is doing: you will
> have people who will demand egg, mac os x .mpkg, windows installers,
> etc.. Basically what I am trying to do with toydist now - I don't mind
> getting help there, though :)

If you ignore the installation phase and focus on just the
configure/build phases (see below), what would you say then?  How much
work needs to go into the install step as compared to the
configure/build steps for waf or scons?

>
> I promised to add decent cython support in numpy.distutils for 1.5.0,
> maybe we should see what we can do for fwrap at the same time.
>
> I am also a bit unclear about what is needed exactly, and what would
> be the workflow: I don't understand why fwrap should care about
> packaging/deployment at all, for example.

I should have emphasized that fwrap primarily needs a good
configure/build system **for the projects that fwrap wraps.**  The
original post probably didn't make this clear.  The workflow is this,
assuming fwrap is installed appropriately on the platform:

 * fwrap is called on a bunch of fortran source files.
     * fwrap generates fortran wrappers, consisting of fortran source
files and Cython source files.
     * If the user wants fwrap to create an extension module then and there:
         * The configure/build system (waf, scons, toydist, or a
setup.py script) kicks in. It...
            * Configures the build, getting appropriate compilers,
making sure Cython is installed, sorting out fortran <-> C type sizes,
etc.
            * Builds the code appropriately, in the right order, etc.
(requires fortran, Cython & C compilation and linking, with scanning
and dependency injection).
            * Puts the extension *.so file in the current working
directory by default.

So the build tool's installation stage is much less important.  It
seems from these discussions that its the installation step that is
particularly nasty.  Fwrap would leave the installation of the
extension module to the user.

>>
>> More specifically: what are the pros/cons between waf and
>> scons/numscons for configure/build/install of a
>> Fortran-C-Cython-Python project?
>
> waf has no fortran support whatsoever, so you would need to add it.
> Waf codebase is much better than scons, but there lacks some internal
> documentation. There were some things that I did not manage to do in
> waf, because the internal API for scanning/dependency injection was
> not very clear to me (by scanning I mean the ability to scan the
> source code to look for dependency, e.g. fortran modules, and by
> dependency injection, I mean adding new targets to the DAG of
> dependencies at runtime - again needed for fortran modules).

I'll likely need both scanning and Dependency Injection in fwrap.  The
complete lack of fortran support is an issue, although not a deal
breaker.

I like that waf is designed to be small, self-contained and not
installed system-wide.  I could distribute waf with fwrap and users
wouldn't have to have yet another external dependency for fwrap to
work.

>
> Basic handling of fortran compilation and fortran detection was
> relatively easy in comparison.
>
> The biggest drawback I see with waf is the lack of users: the only
> significant project I know which uses waf is Ardour. OTOH, I believe
> scons has deep structural problems, and only a few people can change
> some significant parts of the code.

Yes, the lack of adoption is an issue.

Would scons' structural problems affect a project like fwrap, in your
view?  You have a fair amount of experience dealing with Fortran/C
hybrid programming -- is scons still flexible enough to handle it?

>
>>
>> Is scons capable of handling the configure and install stages, or is
>> it only a build system?  As I understand it, numscons is called from
>> distutils; distutils handles the configure/install stages.
>
> Distutils only handles the installation - everything done within the
> build_* command is done by the scons distutils command, and
> configuration is done by scons as well. Scons configure mechanism is
> very primitive - it uses a separate framework than the rest of the
> tool, which means in particular that scons top notch dependency
> handling does not work well for the configuration stage. waf is much
> better in that aspect.
>

Good to know.  Perhaps scons would require a fair amount of work to
get configuration working well, while waf would require fortran work
in the configure/build stages?

>> Scons/numscons have more fortran support that waf, from what I can
>> see.  The main downside of using scons is that I'd still have to mess
>> around with distutils.
>
> My main point should be this: whatever you do, you will end up messing
> with distutils, unless you reimplement everything that distutils does,
> be it waf, scons, etc... In the short term, adding things to
> numpy.distutils is the easiest path.
>
> Long term, I hope toydist will be a tool which will enable exactly
> what you want: using a build system of your choice, and being able to
> reuse existing code for installation/packaging to avoid recreating it
> yourself. You will be able to create an exe/egg/pkg from a simple
> package representation, every package will have a common interface for
> the user independently of the internal build tool, etc...

So is it correct to say that toydist hands off the build to an
external tool, and takes care of the installation/packaging itself?
Since the installation is the least essential part of fwrap and I need
a robust build tool, waf/scons appear to be the best options over
numpy.distutils, even though they will require a good chunk of work.

Its clear I need to test-drive waf and scons to see how they compare
with each other and my existing numpy.distutils and cython.distutils
conglomeration.

Thank you very much for the input, very informative.

Kurt