Waf or scons/numscons for a C/Fortran/Cython/Python project -- what's your recommendation?
My questions here concern those familiar with configure/build/install systems such as distutils, setuptools, scons/numscons or waf (particularly David Cournapeau). I'm creating a tool known as 'fwrap' that has a component that needs to do essentially what f2py does now -- take fortran source code and compile it into a python extension module. It uses Cython to create the extension module, and the current configure/build/install system is a very kludgy monkeypatched Cython.distutils and numpy.distutils setup.py script. The setup.py script works for testing on my system here, but for going prime time, I dread using it. David has made his critiques of distutils known for scientific software, and I agree. What's the best alternative? More specifically: what are the pros/cons between waf and scons/numscons for configure/build/install of a Fortran-C-Cython-Python project? Is scons capable of handling the configure and install stages, or is it only a build system? As I understand it, numscons is called from distutils; distutils handles the configure/install stages. Scons/numscons have more fortran support that waf, from what I can see. The main downside of using scons is that I'd still have to mess around with distutils. It looks like waf has explicit support for all three stages, and could be just what I'm looking for. David has a few threads on the waf-users list about getting fortran working with waf. Has that progressed much? I want to contribute to this, for the benefit of scipy and my project, and to limit duplicated work. From what I gather, the fortran configuration stuff in numscons is separated nicely from the scon-specific stuff :-) Would it be a matter of porting the numscons fortran stuff into waf? Any comments you have on using waf/scons for numerical projects would be welcome! Kurt
Hi, SCons can also do configuration and installation steps. David made it possible to use SCons capabilities from distutils, but you can still make a C/Fortran/Cython/Python project with SCons. Matthieu 2010/1/16 Kurt Smith <kwmsmith@gmail.com>:
My questions here concern those familiar with configure/build/install systems such as distutils, setuptools, scons/numscons or waf (particularly David Cournapeau).
I'm creating a tool known as 'fwrap' that has a component that needs to do essentially what f2py does now -- take fortran source code and compile it into a python extension module. It uses Cython to create the extension module, and the current configure/build/install system is a very kludgy monkeypatched Cython.distutils and numpy.distutils setup.py script. The setup.py script works for testing on my system here, but for going prime time, I dread using it. David has made his critiques of distutils known for scientific software, and I agree. What's the best alternative?
More specifically: what are the pros/cons between waf and scons/numscons for configure/build/install of a Fortran-C-Cython-Python project?
Is scons capable of handling the configure and install stages, or is it only a build system? As I understand it, numscons is called from distutils; distutils handles the configure/install stages. Scons/numscons have more fortran support that waf, from what I can see. The main downside of using scons is that I'd still have to mess around with distutils.
It looks like waf has explicit support for all three stages, and could be just what I'm looking for. David has a few threads on the waf-users list about getting fortran working with waf. Has that progressed much? I want to contribute to this, for the benefit of scipy and my project, and to limit duplicated work. From what I gather, the fortran configuration stuff in numscons is separated nicely from the scon-specific stuff :-) Would it be a matter of porting the numscons fortran stuff into waf?
Any comments you have on using waf/scons for numerical projects would be welcome!
Kurt _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher
Matthieu Brucher wrote:
Hi,
SCons can also do configuration and installation steps. David made it possible to use SCons capabilities from distutils, but you can still make a C/Fortran/Cython/Python project with SCons.
Also, while I think waf looks interesting, I've seen almost 0 projects actually using it.
Kurt Smith wrote:
My questions here concern those familiar with configure/build/install systems such as distutils, setuptools, scons/numscons or waf (particularly David Cournapeau).
I'm creating a tool known as 'fwrap' that has a component that needs to do essentially what f2py does now -- take fortran source code and compile it into a python extension module. It uses Cython to create the extension module, and the current configure/build/install system is a very kludgy monkeypatched Cython.distutils and numpy.distutils setup.py script. The setup.py script works for testing on my system here, but for going prime time, I dread using it. David has made his critiques of distutils known for scientific software, and I agree. What's the best alternative?
More specifically: what are the pros/cons between waf and scons/numscons for configure/build/install of a Fortran-C-Cython-Python project?
Is scons capable of handling the configure and install stages, or is it only a build system? As I understand it, numscons is called from distutils; distutils handles the configure/install stages. Scons/numscons have more fortran support that waf, from what I can see. The main downside of using scons is that I'd still have to mess around with distutils.
Not that I really know anything about it, but note that one of the purposes of David's toydist is to handle the install stage independently of the build system used. That is, it is able to create e.g. Python eggs without using setuptools. The thing is, installing Python software is something of a mess, and every system would want this done differently (making an Ubuntu package, creating a DMG, or creating a Python egg are all different things). So I think it makes sense to decouple this from the build in the tools that are used. Of course, toydist is beta, and I dare say you have enough beta dependencies for fwrap already :-) Dag Sverre
It looks like waf has explicit support for all three stages, and could be just what I'm looking for. David has a few threads on the waf-users list about getting fortran working with waf. Has that progressed much? I want to contribute to this, for the benefit of scipy and my project, and to limit duplicated work. From what I gather, the fortran configuration stuff in numscons is separated nicely from the scon-specific stuff :-) Would it be a matter of porting the numscons fortran stuff into waf?
Any comments you have on using waf/scons for numerical projects would be welcome!
Kurt _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sat, Jan 16, 2010 at 2:38 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
Not that I really know anything about it, but note that one of the purposes of David's toydist is to handle the install stage independently of the build system used. That is, it is able to create e.g. Python eggs without using setuptools.
The thing is, installing Python software is something of a mess, and every system would want this done differently (making an Ubuntu package, creating a DMG, or creating a Python egg are all different things). So I think it makes sense to decouple this from the build in the tools that are used.
Yep. Good points. I expect once I get the configure/build stages in a working state, I'll have most of what people need. The install stage is less crucial, at least for the first version. Seems like people would like the system to just create a .so file in the current directory, and leave it at that. If I can get that working on all platforms I'll be very happy :-)
Of course, toydist is beta, and I dare say you have enough beta dependencies for fwrap already :-)
:-) Hopefully that can be remedied that in the coming months, at least from the fparser and memoryview-support-in-Cython side of things.
Dag Sverre
Kurt Smith wrote:
On Sat, Jan 16, 2010 at 2:38 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
Not that I really know anything about it, but note that one of the purposes of David's toydist is to handle the install stage independently of the build system used. That is, it is able to create e.g. Python eggs without using setuptools.
The thing is, installing Python software is something of a mess, and every system would want this done differently (making an Ubuntu package, creating a DMG, or creating a Python egg are all different things). So I think it makes sense to decouple this from the build in the tools that are used.
Yep. Good points. I expect once I get the configure/build stages in a working state, I'll have most of what people need. The install stage is less crucial, at least for the first version. Seems like people would like the system to just create a .so file in the current directory, and leave it at that. If I can get that working on all platforms I'll be very happy :-)
Of course, toydist is beta, and I dare say you have enough beta dependencies for fwrap already :-)
:-)
Hopefully that can be remedied that in the coming months, at least from the fparser and memoryview-support-in-Cython side of things.
Obviously I didn't get around to that yet... As for the build systems, some things to consider (I have no clue myself as to waf vs. scons): - There's already primitive scons support for Cython, but I'm sure it wouldn't be hard to add to waf - Whatever you pick is likely to become the best supported build system for Cython code in the future, I think, due to our interest in working on it - Does waf have infrastructure for parsing files and finding dependencies? I know that in Scons one can plug in a "Cython parser", which checks the dependencies (which pxds are used, basically), so that pyx files are rebuilt automatically when pxds they depend on change. I'm sure waf supports something similar, if not I'd say it disqualifies it. My own hunch is that waf looks better, but scons has a larger mind share and Cython support right now in scientific Python, and that both must be supported eventually, so why not do scons first... *shrug* But like you I'm anxious to hear from more non-Cython devs as well on this matter. Dag Sverre
On Sat, Jan 16, 2010 at 4:43 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
Kurt Smith wrote:
On Sat, Jan 16, 2010 at 2:38 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
Not that I really know anything about it, but note that one of the purposes of David's toydist is to handle the install stage independently of the build system used. That is, it is able to create e.g. Python eggs without using setuptools.
The thing is, installing Python software is something of a mess, and every system would want this done differently (making an Ubuntu package, creating a DMG, or creating a Python egg are all different things). So I think it makes sense to decouple this from the build in the tools that are used.
Yep. Good points. I expect once I get the configure/build stages in a working state, I'll have most of what people need. The install stage is less crucial, at least for the first version. Seems like people would like the system to just create a .so file in the current directory, and leave it at that. If I can get that working on all platforms I'll be very happy :-)
Of course, toydist is beta, and I dare say you have enough beta dependencies for fwrap already :-)
:-)
Hopefully that can be remedied that in the coming months, at least from the fparser and memoryview-support-in-Cython side of things.
Obviously I didn't get around to that yet...
As for the build systems, some things to consider (I have no clue myself as to waf vs. scons): - There's already primitive scons support for Cython, but I'm sure it wouldn't be hard to add to waf - Whatever you pick is likely to become the best supported build system for Cython code in the future, I think, due to our interest in working on it - Does waf have infrastructure for parsing files and finding dependencies? I know that in Scons one can plug in a "Cython parser", which checks the dependencies (which pxds are used, basically), so that pyx files are rebuilt automatically when pxds they depend on change. I'm sure waf supports something similar, if not I'd say it disqualifies it.
My own hunch is that waf looks better, but scons has a larger mind share and Cython support right now in scientific Python, and that both must be supported eventually, so why not do scons first... *shrug*
But like you I'm anxious to hear from more non-Cython devs as well on this matter.
Dag Sverre _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
From a very brief look at the waf book, I don't really understand what
the cross-platform capabilities of waf are http://freehackers.org/~tnagy/wafbook/single.html : "Installing Waf on a system is unnecessary and discouraged: " "Operating systems: Waf cannot be installed on Windows (yet)" Josef
On Sun, Jan 17, 2010 at 8:36 AM, <josef.pktd@gmail.com> wrote:
On Sat, Jan 16, 2010 at 4:43 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
Kurt Smith wrote:
On Sat, Jan 16, 2010 at 2:38 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
Not that I really know anything about it, but note that one of the purposes of David's toydist is to handle the install stage independently of the build system used. That is, it is able to create e.g. Python eggs without using setuptools.
The thing is, installing Python software is something of a mess, and every system would want this done differently (making an Ubuntu package, creating a DMG, or creating a Python egg are all different things). So I think it makes sense to decouple this from the build in the tools that are used.
Yep. Good points. I expect once I get the configure/build stages in a working state, I'll have most of what people need. The install stage is less crucial, at least for the first version. Seems like people would like the system to just create a .so file in the current directory, and leave it at that. If I can get that working on all platforms I'll be very happy :-)
Of course, toydist is beta, and I dare say you have enough beta dependencies for fwrap already :-)
:-)
Hopefully that can be remedied that in the coming months, at least from the fparser and memoryview-support-in-Cython side of things.
Obviously I didn't get around to that yet...
As for the build systems, some things to consider (I have no clue myself as to waf vs. scons): - There's already primitive scons support for Cython, but I'm sure it wouldn't be hard to add to waf - Whatever you pick is likely to become the best supported build system for Cython code in the future, I think, due to our interest in working on it - Does waf have infrastructure for parsing files and finding dependencies? I know that in Scons one can plug in a "Cython parser", which checks the dependencies (which pxds are used, basically), so that pyx files are rebuilt automatically when pxds they depend on change. I'm sure waf supports something similar, if not I'd say it disqualifies it.
My own hunch is that waf looks better, but scons has a larger mind share and Cython support right now in scientific Python, and that both must be supported eventually, so why not do scons first... *shrug*
But like you I'm anxious to hear from more non-Cython devs as well on this matter.
Dag Sverre _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
From a very brief look at the waf book, I don't really understand what the cross-platform capabilities of waf are
http://freehackers.org/~tnagy/wafbook/single.html : "Installing Waf on a system is unnecessary and discouraged: "
The main waf author claims that waf should never be installed, and always included with your package. I think it makes sense for a lot of practical cases (and that's how autotools work, mostly: autoconf/automake are not needed when building something from sources, because you have a gianting shell script called configure). I know there has been some effort toward better windows support for waf - given that waf is written in python, it is hard to see a architectural reason why waf could not work well on windows. But build tools depend a lot spawning processes and the likes both efficiently and reliably, and that's one of the area where windows and unix-like systems are fundamentally different. David
On Sun, Jan 17, 2010 at 4:12 AM, Kurt Smith <kwmsmith@gmail.com> wrote:
My questions here concern those familiar with configure/build/install systems such as distutils, setuptools, scons/numscons or waf (particularly David Cournapeau).
I'm creating a tool known as 'fwrap' that has a component that needs to do essentially what f2py does now -- take fortran source code and compile it into a python extension module. It uses Cython to create the extension module, and the current configure/build/install system is a very kludgy monkeypatched Cython.distutils and numpy.distutils setup.py script. The setup.py script works for testing on my system here, but for going prime time, I dread using it. David has made his critiques of distutils known for scientific software, and I agree. What's the best alternative?
The best alternative in the short term is no alternative: making sure everything you need is incorporated in numpy.distutils. Otherwise, you will have to recreate everything that distutils is doing: you will have people who will demand egg, mac os x .mpkg, windows installers, etc.. Basically what I am trying to do with toydist now - I don't mind getting help there, though :) I promised to add decent cython support in numpy.distutils for 1.5.0, maybe we should see what we can do for fwrap at the same time. I am also a bit unclear about what is needed exactly, and what would be the workflow: I don't understand why fwrap should care about packaging/deployment at all, for example.
More specifically: what are the pros/cons between waf and scons/numscons for configure/build/install of a Fortran-C-Cython-Python project?
waf has no fortran support whatsoever, so you would need to add it. Waf codebase is much better than scons, but there lacks some internal documentation. There were some things that I did not manage to do in waf, because the internal API for scanning/dependency injection was not very clear to me (by scanning I mean the ability to scan the source code to look for dependency, e.g. fortran modules, and by dependency injection, I mean adding new targets to the DAG of dependencies at runtime - again needed for fortran modules). Basic handling of fortran compilation and fortran detection was relatively easy in comparison. The biggest drawback I see with waf is the lack of users: the only significant project I know which uses waf is Ardour. OTOH, I believe scons has deep structural problems, and only a few people can change some significant parts of the code.
Is scons capable of handling the configure and install stages, or is it only a build system? As I understand it, numscons is called from distutils; distutils handles the configure/install stages.
Distutils only handles the installation - everything done within the build_* command is done by the scons distutils command, and configuration is done by scons as well. Scons configure mechanism is very primitive - it uses a separate framework than the rest of the tool, which means in particular that scons top notch dependency handling does not work well for the configuration stage. waf is much better in that aspect.
Scons/numscons have more fortran support that waf, from what I can see. The main downside of using scons is that I'd still have to mess around with distutils.
My main point should be this: whatever you do, you will end up messing with distutils, unless you reimplement everything that distutils does, be it waf, scons, etc... In the short term, adding things to numpy.distutils is the easiest path. Long term, I hope toydist will be a tool which will enable exactly what you want: using a build system of your choice, and being able to reuse existing code for installation/packaging to avoid recreating it yourself. You will be able to create an exe/egg/pkg from a simple package representation, every package will have a common interface for the user independently of the internal build tool, etc... David
On Sat, Jan 16, 2010 at 10:04 PM, David Cournapeau <cournape@gmail.com> wrote:
On Sun, Jan 17, 2010 at 4:12 AM, Kurt Smith <kwmsmith@gmail.com> wrote:
My questions here concern those familiar with configure/build/install systems such as distutils, setuptools, scons/numscons or waf (particularly David Cournapeau).
I'm creating a tool known as 'fwrap' that has a component that needs to do essentially what f2py does now -- take fortran source code and compile it into a python extension module. It uses Cython to create the extension module, and the current configure/build/install system is a very kludgy monkeypatched Cython.distutils and numpy.distutils setup.py script. The setup.py script works for testing on my system here, but for going prime time, I dread using it. David has made his critiques of distutils known for scientific software, and I agree. What's the best alternative?
The best alternative in the short term is no alternative: making sure everything you need is incorporated in numpy.distutils. Otherwise, you will have to recreate everything that distutils is doing: you will have people who will demand egg, mac os x .mpkg, windows installers, etc.. Basically what I am trying to do with toydist now - I don't mind getting help there, though :)
If you ignore the installation phase and focus on just the configure/build phases (see below), what would you say then? How much work needs to go into the install step as compared to the configure/build steps for waf or scons?
I promised to add decent cython support in numpy.distutils for 1.5.0, maybe we should see what we can do for fwrap at the same time.
I am also a bit unclear about what is needed exactly, and what would be the workflow: I don't understand why fwrap should care about packaging/deployment at all, for example.
I should have emphasized that fwrap primarily needs a good configure/build system **for the projects that fwrap wraps.** The original post probably didn't make this clear. The workflow is this, assuming fwrap is installed appropriately on the platform: * fwrap is called on a bunch of fortran source files. * fwrap generates fortran wrappers, consisting of fortran source files and Cython source files. * If the user wants fwrap to create an extension module then and there: * The configure/build system (waf, scons, toydist, or a setup.py script) kicks in. It... * Configures the build, getting appropriate compilers, making sure Cython is installed, sorting out fortran <-> C type sizes, etc. * Builds the code appropriately, in the right order, etc. (requires fortran, Cython & C compilation and linking, with scanning and dependency injection). * Puts the extension *.so file in the current working directory by default. So the build tool's installation stage is much less important. It seems from these discussions that its the installation step that is particularly nasty. Fwrap would leave the installation of the extension module to the user.
More specifically: what are the pros/cons between waf and scons/numscons for configure/build/install of a Fortran-C-Cython-Python project?
waf has no fortran support whatsoever, so you would need to add it. Waf codebase is much better than scons, but there lacks some internal documentation. There were some things that I did not manage to do in waf, because the internal API for scanning/dependency injection was not very clear to me (by scanning I mean the ability to scan the source code to look for dependency, e.g. fortran modules, and by dependency injection, I mean adding new targets to the DAG of dependencies at runtime - again needed for fortran modules).
I'll likely need both scanning and Dependency Injection in fwrap. The complete lack of fortran support is an issue, although not a deal breaker. I like that waf is designed to be small, self-contained and not installed system-wide. I could distribute waf with fwrap and users wouldn't have to have yet another external dependency for fwrap to work.
Basic handling of fortran compilation and fortran detection was relatively easy in comparison.
The biggest drawback I see with waf is the lack of users: the only significant project I know which uses waf is Ardour. OTOH, I believe scons has deep structural problems, and only a few people can change some significant parts of the code.
Yes, the lack of adoption is an issue. Would scons' structural problems affect a project like fwrap, in your view? You have a fair amount of experience dealing with Fortran/C hybrid programming -- is scons still flexible enough to handle it?
Is scons capable of handling the configure and install stages, or is it only a build system? As I understand it, numscons is called from distutils; distutils handles the configure/install stages.
Distutils only handles the installation - everything done within the build_* command is done by the scons distutils command, and configuration is done by scons as well. Scons configure mechanism is very primitive - it uses a separate framework than the rest of the tool, which means in particular that scons top notch dependency handling does not work well for the configuration stage. waf is much better in that aspect.
Good to know. Perhaps scons would require a fair amount of work to get configuration working well, while waf would require fortran work in the configure/build stages?
Scons/numscons have more fortran support that waf, from what I can see. The main downside of using scons is that I'd still have to mess around with distutils.
My main point should be this: whatever you do, you will end up messing with distutils, unless you reimplement everything that distutils does, be it waf, scons, etc... In the short term, adding things to numpy.distutils is the easiest path.
Long term, I hope toydist will be a tool which will enable exactly what you want: using a build system of your choice, and being able to reuse existing code for installation/packaging to avoid recreating it yourself. You will be able to create an exe/egg/pkg from a simple package representation, every package will have a common interface for the user independently of the internal build tool, etc...
So is it correct to say that toydist hands off the build to an external tool, and takes care of the installation/packaging itself? Since the installation is the least essential part of fwrap and I need a robust build tool, waf/scons appear to be the best options over numpy.distutils, even though they will require a good chunk of work. Its clear I need to test-drive waf and scons to see how they compare with each other and my existing numpy.distutils and cython.distutils conglomeration. Thank you very much for the input, very informative. Kurt
David Cournapeau wrote:
Waf codebase is much better than scons,
I don't know about waf, but I do know that I tried to add OS-X application bundle support to scons, and it was really, really painful. It sure seemed like it should have been easy to do -- it's just a well-defined directory structure.
The biggest drawback I see with waf is the lack of users: the only significant project I know which uses waf is Ardour.
There was some work done to use it for wxWebKit, though I don't know what's come of that: https://bugs.webkit.org/show_bug.cgi?id=27619 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (7)
-
Christopher Barker -
Dag Sverre Seljebotn -
David Cournapeau -
josef.pktd＠gmail.com -
Kurt Smith -
Matthieu Brucher -
Neal Becker