Bitting the bullet: using scons to build extensions inside distutils ?
Hi, Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension. The more I think about it, the more I think than distutils not being maintained, and numpy/scipy building needs being much more complicated (at least different) than usual python extension, trying to circumvent distutils problems is an ever ending fight. Scons, being developped as a Make replacement, can do all we would like to be able to do with distutils, including: - building shared or static libraries, with dependencies (knows how to do it on many platforms). - can build each object file independently (e.g different compiler options) - is much much friendlier than distutils. - can handle external tools like swig, etc... - have basic facility to look for libraries (ala autoconf. By basic, I mean it is far from being as complete as autoconf, but is much better than distutils). Scons has also the following advantages: - written in python, can be distributed with numpy (by this, I mean AFAIK, license-wise, it is ok, and its size is not big): does not add additional dependency. - can be called within distutils quite easily. That is, I don't see big disadvantage to use it with distutils. It would give use some wanted features out of the box (building extensions based on ctypes, much friendlier way to customize building option). There are some things I am not sure about : - how to build python extension with it: this is of course mandatory - what is required for a "bi-directional" communication with distutils: for this to work, distutils needs to be aware of what scons builds (for things like bdist to work, for example). There is no question this will require some work. But anyway, my feeling is there is a need to improve the distutils thing, and I feel like this may be an easier path than patching over distutils defficiencies. I know scons quite a bit, and am willing to develop at least a prototype to see the feasibility of the whole thing. But before starting, I would like to know whether other find the idea attractive, dumb, is a waste of time, etc... cheers, David
Hi David,
On 9/15/07, David Cournapeau
Hi,
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension. The more I think about it, the more I think than distutils not being maintained, and numpy/scipy building needs being much more complicated (at least different) than usual python extension, trying to circumvent distutils problems is an ever ending fight. Scons, being developped as a Make replacement, can do all we would like to be able to do with distutils, including: - building shared or static libraries, with dependencies (knows how to do it on many platforms). - can build each object file independently (e.g different compiler options) - is much much friendlier than distutils. - can handle external tools like swig, etc... - have basic facility to look for libraries (ala autoconf. By basic, I mean it is far from being as complete as autoconf, but is much better than distutils).
Scons has also the following advantages: - written in python, can be distributed with numpy (by this, I mean AFAIK, license-wise, it is ok, and its size is not big): does not add additional dependency. - can be called within distutils quite easily.
That is, I don't see big disadvantage to use it with distutils. It would give use some wanted features out of the box (building extensions based on ctypes, much friendlier way to customize building option).
I think there was a thread on this subject before, although I may be thinking of another project. I would certainly welcome anything that made it easier to understand the setup and configuration of numpy, but I am not one of the build guys. Chuck
On Sat, September 15, 2007 9:41 pm, David Cournapeau wrote:
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension.
Are you thinking of Python distutils or numpy.distutils? In the case of Python distutils, then I think this is a wrong list to discuss this subject.
The more I think about it, the more I think than distutils not being maintained,
numpy.distutils is being maintained.
and numpy/scipy building needs being much more complicated (at least different) than usual python extension, trying to circumvent distutils problems is an ever ending fight.
why would be scons be different when new architectures and platforms that need to be supported are developed now and in future?
Scons, being developped as a Make replacement, can do all we would like to be able to do with distutils, including: - building shared or static libraries, with dependencies (knows how to do it on many platforms). - can build each object file independently (e.g different compiler options)
IMHO, this is the only important feature that distutils is missing and would be difficult to add to numpy.distutils because of distutils design.
- is much much friendlier than distutils. - can handle external tools like swig, etc...
numpy.distutils can handle swig, psyco,.., also, it can generate sources on-fly. It's not so difficult to add support for some external tool to numpy.distutils.
- have basic facility to look for libraries (ala autoconf. By basic, I mean it is far from being as complete as autoconf, but is much better than distutils).
Can it achive the same as numpy/distutils/system_info.py?
Scons has also the following advantages: - written in python, can be distributed with numpy (by this, I mean AFAIK, license-wise, it is ok, and its size is not big): does not add additional dependency. - can be called within distutils quite easily.
Scons does not have Fortran compiler support, at least not as advanced as numpy.distutils.fcompiler provides, afaik. I think this is the main reason why numpy.distutils cannot be replaced with scons without having much of effort adding this feature to scons.
That is, I don't see big disadvantage to use it with distutils. It would give use some wanted features out of the box (building extensions based on ctypes, much friendlier way to customize building option).
I agree that distutils is a difficult code. But over the years lots of fixes and useful features have been implemented in numpy.distutils that has made using numpy.distutils easier and less errorprone than Python distutils. I don't even remember the last serious issue with numpy.distutils what comes to building scipy or numpy packages. Adding new features is another story of course.
There are some things I am not sure about : - how to build python extension with it: this is of course mandatory - what is required for a "bi-directional" communication with distutils: for this to work, distutils needs to be aware of what scons builds (for things like bdist to work, for example).
It's fine to add new commands to numpy.distutils.
There is no question this will require some work. But anyway, my feeling is there is a need to improve the distutils thing, and I feel like this may be an easier path than patching over distutils defficiencies. I know scons quite a bit, and am willing to develop at least a prototype to see the feasibility of the whole thing.
But before starting, I would like to know whether other find the idea attractive, dumb, is a waste of time, etc...
I think the idea of replacing distutils with some other tool which would be easier to use and to extend has brought up every one or two years. I think it hasn't happened because it would require not just some work but lots of it. Whatever people say about distutils, it is still a good tool that has lots of know-how in it for doing its job in a large variety of platforms we have today. But if you think that you can make scons to support the numpy.distutils features that are important for building numpy and scipy, then go for it. Pearu
On 9/16/07, Pearu Peterson
On Sat, September 15, 2007 9:41 pm, David Cournapeau wrote:
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension.
Are you thinking of Python distutils or numpy.distutils? In the case of Python distutils, then I think this is a wrong list to discuss this subject.
First, let's get one thing straight: this is not about replacing distutils, but about using scons for what it is good at (building code), instead of extending distutils for what it is was not conceived for in the first place (building compiled code in a flexible way). I certainly do not intend to replace distutils, but to use scons a a build tool for extensions (that is, scons would not be aware of python code, but would be used to compile all the extensions; for pure python code, I don't think there are major issues related to distutils ?).
The more I think about it, the more I think than distutils not being maintained,
numpy.distutils is being maintained. I was speaking about python distutils, of course.
and numpy/scipy building needs being much more complicated (at least different) than usual python extension, trying to circumvent distutils problems is an ever ending fight.
why would be scons be different when new architectures and platforms that need to be supported are developed now and in future?
scons already knows about many platforms/compiler combinations: except fortran, I think scons supports more than numpy.distutils now. And it is much less fragile: you can extend it much more easily, which is a key point IMHO.
Scons, being developped as a Make replacement, can do all we would like to be able to do with distutils, including: - building shared or static libraries, with dependencies (knows how to do it on many platforms). - can build each object file independently (e.g different compiler options)
IMHO, this is the only important feature that distutils is missing and would be difficult to add to numpy.distutils because of distutils design.
Whereas it would be trivial with scons :)
- have basic facility to look for libraries (ala autoconf. By basic, I mean it is far from being as complete as autoconf, but is much better than distutils).
Can it achive the same as numpy/distutils/system_info.py?
I think so. Finding functions, headers, types, is available, and you have tools like TryBuild, TryLink and so on which make customizing our own really easy. That is, instead of digging into undocumented python distutils api, you have basic, abstracted tools, on which you can build more high level things.
Scons has also the following advantages: - written in python, can be distributed with numpy (by this, I mean AFAIK, license-wise, it is ok, and its size is not big): does not add additional dependency. - can be called within distutils quite easily.
Scons does not have Fortran compiler support, at least not as advanced as numpy.distutils.fcompiler provides, afaik. I think this is the main reason why numpy.distutils cannot be replaced with scons without having much of effort adding this feature to scons.
I don't know much about fortran, so what are the specific needs for numpy ? scons has a fortran builder, and can be customized. When I take a look at the fcompiler module in numpy.distutils, I get the feeling that most of the hard work would be unnecessary for scons, no ?
I think the idea of replacing distutils with some other tool which would be easier to use and to extend has brought up every one or two years. I think it hasn't happened because it would require not just some work but lots of it. Whatever people say about distutils, it is still a good tool that has lots of know-how in it for doing its job in a large variety of platforms we have today.
I agree, and I do not believe much is throwing code away. Here, I am merely suggesting to extend distutils using scons for extension. Actually, some people got the idea of what I have in mind before me: http://openalea.gforge.inria.fr/dokuwiki/doku.php?id=packages:compilation_in... (I didn't look too much at the code, because this is GPL, and we cannot just copy it without infringing copyright; I just checked it was doing what I thought it was doing). To sum up: keep distutils for what it is not too bad at, and use scons instead of extending distutils for what it was not conceived for and where scons shines. David
There are some things I am not sure about : - how to build python extension with it: this is of course mandatory
We use Scons at the labs for the next version of the tool we use, and it is very simple to buil extensions, at least SWIG ones, for Python 2.5 on Windows, there is the need of adding one more line, but it is very straightforward. Matthieu
Hi,
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension.
In general this seems like an excellent idea. If we can contribute what we need to scons, that would greatly ease the burden of maintenance, and benefit both projects. The key problem will be support. At the moment Pearu maintains and owns numpy.distutils. Will we have the same level of commitment and support for this alternative do you think? How easy would it be to throw up a prototype for the rest of us to look at and get a feel for what the benefits would be? Matthew
Hi,
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension.
In general this seems like an excellent idea. If we can contribute what we need to scons, that would greatly ease the burden of maintenance, and benefit both projects. The key problem will be support. At the moment Pearu maintains and owns numpy.distutils. Will we have the same level of commitment and support for this alternative do you think? I have started to ask some questions related to fortran to the scons ML. At least one guy reports using scons for complex fortran builds (with
Matthew Brett wrote: pre processing, modules, etc...). There are many tools already available for scons. Taking the sources, here is the function which defines the default tools for supported platforms: if str(platform) == 'win32': "prefer Microsoft tools on Windows" linkers = ['mslink', 'gnulink', 'ilink', 'linkloc', 'ilink32' ] c_compilers = ['msvc', 'mingw', 'gcc', 'intelc', 'icl', 'icc', 'cc', 'bcc32' ] cxx_compilers = ['msvc', 'intelc', 'icc', 'g++', 'c++', 'bcc32' ] assemblers = ['masm', 'nasm', 'gas', '386asm' ] fortran_compilers = ['g77', 'ifl', 'cvf', 'f95', 'f90', 'fortran'] ars = ['mslib', 'ar', 'tlib'] elif str(platform) == 'os2': "prefer IBM tools on OS/2" linkers = ['ilink', 'gnulink', 'mslink'] c_compilers = ['icc', 'gcc', 'msvc', 'cc'] cxx_compilers = ['icc', 'g++', 'msvc', 'c++'] assemblers = ['nasm', 'masm', 'gas'] fortran_compilers = ['ifl', 'g77'] ars = ['ar', 'mslib'] elif str(platform) == 'irix': "prefer MIPSPro on IRIX" linkers = ['sgilink', 'gnulink'] c_compilers = ['sgicc', 'gcc', 'cc'] cxx_compilers = ['sgic++', 'g++', 'c++'] assemblers = ['as', 'gas'] fortran_compilers = ['f95', 'f90', 'f77', 'g77', 'fortran'] ars = ['sgiar'] elif str(platform) == 'sunos': "prefer Forte tools on SunOS" linkers = ['sunlink', 'gnulink'] c_compilers = ['suncc', 'gcc', 'cc'] cxx_compilers = ['sunc++', 'g++', 'c++'] assemblers = ['as', 'gas'] fortran_compilers = ['f95', 'f90', 'f77', 'g77', 'fortran'] ars = ['sunar'] elif str(platform) == 'hpux': "prefer aCC tools on HP-UX" linkers = ['hplink', 'gnulink'] c_compilers = ['hpcc', 'gcc', 'cc'] cxx_compilers = ['hpc++', 'g++', 'c++'] assemblers = ['as', 'gas'] fortran_compilers = ['f95', 'f90', 'f77', 'g77', 'fortran'] ars = ['ar'] elif str(platform) == 'aix': "prefer AIX Visual Age tools on AIX" linkers = ['aixlink', 'gnulink'] c_compilers = ['aixcc', 'gcc', 'cc'] cxx_compilers = ['aixc++', 'g++', 'c++'] assemblers = ['as', 'gas'] fortran_compilers = ['f95', 'f90', 'aixf77', 'g77', 'fortran'] ars = ['ar'] elif str(platform) == 'darwin': "prefer GNU tools on Mac OS X, except for some linkers and IBM tools" linkers = ['applelink', 'gnulink'] c_compilers = ['gcc', 'cc'] cxx_compilers = ['g++', 'c++'] assemblers = ['as'] fortran_compilers = ['f95', 'f90', 'g77'] ars = ['ar'] else: "prefer GNU tools on all other platforms" linkers = ['gnulink', 'mslink', 'ilink'] c_compilers = ['gcc', 'msvc', 'intelc', 'icc', 'cc'] cxx_compilers = ['g++', 'msvc', 'intelc', 'icc', 'c++'] assemblers = ['gas', 'nasm', 'masm'] I don't see important platforms missing: all commercial Unices which matter are there with their default compiler, the "big 3" are there too (Mac Os X, Windows, Linux/*BSD). On all those platforms, scons knows how to build static and shared libraries, support rpath on the combinations platform/tools which support it, etc... And adding new tools is much easier than with distutils, I think. Support-wise, scons is used by many project, both open source and commercial. Although not extremely knowledgeable about it, I have done non trivial things with it (including the equivalent of autoconf macro to look for BLAS/LAPACK on many platforms, cross compilation, convertion of some projects from autotools to scons), so I think I know where its default are (e.g. it is terrible for deployment, compared to autotools; as we would drive scons from distutils, this does not matter, though).
How easy would it be to throw up a prototype for the rest of us to look at and get a feel for what the benefits would be?
I don't intend to do everything at once. I was thinking about first getting a new command scons for numpy.distutils: distutils would simply launch scons with all the necessary arguments (compilers and so on); this would make compiling ctypes extension possible at last, without touching much of the code. I have already asked the openalea people if I could borrow some code from them under acceptable license for us; if they accept, it should take only a few days before having something to show. cheers, David
On 9/17/07, Matthew Brett
Hi,
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension.
In general this seems like an excellent idea. If we can contribute what we need to scons, that would greatly ease the burden of maintenance, and benefit both projects. The key problem will be support. At the moment Pearu maintains and owns numpy.distutils. Will we have the same level of commitment and support for this alternative do you think?
How easy would it be to throw up a prototype for the rest of us to look at and get a feel for what the benefits would be? Here we are for those interested: all the work is in the branch numpy.scons.
Basically, I added a command scons to numpy.distutils, and add a add_sconscript hook to the config class. An example can be found in numpy/scons_fake. You are expected to do python setup.py scons To build them (I have not found a way yet to tell the install command to call scons command; but the current implementation put the built code where distutils expects it, so if you call scons and then install, it should be ok). This is only a proof of concept, but as an example, I started to implement a small support library for sconscript to be used inside numpy, and got ctypes extension building working on both windows and linux. The Sconscript file: # vim:syntax=python from numpy.distutils.scons import GetNumpyEnvironment env = GetNumpyEnvironment(ARGUMENTS) config.CheckHeader('stdio.h') config.CheckLib('c', 'printf') config.Finish() source = ['foo.c'] import sys if sys.platform == 'win32': env.AppendUnique(CPPDEFINES = 'WIN32') env.NumpyCTypes('foo', source) And in the setup.py, you just do: config.add_sconscript('SConstruct') Once numpy with scons support is installed, partial build is possible (by partial I mean only using python setup.py scons in a subpackage). This should give a feel on how it would behave for users. If people think this is the right direction, then I can add more complete support (mostly making scons behave as distutils with respect to how to find extension using setup.cfg, implementing the checks in numpy/distutils/system_info for scons, passing the compilers/linkers to scons, etc...). cheers, David
David Cournapeau wrote:
Hi,
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension. The more I think about it, the more I think than distutils not being maintained, and numpy/scipy building needs being much more complicated (at least different) than usual python extension, trying to circumvent distutils problems is an ever ending fight. Scons, being developped as a Make replacement, can do all we would like to be able to do with distutils, including:
Pearu is correct that numpy.distutils has quite a few features that are not easily replaced and so there will be some skepticism initially in doing things differently. However, I don't think there will be much resistance to distributing scons along with numpy so that it may be used to supplement what is done in numpy.distutils as long as it is integrated into the numpy.distutils chain some-how. In short, I'm all for integrating scons with numpy if possible as long as we don't just toss what numpy.distutils has done. -Travis
- building shared or static libraries, with dependencies (knows how to do it on many platforms). - can build each object file independently (e.g different compiler options) - is much much friendlier than distutils. - can handle external tools like swig, etc... - have basic facility to look for libraries (ala autoconf. By basic, I mean it is far from being as complete as autoconf, but is much better than distutils).
Scons has also the following advantages: - written in python, can be distributed with numpy (by this, I mean AFAIK, license-wise, it is ok, and its size is not big): does not add additional dependency. - can be called within distutils quite easily.
That is, I don't see big disadvantage to use it with distutils. It would give use some wanted features out of the box (building extensions based on ctypes, much friendlier way to customize building option).
There are some things I am not sure about : - how to build python extension with it: this is of course mandatory - what is required for a "bi-directional" communication with distutils: for this to work, distutils needs to be aware of what scons builds (for things like bdist to work, for example).
There is no question this will require some work. But anyway, my feeling is there is a need to improve the distutils thing, and I feel like this may be an easier path than patching over distutils defficiencies. I know scons quite a bit, and am willing to develop at least a prototype to see the feasibility of the whole thing.
But before starting, I would like to know whether other find the idea attractive, dumb, is a waste of time, etc...
cheers,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On 9/20/07, Travis E. Oliphant
David Cournapeau wrote:
Hi,
Starting thinking over the whole distutils thing, I was thinking what people would think about using scons inside distutils to build extension. The more I think about it, the more I think than distutils not being maintained, and numpy/scipy building needs being much more complicated (at least different) than usual python extension, trying to circumvent distutils problems is an ever ending fight. Scons, being developped as a Make replacement, can do all we would like to be able to do with distutils, including:
Pearu is correct that numpy.distutils has quite a few features that are not easily replaced and so there will be some skepticism initially in doing things differently.
Now I understand the skepticism, my email was really not clear wrt my intention. I do not intend to replace numpy.distutils by writing everything from scratch: that would be stupid, it would require several weeks of full time programming and testing on various platforms, for no good reasons. Keeping distutils is anyway more or less mandatory because of eggs, which (at least now), require setuptools, which requires distutils. Being incompatible with everybody would also be stupid. For those reasons, I never intended to replace everything, but gradually replacing the parts of distutils/numpy.distutils which are limitating today (no custom library to be used with ctypes, no per extension compilation options) by scons. If this is successfull and people are satisfied with it, we may then gradually start to build numpy extension themselves using scons instead of numpy.distutils where it makes sense. I started some work for a scons command inside numpy.distutils in numpy.scon branch. The idea is that if you want to use scons instead of distutils for one extension, you add the scons script in the setup.py, and distutils then calls scons to build the extension. I still have to figure how to get the various build directories, and how to pass them from distutils to scons, but once this is done, there will be a first prototype which gives an idea on how I expect to make things work. This alone will enable different compilation environments (under scons, an environment is an object which keeps all the build tools names and options, so that you can build different targets with different environments: for example pyrex C code will have no warnings, but by default, python extension will have full warning enabled under gcc, etc...) and ctypes support in a cross platform way (to build our own C libraries callable from ctypes). David
participants (7)
-
Charles R Harris -
David Cournapeau -
David Cournapeau -
Matthew Brett -
Matthieu Brucher -
Pearu Peterson -
Travis E. Oliphant