build system for scikit
So I've had some reports of problems with the scikits.sparse build system; I'm sure the general thrust will be familiar to many people here: -- Even on Linux, different distributions put the CHOLMOD headers in different places (argh wft people) -- Apparently numpy headers are not always placed in the Python include path? This seems broken to me, but okay... -- In principle, it might be nice to be able to build just some of the scikit, e.g. if the person has the right libraries installed for scikits.sparse.cholmod but not scikits.sparse.umfpack. I know that Python building is completely broken in many ways, but there must be some kind of standard solutions (or at least standard hacks!) for these. I did check numpy.distutils, but the docs I can find are all about how to piece together a build for a giant project with many pieces, which is not very relevant, and I couldn't figure out how to use it and Cython together (they both want to replace build_ext). Help! -- Nathaniel
On Wed, Jan 13, 2010 at 1:43 PM, Nathaniel Smith <njs@pobox.com> wrote:
So I've had some reports of problems with the scikits.sparse build system; I'm sure the general thrust will be familiar to many people here: -- Even on Linux, different distributions put the CHOLMOD headers in different places (argh wft people)
Yes, that's annoying, OTOH, dumping a whole bunch of headers in /usr/include is not scalable either.
-- Apparently numpy headers are not always placed in the Python include path? This seems broken to me, but okay...
This should not matter if you use the function to get numpy headers from numpy.distutils,misc_util (get_numpy_include_headers).
-- In principle, it might be nice to be able to build just some of the scikit, e.g. if the person has the right libraries installed for scikits.sparse.cholmod but not scikits.sparse.umfpack.
This sounds like a good idea but it is almost always wrong in my opinion, because you introduce more possible different configurations. It increases more than decreases the distribution burden in my experience.
I know that Python building is completely broken in many ways, but there must be some kind of standard solutions (or at least standard hacks!) for these.
The only way that works if you care about being cross platform is doing the autotools way: check for features instead of versions. So for example, for CHOLMOD headers, you would test into different paths until you found the one you want by compiling some small code snippets. numpy.distutils can help you for that, and you can find a extensive (but complicated) example in numpy/core/setup.py. There is no general solution to your problem with python - autotools and cmake are the best changes ATM, but interfacing distutils with them is a big PITA, cheers, David
Nathaniel Smith wrote:
So I've had some reports of problems with the scikits.sparse build system; I'm sure the general thrust will be familiar to many people here: -- Even on Linux, different distributions put the CHOLMOD headers in different places (argh wft people) -- Apparently numpy headers are not always placed in the Python include path? This seems broken to me, but okay... -- In principle, it might be nice to be able to build just some of the scikit, e.g. if the person has the right libraries installed for scikits.sparse.cholmod but not scikits.sparse.umfpack.
Most other libraries seem to "fix" this by statically linking all the .c files directly into the extension .so. That's the state we're currently in with Python building. (Please, plese don't do this.) It seems to me that the best solution for now is to make things injectable from a configuration file, so that one has to explicitly set CHOLMOD header directory location in a configuration file. This is no different I think from NumPy and SciPy, which will get their paths to MKL etc. (if used) from a configuration file. (The Cython files should probably be changed to use include "cholmod.h" rather than "suitesparse/cholmod.h", and then rely on being passed -I flags). That should give a baseline that's buildable with the Python tools with some manual configuration. Then other people would automate this as appropriate for their platform (e.g. to create a Linux package you'd hook it up with autotools as David suggested -- in my Sage package I am already autogenerating a setup.cfg for scikits.sparse with the Sage-specific NumPy include path in the package build process). Creating something new which is "automatic" and tries to guess the location of CHOLMOD across different platforms is in my opinion a waste of time and makes things worse. What if you want to build your own SuiteSparse? Etc.etc. Explicit is better than implicit! -- Dag Sverre
Dag Sverre Seljebotn wrote:
Creating something new which is "automatic" and tries to guess the location of CHOLMOD across different platforms is in my opinion a waste of time and makes things worse.
It depends on what you mean by automatic: autoconf/cmake configuration schemes are automatic, and they work very well for packages much more complex than python will ever need (e.g. KDE and Gnome). Ideally, there should be a way to do: configure --with-libfoo=/usr/local or configure --with-libfoo-includedir=/usr/local/include --with-libfoo-libdir=/usr/local/lib So that you can configure foo how you want, and inside your setup.py, there would be something like def check_foo(env): try: env.check_header("amd_foo.h") except CompileError: old_env = env.Append({"CPPPATH": ["$prefix/suitesparse"]}) try: env.check_header("amd_foo.h") finally: env = old_env With abstracted machinery to deal with the options given at the command line. But that's typically the kind of things made difficult with distutils commands scheme. David
participants (4)
-
Dag Sverre Seljebotn -
David Cournapeau -
David Cournapeau -
Nathaniel Smith