[SciPy-user] Bundling numpy/scipy with applications

Tue Nov 13 01:10:32 EST 2007

Johann Cohen-Tanugi wrote:
> David Cournapeau wrote:
>> Michael Hearne wrote:
>>   
>>> I'm creating a Python application that is internal to my organization, 
>>> but will be installed on both Linux and Mac OS X machines.  The 
>>> application depends heavily on a number of "non-pure" modules (those 
>>> that have C/C++/FORTRAN components), like numpy, scipy, gdal, etc.
>>>
>>> What is the most "pythonic" way to bundle these types of modules 
>>> inside my application?
>>>
>>> I've investigated dist_utils and setuptools, and I don't see an easy 
>>> way with those tools to include build instructions for packages built 
>>> with autconf tools.
>>>
>>> Is my only recourse to write a custom install script that calls the 
>>> "configure;make;make install" from a shell?
>>>
>>>     
>> You have two problems: building and packaging. For packaging, autoconf 
>> will not help you much outside providing a coherence between packages, I 
>> think; distutils/setuptools have packaging tools, but I don't know how 
>> good they are.
>>
>> On Linux, the packaging system depends on the distribution: since it is 
>> only internal, the problem is severely simplified, since you are likely 
>> to target only one format (rpm, deb, etc...). On Mac OS X, you have also 
>> tools to package .pkg and .mpkg (which are a set of .pkg), available 
>> freely on apple dev website; I have not used them much, but they look 
>> simple and easy to use for simple command line packages.
>>
>> You could hack something in distutils to call configure and so on, but I 
>> don't think it will be pleasant. Building packages using 
>> distutils/setuptools is easy, and do not need much configuration. So I 
>> think it is more natural to use a top level tool calling 
>> distutils/setuptools than to use distutils as the top level tool. As a 
>> pythonic way, you may consider build tools, such as scons or waf, both 
>> written in python.
>>
>> But frankly, I would not bother too much about pythonic way: since some 
>> tools will be shell based anyway (autotools, packaging tools), and you 
>> don't have to care about windows, using make and shell may actually be 
>> easier.
>>
>> For the build part, you may take a look at my garnumpy package: it is 
>> essentially a set of rules for Gnu Makefiles, and it can build numpy + 
>> scipy, with a configurable set of dependencies (ATLAS, NETLIB 
>> BLAS/LAPACK, fftw, etc....). It can build both distutils and 
>> autotools-based packages.
>>
>> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/garnumpy/garnumpy-0.4.tbz2
>>
>> I used it sucessfully on linux and cygwin, so I would say it should work 
>> on Mac OS X without too much trouble. The only thing which will be 
>> likely to be a pain is fat binaries (Universal). I use it to build a 
>> totally self contained numpy/scipy installation, which is a first step 
>> toward packaging. If you think it can be useful to you, don't hesitate 
>> to ask questions; there is also a bzr archive if you want to have access 
>> to the dev history of the tool
>>
>> cheers,
>>
>> David
>> _______________________________________________
>> SciPy-user mailing list
>> SciPy-user at scipy.org
>> http://projects.scipy.org/mailman/listinfo/scipy-user
>>   
> Hi David,
> I am having a hell of a time with lapack/atlas. 
That's because they are far from trivial to build correctly. As always 
with build problems, it is never complicated, but problems are often 
difficult to track down. To make things worse, some linux distribution 
(including fedora and suse) did include bogus packages for blas and 
lapack. That's why I did this garnumpy thing in the first place: quickly 
build blas/lapack correctly, so that I can test things more easily.
> I already posted the 
> issue I currently have with scipy after following the building doc in 
> scipy web site (see 
> http://projects.scipy.org/pipermail/scipy-user/2007-November/014506.html). 
> Besides, octave build seems to also have serious issues with my 
> lapack/atlas build.
Which distribution are you using ?
>
> So I immediately downloaded youor package, but I seem to have a problem 
> with the numpy download :
> make[3]: Leaving directory 
> `/data1/sources/python/garnumpy-0.4/platform/numpy'
> # Change default path when looking for libs to fake dir,
> # so we can set everything by env variables
> cd work/main.d/numpy-1.0.3.1 && 
> PYTHONPATH=/home/cohen/garnumpyinstall/lib/python2.5/site-packages:/home/cohen/garnumpyinstall/lib/python2.5/site-packages/gtk-2.0 
> /usr/bin/python \
>         setup.py config_fc --fcompiler=gnu config
> Traceback (most recent call last):
>   File "setup.py", line 90, in <module>
>     setup_package()
>   File "setup.py", line 60, in setup_package
>     from numpy.distutils.core import setup
>   File 
> "/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/__init__.py", 
> line 39, in <module>
>     import core
>   File 
> "/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/__init__.py", 
> line 8, in <module>
>     import numerictypes as nt
>   File 
> "/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/numerictypes.py", 
> line 83, in <module>
>     from numpy.core.multiarray import typeinfo, ndarray, array, empty, dtype
> ImportError: No module named multiarray
> make[2]: *** [configure-custom] Error 1
> make[2]: Leaving directory 
> `/data1/sources/python/garnumpy-0.4/platform/numpy'
> make[1]: *** [../../platform/numpy/cookies/main.d/install] Error 2
> make[1]: Leaving directory 
> `/data1/sources/python/garnumpy-0.4/platform/scipy'
> make: *** [imgdep-main] Error 2
>
This is really strange, I have never seen this error. This is what I 
would do:
    - remove garnumpyinstall directory
    - go into bootstrap/lapack, and do make install
    - go into platform/numpy, do make install
At this point, you should have an installed numpy in garnumpyinstall: 
test it (import numpy; numpy.test(level = 9999, verbosity = 9999). If 
this works, then go into platform/scipy. The errors should be clearer.
> Any idea? Your package is anyway a great idea. I would actually love to 
> see it work, of course, but also allow for the possibility to point to 
> already exisiting source directories for numpy and scipy (for instance 
> from a previous svn checkout). I did not read the doc of your package so 
> it might actually be there....
You cannot use svn, because this prevents patching from working, and 
also there is a checksum when downloading: this is a major limitation 
(but not so major if you realize that the only thing you really want to 
use from svn are numpy and scipy; you can reuse blas/lapack/fftw/atlas, 
and that's my main use when testing on different platforms).

You can reuse downloaded archives, though:

make garchive

it will download all the tarballs (you cannot choose a subset, 
unfortunately) and put them into a directory which will be reused 
automatically afterwards. This means that even if you do make clean 
anywhere in the source tree, you won't need to download over and over 
the same things.

cheers,

David