building with static dependencies
Hi All, I'd like to build some packages into eggs such that I can easilly install them with virtual_env or buildout. The packages in question are numpy and scipy, at the moment. Are there parameters I can pass to "python setup.py build" or "python setup.py bdist_egg" that will cause the egg to be built will all its dependencies statically linked and included within the egg? (ie: so I can build once on linux and once on windows, put the binary eggs in our local index and then not have to worry about the build proceses and convoluted dependencies on other machines of a similar architecture) cheers, Chris
Similar to "STATICDEPS=True python setup.py bdist_egg" in lxml? For numpy you only need to use the pre-built ATLAS libraries (see scipy installation document) on windows 32-bit (for 64-bit, you will need the proprietary MKL to build atlas), and build atlas yourself on linux. In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked. To statically link libgfortran: on Mac, take a look at the libgfotran.a copying hack in pavement.py (scipy trunk); on Linux, I haven't yet figured out a way to reliably link libgfortran statically to numpy. In PyPM, we are standardizing on the 'STATIC=True' environment variable for configuring packages to build with static linking. Eg: we patched mysql-python to test this environment variable and then build statically against mysql (after automatically downloading/building it). You will have to patch setup.py yourself in order to make the packages build statically (eg: see buildlibxml.py in lxml source) -srid On 2010-04-28, at 6:14 AM, Chris Withers wrote:
Hi All,
I'd like to build some packages into eggs such that I can easilly install them with virtual_env or buildout.
The packages in question are numpy and scipy, at the moment.
Are there parameters I can pass to "python setup.py build" or "python setup.py bdist_egg" that will cause the egg to be built will all its dependencies statically linked and included within the egg?
(ie: so I can build once on linux and once on windows, put the binary eggs in our local index and then not have to worry about the build proceses and convoluted dependencies on other machines of a similar architecture)
cheers,
Chris
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Sridhar Ratnakumar wrote:
Similar to "STATICDEPS=True python setup.py bdist_egg" in lxml?
If that lets me build and egg that I can then shove onto another box of the same OS and install with easy_install without installing any other dependencies, then yes, sure :-)
and build atlas yourself on linux.
Is there really no simpler way than following the huge and convolute process described here: http://www.scipy.org/Installing_SciPy/Linux#head-eecf834fad12bf7a62575252854...
In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.
...which is why I guess you can't rely on the operating system supplied ATLAS libraries if you want them to be statically linked?
To statically link libgfortran: on Mac, take a look at the libgfotran.a copying hack in pavement.py (scipy trunk); on Linux, I haven't yet figured out a way to reliably link libgfortran statically to numpy.]]
We're on Windows, Linux and maybe Mac... cheers, Chris
On 2010-04-30, at 5:58 AM, Chris Withers wrote:
Sridhar Ratnakumar wrote:
Similar to "STATICDEPS=True python setup.py bdist_egg" in lxml?
If that lets me build and egg that I can then shove onto another box of the same OS and install with easy_install without installing any other dependencies, then yes, sure :-)
Only lxml is aware of the STATICDEPS environment variable (see buildliblxml.py) and builds dependencies with static linking accordingly. AFAIK, no other package does this .. though I imagine someone can work on one of the following: 1. [PEP] Infrastructure to automatically build dependencies with shared or static linking 2. A cross-platform Macports/Portage like system to build 'external' dependencies for Python packages (Enthought may have something similar to this, as they have .egg for non-Python packages)
and build atlas yourself on linux.
Is there really no simpler way than following the huge and convolute process described here:
http://www.scipy.org/Installing_SciPy/Linux#head-eecf834fad12bf7a62575252854...
Yes, no other simpler documented process that I know of. The ActiveState PyPM repository does come with scipy/numpy linked with the ATLAS libraries (except Windows 64-bit which apparently does not work with ATLAS yet; and perhaps requires MKL). You will need the non-free Business Edition subscription to access these modules. Like virtualenv, you can install these modules on any virtualenv ("pypm -E /tmp/myvenv install scipy numpy") and any Linux/Windows/Mac machine. http://www.activestate.com/business_solutions/compare/ (Try it with the free 'pil' package, if you are interested)
In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.
...which is why I guess you can't rely on the operating system supplied ATLAS libraries if you want them to be statically linked?
To statically link libgfortran: on Mac, take a look at the libgfotran.a copying hack in pavement.py (scipy trunk); on Linux, I haven't yet figured out a way to reliably link libgfortran statically to numpy.]]
We're on Windows, Linux and maybe Mac...
Recently I figured out that to link with libgfortran (GPL) on Linux, you have to use GCC 4.3 or later, rebuild ATLAS with it and use the -static-libgfortran flag to the gfortran compiler. -srid
Sridhar Ratnakumar wrote:
For numpy you only need to use the pre-built ATLAS libraries (see scipy installation document) on windows 32-bit (for 64-bit, you will need the proprietary MKL to build atlas), and build atlas yourself on linux. In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.
As another tack, I tried, as I'm on Ubuntu 8.04LTS: sudo apt-get build-dep python-numpy bin/easy_install bin/easy_install numpy ...which left me with: Installed .../lib/python2.5/site-packages/numpy-1.4.1-py2.5-linux-x86_64.egg Processing dependencies for numpy Finished processing dependencies for numpy Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.5/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/tmp/easy_install-TFDAD2/numpy-1.4.1/numpy/distutils/misc_util.py", line 248, in clean_up_temporary_directory SystemError: Parent module 'numpy.distutils' not loaded Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.5/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/tmp/easy_install-TFDAD2/numpy-1.4.1/numpy/distutils/misc_util.py", line 248, in clean_up_temporary_directory SystemError: Parent module 'numpy.distutils' not loaded ...and yet: $ bin/python Python 2.5.2 (r252:60911, Jan 20 2010, 23:14:04) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import numpy
Any idea what those weird atexit handlers are supposed to do?! Chris PS: I would ask on the numpy or scipy mailing list if I could ever succeed in joining them...
participants (2)
-
Chris Withers -
Sridhar Ratnakumar