[Distutils] reservations about pythonv
sienkiew at stsci.edu
Mon Mar 21 22:38:47 CET 2011
Carl Meyer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Hi Mark,
> On 03/18/2011 01:46 PM, Mark Sienkiewicz wrote:
>> Here is a case that might resemble what you are talking about:
>> Compile a C extension that requires a shared library that is not in the
>> standard system path. To import it, LD_LIBRARY_PATH needs to be right.
> But this required shared library that is not on the standard system path
> - - is it something that might have been installed, via Python
> (distutils/setuptools/d2/packaging), in the virtualenv, where if it had
> been installed in that same way in the system Python installation it
> would be on the standard system path?
No. I'm thinking of examples like this:
- In some of my Macs and Solaris machines, the BLAS library does not
contain all of the functions that numpy wants. I download the FORTRAN
sources for BLAS, compile them, and put the libraries in
- I need a newer version of the freetype library, so I ./configure
python is never involved, except as an application that uses the shared
I have several different linux releases to support, so I have a lot of
libraries that I install in order to have the same version on each machine.
If I had a way for python to automatically augment LD_LIBRARY_PATH when
it starts up, I would use it, whether virtualenv is involved or not.
If I had a way to make distutils link shared libraries by a full path
name instead of just libfoo.a, I would be happy to have that too.
In practice, I usually try to ensure that I have only static libraries
available. Everything gets statically linked, so I don't need
> If so, do you have any specific examples of installable Python projects
> that install a shared library that other ones might need in order to
> compile, so that I can test this case and make it work?
I know of an example, but I don't have much to show you that I think
would be useful.
We have two packages:
pywcs does coordinate transformations on astronomical images; it is a
python wrapper for wcslib (or libwcs or something), written in C.
betadrizzle does some complex manipulations of multiple images. In
doing that, it sometimes needs to do huge numbers of coordinate
The betadrizzle C extension would like to call directly into the pywcs
copy of wcslib, but without calling into python and back to C because of
the overhead of doing it potentially millions of times.
I'm not sure what we are doing with that right now. I remember
suggesting that pywcs offer some C entry points through a jump table,
but I haven't caught up with the guy doing the actual work to find out
what he ended up doing.
Unfortunately, betadrizzle is still an experimental project and I'm not
sure I could tell you how to compile it. I am sure that I could not
tell you how to run it. But you see the idea.
> If not, then it seems this is a general Python packaging issue, not
> anything specific to this virtual-python proposal. In which case I don't
> have to care, at least not just now ;-)
No, you don't have to care. Shared libraries cause me way more problems
than they solve, and nothing you do or don't do is going to change that.
More information about the Distutils-SIG