[Distutils] reservations about pythonv

Mark Sienkiewicz sienkiew at stsci.edu
Mon Mar 21 22:38:47 CET 2011


Carl Meyer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Mark,
>
> On 03/18/2011 01:46 PM, Mark Sienkiewicz wrote:
>   
>> Here is a case that might resemble what you are talking about:
>>
>> Compile a C extension that requires a shared library that is not in the
>> standard system path.  To import it, LD_LIBRARY_PATH needs to be right.
>>     
>
> But this required shared library that is not on the standard system path
> - - is it something that might have been installed, via Python
> (distutils/setuptools/d2/packaging), in the virtualenv, where if it had
> been installed in that same way in the system Python installation it
> would be on the standard system path?
>   

No.  I'm thinking of examples like this:

- In some of my Macs and Solaris machines, the BLAS library does not 
contain all of the functions that numpy wants.  I download the FORTRAN 
sources for BLAS, compile them, and put the libraries in 
$HOME/I_wish_I_had_root

- I need a newer version of the freetype library, so I ./configure 
--prefix $HOME/newer_freetype

python is never involved, except as an application that uses the shared 
libraries.

I have several different linux releases to support, so I have a lot of 
libraries that I install in order to have the same version on each machine.

If I had a way for python to automatically augment LD_LIBRARY_PATH when 
it starts up, I would use it, whether virtualenv is involved or not.

If I had a way to make distutils link shared libraries by a full path 
name instead of just libfoo.a, I would be happy to have that too.

In practice, I usually try to ensure that I have only static libraries 
available.  Everything gets statically linked, so I don't need 
LD_LIBRARY_PATH.


> If so, do you have any specific examples of installable Python projects
> that install a shared library that other ones might need in order to
> compile, so that I can test this case and make it work?
>   

I know of an example, but I don't have much to show you that I think 
would be useful.

We have two packages:

pywcs does coordinate transformations on astronomical images; it is a 
python wrapper for wcslib (or libwcs or something), written in C.

betadrizzle does some complex manipulations of multiple images.  In 
doing that, it sometimes needs to do huge numbers of coordinate 
transformations.

The betadrizzle C extension would like to call directly into the pywcs 
copy of wcslib, but without calling into python and back to C because of 
the overhead of doing it potentially millions of times.

I'm not sure what we are doing with that right now.  I remember 
suggesting that pywcs offer some C entry points through a jump table, 
but I haven't caught up with the guy doing the actual work to find out 
what he ended up doing.

Unfortunately, betadrizzle is still an experimental project and I'm not 
sure I could tell you how to compile it.  I am sure that I could not 
tell you how to run it.  But you see the idea.


> If not, then it seems this is a general Python packaging issue, not
> anything specific to this virtual-python proposal. In which case I don't
> have to care, at least not just now ;-)
>   

No, you don't have to care.  Shared libraries cause me way more problems 
than they solve, and nothing you do or don't do is going to change that. 
:( 

Mark S.



More information about the Distutils-SIG mailing list