Dynamic linking on Linux, Mac OS, et al
Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes.
This seems to be a bit of a problem, in that it appears to mean there's no sane way to include shared libraries in an egg on any platform other than Windows. It's common practice for Python extensions built for Windows to include shared libraries to avoid having to install them in the Windows system directories, but for Unix-like OSes this isn't really the normal practice anyway.
But, in the case where a developer wants to include a shared library as part of the distribution, this remains somewhat problematic. One can of course install the library to a platform-appropriate location ($prefix/lib on most platforms), but to do it from an egg is harder. You can't simply unzip the egg any more, you would have to also move the libraries to a standard installation location, breaking any hope of multi-version support. (And this would have to be able to happen on the fly, when shared libraries and extensions are extracted to the cache directory, if running zipped.)
I don't suppose you can modify LD_LIBRARY_PATH on the fly? Any other options? Should I just give up on supporting shared libraries on anything but Windows, perhaps silently converting the shared libraries to static ones?
On 1/6/06, Phillip J. Eby pje@telecommunity.com wrote:
Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes.
Are these just extensions that are imported? Because if they are, then you don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on Linux as long as PYTHONPATH is correct. -- -jeff
Jeff Pitman wrote:
Are these just extensions that are imported? Because if they are, then you don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on Linux as long as PYTHONPATH is correct.
No, they are things like libpng.so that the extension modules use.
Phillip J. Eby wrote:
Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes.
On OS X, the headers of the dylib can be rewritten to point to a specific path. IIRC, this can be a relative path. Bob Ippolito's py2app does this kind of rewriting for making relocatable .app bundles. I'm sure he'll chime in soon, so I'm not entirely sure why I bothered replying myself.
On Jan 5, 2006, at 9:00 PM, Robert Kern wrote:
Phillip J. Eby wrote:
Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes.
On OS X, the headers of the dylib can be rewritten to point to a specific path. IIRC, this can be a relative path. Bob Ippolito's py2app does this kind of rewriting for making relocatable .app bundles. I'm sure he'll chime in soon, so I'm not entirely sure why I bothered replying myself.
It's actually the extension that needs the rewriting.. but it's good to rewrite the dylib also.
It can be a relative path, but only relative to the executable. You can be relative to the bundle on Mac OS X 10.4+, but it really ought to just make the path static upon installation and remain backwards compatible. In order to make that bulletproof, the linker flag - headerpad_max_install_names should be used.
-bob
Phillip J. Eby wrote:
Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes.
This seems to be a bit of a problem, in that it appears to mean there's no sane way to include shared libraries in an egg on any platform other than Windows. It's common practice for Python extensions built for Windows to include shared libraries to avoid having to install them in the Windows system directories, but for Unix-like OSes this isn't really the normal practice anyway.
But, in the case where a developer wants to include a shared library as part of the distribution, this remains somewhat problematic. One can of course install the library to a platform-appropriate location ($prefix/lib on most platforms), but to do it from an egg is harder. You can't simply unzip the egg any more, you would have to also move the libraries to a standard installation location, breaking any hope of multi-version support. (And this would have to be able to happen on the fly, when shared libraries and extensions are extracted to the cache directory, if running zipped.)
I don't suppose you can modify LD_LIBRARY_PATH on the fly? Any other options? Should I just give up on supporting shared libraries on anything but Windows, perhaps silently converting the shared libraries to static ones?
On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options. This doesn't work on Mac OS X, but it sounds like someone else may know how to make this work.
BTW, I often have need to be able to create what we call "application buildouts" that contain various facilities needed by a particular application. These facilities are mostly Python-based but often include things like relational databases, ICU, ldap servers, etc. We want to be able to have buildouts for multiple applications (or multiple application versions) on a single development machine. Each buildout needs to be able to control versions of things it uses (including Python versions). We try to minimize the dependence on system libraries, as these are usually different on development and deployment systems and because we may need different versions for different applications on a development machine. I'm just beginning to learn about eggs. I certainly hope I'll have the same flexibility when installing eggs that I have now when using distutils.
Jim
At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote:
On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options.
But that's not relocatable (to different directories), and eggs are intended to be "zero-install" for simple use cases. So, the solution I came up with uses '.' in the rpath, and a stub loader that changes directories long enough to get the extension to load and find its library (or libraries). This appears to work decently enough on Linux.
BTW, I often have need to be able to create what we call "application buildouts" that contain various facilities needed by a particular application. These facilities are mostly Python-based but often include things like relational databases, ICU, ldap servers, etc. We want to be able to have buildouts for multiple applications (or multiple application versions) on a single development machine. Each buildout needs to be able to control versions of things it uses (including Python versions). We try to minimize the dependence on system libraries, as these are usually different on development and deployment systems and because we may need different versions for different applications on a development machine. I'm just beginning to learn about eggs. I certainly hope I'll have the same flexibility when installing eggs that I have now when using distutils.
I'm not sure I can answer that question, or that I even know what your question is, actually. :) Eggs don't really try to encompass building arbitrary executables, only Python code, C/C++/Pyrex/SWIG extensions, and libraries linked to the extensions. It sort of sounds like you're saying you'd like to be able to build a bunch of non-Python stuff too, which is likely to remain out of scope for the foreseeable future. If you have managed to shoehorn the distutils into doing it for you, great, but I would be surprised if such extensions would be egg-able in any meaningful way.
Phillip J. Eby wrote:
At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote:
On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options.
But that's not relocatable (to different directories), and eggs are intended to be "zero-install" for simple use cases. So, the solution I came up with uses '.' in the rpath, and a stub loader that changes directories
Changes directories? Uh ... this sounds brittle.
long enough to get the extension to load and find its library (or libraries). This appears to work decently enough on Linux.
BTW, I often have need to be able to create what we call "application buildouts" that contain various facilities needed by a particular application. These facilities are mostly Python-based but often include things like relational databases, ICU, ldap servers, etc. We want to be able to have buildouts for multiple applications (or multiple application versions) on a single development machine. Each buildout needs to be able to control versions of things it uses (including Python versions). We try to minimize the dependence on system libraries, as these are usually different on development and deployment systems and because we may need different versions for different applications on a development machine. I'm just beginning to learn about eggs. I certainly hope I'll have the same flexibility when installing eggs that I have now when using distutils.
I'm not sure I can answer that question, or that I even know what your question is, actually. :) Eggs don't really try to encompass building arbitrary executables, only Python code, C/C++/Pyrex/SWIG extensions, and libraries linked to the extensions. It sort of sounds like you're saying you'd like to be able to build a bunch of non-Python stuff too, which is likely to remain out of scope for the foreseeable future.
No. I want to be able to get a Python egg built in such a way that it uses a library in a location I tell it to, rather than in the standard system libraries. For example, suppose I have an egg with an extension that uses ICU, and that I've installed the ICU libraries in a non-standard place. When I build the egg, I want to tell it to use the location that I specify for ICU. I can do this with setup now with the -R option to build_ext.
Jim
At 04:01 PM 1/14/2006 -0500, Jim Fulton wrote:
Phillip J. Eby wrote:
At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote:
On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options.
But that's not relocatable (to different directories), and eggs are intended to be "zero-install" for simple use cases. So, the solution I came up with uses '.' in the rpath, and a stub loader that changes directories
Changes directories? Uh ... this sounds brittle.
Yep. All the solutions are, unfortunately, except on Windows where it just works in a sane way to start with.
No. I want to be able to get a Python egg built in such a way that it uses a library in a location I tell it to, rather than in the standard system libraries. For example, suppose I have an egg with an extension that uses ICU, and that I've installed the ICU libraries in a non-standard place. When I build the egg, I want to tell it to use the location that I specify for ICU. I can do this with setup now with the -R option to build_ext.
Oh. Sure; you can set it on the command line, e.g. "build_ext -R whatever bdist_egg", or via any of the normal distutils hooks, including the various Extension() keyword arguments, config files, etc.
participants (5)
-
Bob Ippolito -
Jeff Pitman -
Jim Fulton -
Phillip J. Eby -
Robert Kern