Re: [Distutils] Dynamic linking on Linux, Mac OS, et al
At 12:29 PM 1/6/2006 +0800, Jeff Pitman wrote:
Are these just extensions that are imported? Because if they are, then you don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on Linux as long as PYTHONPATH is correct. -- -jeff
Nope. They're shared libraries being built by the distribution itself, as in the case of PyICU, and certain Windows C extensions that need to include the target library (e.g. sqlite) because it's not a "system" library there. It could perhaps be questioned whether it's a good idea to include any other kind of shared library in an extension project, and in fact the common practice to link between Python extensions is to use a Python CObject to wrap an API function pointer table. This neatly bypasses all of the issues... *if* you control the library in question. So far I've done a few experiments with trying to hack LD_LIBRARY_PATH at runtime, use dl.open() with RTLD_GLOBAL, etc. to force finding the right library, but so far it seems like there's no way to get the extensions to consider the opened library to be "the same" as the one they're looking for. (Although it may just be that I'm missing some trick on the build side of the equation.)
Hi, Phillip J. Eby:
It could perhaps be questioned whether it's a good idea to include any other kind of shared library in an extension project
It is not. You will end up with multiple copies of extension libraries, all subtly different, scattered in various .egg directories. You guess what happens if two of these try to use the "same" library (hint: it's not pretty and even less debuggable). You guess what happens, or rather what will not happen, if a security problem is discovered in one of these libraries. Besides, Unix systems these days tend to have the required libraries installed anyway. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - "Pok pok pok, P'kok!" -- Superchicken
At 06:20 AM 1/6/2006 +0100, Matthias Urlichs wrote:
Phillip J. Eby:
It could perhaps be questioned whether it's a good idea to include any other kind of shared library in an extension project
It is not.
You will end up with multiple copies of extension libraries, all subtly different, scattered in various .egg directories. You guess what happens if two of these try to use the "same" library (hint: it's not pretty and even less debuggable). You guess what happens, or rather what will not happen, if a security problem is discovered in one of these libraries.
Um, the "other kind" I'm referring to here are the ones that are exclusively part of the extension in question, e.g. libPyICU. This library is not going to be found anywhere else except in other versions of PyICU, so that issue doesn't really apply here, since only one egg will have extensions loading the library into a given process.
Besides, Unix systems these days tend to have the required libraries installed anyway.
Not libPyICU, since it's a part of PyICU. Similar issues may apply to other multi-extension projects that need to share a large chunk of code.
Hi, Phillip J. Eby:
Um, the "other kind" I'm referring to here are the ones that are exclusively part of the extension in question, e.g. libPyICU. This library is not going to be found anywhere else except in other versions of PyICU, so that issue doesn't really apply here, since only one egg will have extensions loading the library into a given process.
So why don't you simply link the Python extension and the library to one single .so file? -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - BOFH excuse #263: It's stuck in the Web.
At 05:43 PM 1/6/2006 +0100, Matthias Urlichs wrote:
Hi,
Um, the "other kind" I'm referring to here are the ones that are exclusively part of the extension in question, e.g. libPyICU. This
Phillip J. Eby: library
is not going to be found anywhere else except in other versions of PyICU, so that issue doesn't really apply here, since only one egg will have extensions loading the library into a given process.
So why don't you simply link the Python extension and the library to one single .so file?
Because there are a half-dozen Python extensions in the project sharing the library. Hence, the need for a "shared library". :)
Hi, Phillip J. Eby:
So why don't you simply link the Python extension and the library to one single .so file?
Because there are a half-dozen Python extensions in the project sharing the library. Hence, the need for a "shared library". :)
So put them all into one .so file and import the required parts from the modules you want them in. Anyway, ELF libraries do have an rpath embedded in them. It *is* possible to hotpatch that to point at wherever the shared library ends up being installed. Debian has a package "chrpath" with a tool "chrpath" (what a surprise ;-) which can do that. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - "I daresay," said Granny, pushing the Fool aside and stepping over a writhing taproot. "If anyone locked *me* in a dungeon, there'd be screams." -- Terry Pratchett (Wyrd Sisters)
On 1/7/06, Phillip J. Eby
At 06:20 AM 1/6/2006 +0100, Matthias Urlichs wrote:
Besides, Unix systems these days tend to have the required libraries installed anyway.
Not libPyICU, since it's a part of PyICU. Similar issues may apply to other multi-extension projects that need to share a large chunk of code.
Whether this is the right way to do things is up for debate. Earlier in the thread libpng.so was thrown out as example--this immediately threw some red flags up because it's a common/standard lib that comes with most distros. If you need .so that's built and exported by the same packaged egg, you may just consider installing it in a more benign area such as /usr/local/lib. Though, the *BSD folks may also have some thoughts on the matter. Messing with LD_LIBRARY_PATH is never the right way to go about solving this. Unless you're Oracle... Please keep in mind that the different *nixes will want to stick with their own package formats, managers, and automatic dep solvers. Enhancing setuptools to facilitate these (bdist_rpm/bdist_deb), will be more beneficial in the long run. -- -jeff
On 1/6/06, Phillip J. Eby
At 12:29 PM 1/6/2006 +0800, Jeff Pitman wrote:
Are these just extensions that are imported? Because if they are, then you don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on Linux as long as PYTHONPATH is correct. -- -jeff
Nope. They're shared libraries being built by the distribution itself, as in the case of PyICU, and certain Windows C extensions that need to include the target library (e.g. sqlite) because it's not a "system" library there.
Unless you do something really magical, you're going to find a lot of resistance to this in most Linux distro camps. Gentoo will want to recompile via emerge. Fedora/CentOS will want to grab from yum. Debian will want to grab libs via apt. This is why I made the comment that the focus on Linux should really be all about $HOME on servers where the user does not admin the box. Because, in a sense, you're looking to overwrite or replace existing infrastructure and since eggs/setuptools has no visual into rpm/deb, things will get messy. And, admins, for the most part, don't like messes. -- -jeff
participants (3)
-
Jeff Pitman
-
Matthias Urlichs
-
Phillip J. Eby