2 platforms down, only Mac to go...
I've got shared libraries working on Windows and Linux -- without any LD_LIBRARY_PATH nonsense, or any .so file rewriting. I have not had a chance to try this on OS X, however. If there's anybody out there who has experience wrestling with the Mac OS linker (hi, Bob! <wink>) has a chance to try it out and maybe tweak it a little bit, I'd appreciate it. The code is in the subversion trunk now, and the source includes a tests/shlib_test subdirectory that you can change to and run "setup.py test" to build/link/test, e.g.: cd tests/shlib_test PYTHONPATH=../.. python setup.py test This just verifies that the basic linking works, i.e., whether my guesses for the linker options in build_ext.py were correct. The second test is to verify that it works from a different directory: cd .. PYTHONPATH=..:shlib_test python -c "import hello" If this works without crashing, then the dynamic linking works even from a different directory. Based on my limited guesswork, the final test is the one most likely to fail on OS X (assuming the link step succeeded, and I'm not sure it will!). Rename the test directory temporarily: mv shlib_test something_else PYTHONPATH=..:something_else python -c "import hello" mv something_else shlib_test If this works without a bit of hacking, I'll be rather surprised, but *very* pleased. If all three tests work without any changes to the source, it's probably a bug. ;) (Specifically, it probably means setuptools fell back to static linking, and thus isn't having any dynamic link problems.) The basic idea of what I'm doing here is that I try to set the runtime library search path of each extension (that depends on a shared library in the project), to include the current directory *at runtime*, then make a Python wrapper for the extension that temporarily changes the current directory while doing the import. The tricky bit -- if I understand correctly -- is that OS X has no concept of a runtime library search path, so what this probably *should* be doing is specifying something like "-dylib_file libhellolib.dylib:./libhellolib.dylib" in the link options for the 'hello' extension. What I'm not clear on is whether this is actually allowed by the linker, or if it even does what I think it does in the first place. Even if it's allowed and does the right thing, I have a sneaky suspicion that perhaps the path will be converted to an absolute path at link time, rather than leaving the './' in place, given the general reputation for deviousness possessed by the Mac OS linker. :) On the other hand, perhaps it's possible to fudge the paths in the dylib once it's built, such that the './' is in the file? Any thoughts or suggestions would be welcome. In a couple of weeks I'll have a chance to play around with this on a Mac myself, but since I'm starting from scratch (zero Mac experience whatsoever), I'm hoping maybe someone more experienced could provide some degree of guidance by then. Thanks.
Hi,
(initially this was a quick "did it work for me" report, but I kept
hacking at it, so it rambles on a bit, with some partial solutions,
complete diff attached.)
On 14/01/06, Phillip J. Eby
I've got shared libraries working on Windows and Linux -- without any LD_LIBRARY_PATH nonsense, or any .so file rewriting. I have not had a chance to try this on OS X, however. If there's anybody out there who has experience wrestling with the Mac OS linker (hi, Bob! <wink>) has a chance to try it out and maybe tweak it a little bit, I'd appreciate it.
I just gave this a quick go using python 2.4 and 2.5 from trunk. To get the _config_vars mangling to work it needs to be setup first (you get an AttributeError on the copy() otherwise). distutils does this in one of the get_confi_var* functions, so I stuck a quick hack in to call this: Index: setuptools/command/build_ext.py =================================================================== --- setuptools/command/build_ext.py (revision 42043) +++ setuptools/command/build_ext.py (working copy) @@ -9,7 +9,9 @@ from distutils.file_util import copy_file from setuptools.extension import Library from distutils.ccompiler import new_compiler -from distutils.sysconfig import customize_compiler, _config_vars +from distutils.sysconfig import customize_compiler, get_config_var +get_config_var("LDSHARED") +from distutils.sysconfig import _config_vars from distutils import log from distutils.errors import *
If this works without a bit of hacking, I'll be rather surprised, but *very* pleased. If all three tests work without any changes to the source, it's probably a bug. ;) (Specifically, it probably means setuptools fell back to static linking, and thus isn't having any dynamic link problems.)
When I ran the tests they all worked, but setuptools had indeed built a static library. Looking at the compiler output it looks like none of the LDSHARED or CCSHARED flags you set got used. Poking further I found that you are using the dl module to test for RTLD_NOW, unfortunately dl isn't present on my mac. In fact setup.py in python's source indicates that it isn't built at all: if (dl_inc is not None) and (platform not in ['atheos', 'darwin']): exts.append( Extension('dl', ['dlmodule.c']) ) I temporarily hacked my copy to always set have_rtld to True and I got errors when setup.py tried to use the flags. It looks like the compiler has to be mentioned in the LDSHARED: tmp = _config_vars.copy() try: # XXX Help! I don't have any idea whether these are right... - _config_vars['LDSHARED'] = "-dynamiclib -undefined dynamic_lookup" + _config_vars['LDSHARED'] = "gcc -Wl,-x -dynamiclib -undefined dynamic_lookup" _config_vars['CCSHARED'] = " -dynamiclib" _config_vars['SO'] = ".dylib" customize_compiler(compiler) (For reference my flags: $ python -c"import distutils.sysconfig; print distutils.sysconfig.get_config_vars('LDSHARED', 'CCSHARED', 'SO')" ['gcc -Wl,-x -bundle -undefined dynamic_lookup', '', '.so'] ) (I've used distutils' get_config_var before to pull out compiler flags and I remember having to do fun things to build up the LDSHARED across platforms. I'll see if I have any code.) Using these flags the module built fine. The otool output doesn't look quite right though (note how the build directory is mentioned in the built module, it leads to problems, see below): $ otool -L libhellolib.dylib libhellolib.dylib: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.2) $ otool -L dl-hello.so dl-hello.so: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.2) When the test tries to use the module it barfs on the use of dl in the bootstrap code (though it does import without linker errors): testHelloMsg (test_hello.HelloWorldTest) ... ERROR ====================================================================== ERROR: testHelloMsg (test_hello.HelloWorldTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/Shared/Source/svn/python/setuptools/tests/shlib_test/test_hello.py", line 5, in testHelloMsg from hello import hello File "./hello.py", line 17, in ? File "./hello.py", line 3, in __bootstrap__ ImportError: No module named dl ---------------------------------------------------------------------- Ran 1 test in 0.035s When I comment out the dl stuff in the bootstrap code things look a lot better, the module imports and runs without problems. I can move the directory to something_else and it still imports fine. The only major niggle left is the way the linker has linked in the build directory locations: $ mv build/ build.broken $ python -c "import hello" Traceback (most recent call last): File "<string>", line 1, in ? File "./hello.py", line 17, in ? File "./hello.py", line 13, in __bootstrap__ ImportError: Failure linking new module: /Users/Shared/Source/svn/python/setuptools/tests/shlib_test/dl-hello.so: Library not loaded: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib Referenced from: /Users/Shared/Source/svn/python/setuptools/tests/shlib_test/dl-hello.so Reason: image not found
The tricky bit -- if I understand correctly -- is that OS X has no concept of a runtime library search path, so what this probably *should* be doing is specifying something like "-dylib_file libhellolib.dylib:./libhellolib.dylib" in the link options for the 'hello' extension. What I'm not clear on is whether this is actually allowed by the linker, or if it even does what I think it does in the first place. Even if it's allowed and does the right thing, I have a sneaky suspicion that perhaps the path will be converted to an absolute path at link time, rather than leaving the './' in place, given the general reputation for deviousness possessed by the Mac OS linker. :)
If I compile and link by hand all in the single directory then I get a fully relocatable module free of references to the build directory. It's looking like the linker is building paths relative to where it's invoked from. Re-ordering the -L flags didn't seem to help either.
On the other hand, perhaps it's possible to fudge the paths in the dylib once it's built, such that the './' is in the file? Any thoughts or suggestions would be welcome.
In a couple of weeks I'll have a chance to play around with this on a Mac myself, but since I'm starting from scratch (zero Mac experience whatsoever), I'm hoping maybe someone more experienced could provide some degree of guidance by then. Thanks.
I'll see if I can tweak the flags a bit, I mightn't be able to come up with anything but I thought you might like the early feedback. I'm sure someone with more OS X linking experience than me will come up with better solutions. cheers, Michael
At 01:12 PM 1/14/2006 +0000, Michael Twomey wrote:
When I ran the tests they all worked, but setuptools had indeed built a static library. Looking at the compiler output it looks like none of the LDSHARED or CCSHARED flags you set got used.
Poking further I found that you are using the dl module to test for RTLD_NOW, unfortunately dl isn't present on my mac. In fact setup.py in python's source indicates that it isn't built at all:
if (dl_inc is not None) and (platform not in ['atheos', 'darwin']): exts.append( Extension('dl', ['dlmodule.c']) )
I temporarily hacked my copy to always set have_rtld to True and I got errors when setup.py tried to use the flags.
Okay, I've changed this so that the 'dl' stuff is only done on non-Mac platforms, and Mac is still recognized as needing stub loaders.
It looks like the compiler has to be mentioned in the LDSHARED:
Fixed in SVN.
Using these flags the module built fine. The otool output doesn't look quite right though (note how the build directory is mentioned in the built module, it leads to problems, see below):
$ otool -L libhellolib.dylib libhellolib.dylib: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.2)
$ otool -L dl-hello.so dl-hello.so: build/lib.darwin-8.4.0-Power_Macintosh-2.4/libhellolib.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.2)
It looks like we probably need to add "-install_name libhellolib.dylib" or maybe "-dylib_install_name libhellolib.dylib" to the library link step for hellolib, in order to get that out of there.
When I comment out the dl stuff in the bootstrap code things look a lot better, the module imports and runs without problems. I can move the directory to something_else and it still imports fine.
Yeah, it looks like that's only because it used the build directory, relative to the extension's location. :(
If I compile and link by hand all in the single directory then I get a fully relocatable module free of references to the build directory. It's looking like the linker is building paths relative to where it's invoked from.
So, you're using the same options supplied by the setup script, just not using paths under build/? Does it still then work with all the test cases?
I'll see if I can tweak the flags a bit, I mightn't be able to come up with anything but I thought you might like the early feedback. I'm sure someone with more OS X linking experience than me will come up with better solutions.
I appreciate it! It sounds like we might be getting close. I wonder, though, if perhaps what's happening in the configuration that currently works, is that the path is being interpreted as "bundle relative" to the extension. This is fine and is really what we want, but I'm not sure that it can be done with older Mac OSes. I seem to recall that bundle-relative linking was added in 10.4? Or was it 10.3? I need to start actually *using* a Mac one of these days. :)
Hi again,
On 14/01/06, Phillip J. Eby
Okay, I've changed this so that the 'dl' stuff is only done on non-Mac platforms, and Mac is still recognized as needing stub loaders.
Working for me now.
It looks like the compiler has to be mentioned in the LDSHARED:
Fixed in SVN.
This too.
It looks like we probably need to add "-install_name libhellolib.dylib" or maybe "-dylib_install_name libhellolib.dylib" to the library link step for hellolib, in order to get that out of there.
-install_name did the trick (in my manual compilation anyway). The man page says it all: If this is not specified, the name specified in the -o name option will be used. So this worked: gcc -Wl,-x -dynamiclib -undefined dynamic_lookup $tempdir/hellolib.o \ -install_name libhellolib.dylib -o $libdir/libhellolib.dylib Interestingly you can use "-install_name [any_path]libhellolib.dylib" too, so ../ and ./ combinations are accepted as well. -dylib_install_name didn't work (which surprised me), I got this error: powerpc-apple-darwin8-gcc-4.0.1: libhellolib.dylib: No such file or directory This implies that it was trying to look for libhellolib.dylib before it was built, which is odd.
If I compile and link by hand all in the single directory then I get a fully relocatable module free of references to the build directory. It's looking like the linker is building paths relative to where it's invoked from.
So, you're using the same options supplied by the setup script, just not using paths under build/? Does it still then work with all the test cases?
I copied the output of distutils into a shell script and modified by hand. Based on the comment in the man page above it was using the name from the -o flag which is why it worked when I built manually. It worked in the test cases.
I'll see if I can tweak the flags a bit, I mightn't be able to come up with anything but I thought you might like the early feedback. I'm sure someone with more OS X linking experience than me will come up with better solutions.
I appreciate it! It sounds like we might be getting close.
It looks like with the libhellolib.dylib built with the "-install_name libhellolib.dylib" does the trick. Everything works fine when I try the various tests. I've attached the shell script I was using to build and test for reference.
I wonder, though, if perhaps what's happening in the configuration that currently works, is that the path is being interpreted as "bundle relative" to the extension. This is fine and is really what we want, but I'm not sure that it can be done with older Mac OSes. I seem to recall that bundle-relative linking was added in 10.4? Or was it 10.3? I need to start actually *using* a Mac one of these days. :)
I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), I'm pretty sure it was the "-undefined dynamic_lookup" which got introduced, which had a big impact on shared libraries in bundles. I'm not sure how this would affect the linking stuff above, it is "used to allow any undefined symbols to be looked up dynamically at runtime" according to the man page. cheers, Michael
On Jan 16, 2006, at 3:21 PM, Michael Twomey wrote:
On 14/01/06, Phillip J. Eby
wrote: I wonder, though, if perhaps what's happening in the configuration that currently works, is that the path is being interpreted as "bundle relative" to the extension. This is fine and is really what we want, but I'm not sure that it can be done with older Mac OSes. I seem to recall that bundle-relative linking was added in 10.4? Or was it 10.3? I need to start actually *using* a Mac one of these days. :)
I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), I'm pretty sure it was the "-undefined dynamic_lookup" which got introduced, which had a big impact on shared libraries in bundles. I'm not sure how this would affect the linking stuff above, it is "used to allow any undefined symbols to be looked up dynamically at runtime" according to the man page.
You are correct, MACOSX_DEPLOYMENT_TARGET=10.3 allows you to use - undefined dynamic_lookup. However, requiring Mac OS X 10.4 would allow you to use @loader_path [1], which would let you skip out on the chdir hack. [1] http://developer.apple.com/releasenotes/DeveloperTools/dyld.html -bob
At 03:32 PM 01/16/2006 -0800, Bob Ippolito wrote:
On Jan 16, 2006, at 3:21 PM, Michael Twomey wrote:
On 14/01/06, Phillip J. Eby
wrote: I wonder, though, if perhaps what's happening in the configuration that currently works, is that the path is being interpreted as "bundle relative" to the extension. This is fine and is really what we want, but I'm not sure that it can be done with older Mac OSes. I seem to recall that bundle-relative linking was added in 10.4? Or was it 10.3? I need to start actually *using* a Mac one of these days. :)
I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), I'm pretty sure it was the "-undefined dynamic_lookup" which got introduced, which had a big impact on shared libraries in bundles. I'm not sure how this would affect the linking stuff above, it is "used to allow any undefined symbols to be looked up dynamically at runtime" according to the man page.
You are correct, MACOSX_DEPLOYMENT_TARGET=10.3 allows you to use - undefined dynamic_lookup. However, requiring Mac OS X 10.4 would allow you to use @loader_path [1], which would let you skip out on the chdir hack.
[1] http://developer.apple.com/releasenotes/DeveloperTools/dyld.html
So, should setuptools maybe check the deployment target and use that to decide whether to chdir-hack or to use -install_name @loader_path/blah.dylib? That won't make the current platform tests and build customization any more complex on a fundamental level. I already have to decide whether to use a chdir loader (not needed at all on Windows) and whether the loader should use dl (Linux) or not (Mac). So, adding an environment variable check doesn't sound like a big deal. Do any OS X versions prior to 10.3 even have Python installed, btw? I'm figuring I can live with not supporting those, though I should double-check what OSAF is targeting, of course.
On Jan 16, 2006, at 6:09 PM, Phillip J. Eby wrote:
At 03:32 PM 01/16/2006 -0800, Bob Ippolito wrote:
On Jan 16, 2006, at 3:21 PM, Michael Twomey wrote:
On 14/01/06, Phillip J. Eby
wrote: I wonder, though, if perhaps what's happening in the configuration that currently works, is that the path is being interpreted as "bundle relative" to the extension. This is fine and is really what we want, but I'm not sure that it can be done with older Mac OSes. I seem to recall that bundle-relative linking was added in 10.4? Or was it 10.3? I need to start actually *using* a Mac one of these days. :)
I think it was in 10.3 (the MACOSX_DEPLOYMENT_TARGET affects this), I'm pretty sure it was the "-undefined dynamic_lookup" which got introduced, which had a big impact on shared libraries in bundles. I'm not sure how this would affect the linking stuff above, it is "used to allow any undefined symbols to be looked up dynamically at runtime" according to the man page.
You are correct, MACOSX_DEPLOYMENT_TARGET=10.3 allows you to use - undefined dynamic_lookup. However, requiring Mac OS X 10.4 would allow you to use @loader_path [1], which would let you skip out on the chdir hack.
[1] http://developer.apple.com/releasenotes/DeveloperTools/dyld.html
So, should setuptools maybe check the deployment target and use that to decide whether to chdir-hack or to use -install_name @loader_path/blah.dylib? That won't make the current platform tests and build customization any more complex on a fundamental level. I already have to decide whether to use a chdir loader (not needed at all on Windows) and whether the loader should use dl (Linux) or not (Mac). So, adding an environment variable check doesn't sound like a big deal.
Apple has actually shifted their recommended runtime dynamic linkage API to plain old dl.. it's part of libc in Mac OS X 10.4, available for dynamic linkage in Mac OS X 10.3, and provided as a static library for earlier versions of OS X. It might be worth considering shifting Python 2.5 towards that API instead of the old NeXT-style APIs. I don't want to do that work, of course, the APIs are equivalent and what we have works. Personally I'd probably just go with the chdir loader for now, because people are still running 10.3. In a year or two, it might be worth dropping 10.3 support and switching over to @loader_path and removing the cruft. Until then, I don't see the benefit of maintaining two code paths.
Do any OS X versions prior to 10.3 even have Python installed, btw? I'm figuring I can live with not supporting those, though I should double-check what OSAF is targeting, of course.
OSAF is not going to want to target any vendor-installed Python. They should be bundling a self-contained Python installation with Chandler (as I'm sure they do on Windows, too). If they do depend on a vendor-installed Python, they'll have to have separate versions of everything for almost every OS release. I wouldn't even worry about it. To answer your question, 10.2 had a horrifically bad build of Python 2.2.0. -bob
participants (3)
-
Bob Ippolito
-
Michael Twomey
-
Phillip J. Eby