[Distutils] 2 platforms down, only Mac to go...

Michael Twomey micktwomey at gmail.com
Sat Jan 14 14:12:23 CET 2006


(initially this was a quick "did it work for me" report, but I kept
hacking at it, so it rambles on a bit, with some partial solutions,
complete diff attached.)

On 14/01/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> I've got shared libraries working on Windows and Linux -- without any
> LD_LIBRARY_PATH nonsense, or any .so file rewriting.  I have not had a
> chance to try this on OS X, however.  If there's anybody out there who has
> experience wrestling with the Mac OS linker (hi, Bob! <wink>) has a chance
> to try it out and maybe tweak it a little bit, I'd appreciate it.

I just gave this a quick go using python 2.4 and 2.5 from trunk.

To get the _config_vars mangling to work it needs to be setup first
(you get an AttributeError on the copy() otherwise). distutils does
this in one of the get_confi_var* functions, so I stuck a quick hack
in to call this:

Index: setuptools/command/build_ext.py
--- setuptools/command/build_ext.py     (revision 42043)
+++ setuptools/command/build_ext.py     (working copy)
@@ -9,7 +9,9 @@
 from distutils.file_util import copy_file
 from setuptools.extension import Library
 from distutils.ccompiler import new_compiler
-from distutils.sysconfig import customize_compiler, _config_vars
+from distutils.sysconfig import customize_compiler, get_config_var
+from distutils.sysconfig import _config_vars
 from distutils import log
 from distutils.errors import *

> If this works without a bit of hacking, I'll be rather surprised, but
> *very* pleased.  If all three tests work without any changes to the source,
> it's probably a bug.  ;)  (Specifically, it probably means setuptools fell
> back to static linking, and thus isn't having any dynamic link problems.)

When I ran the tests they all worked, but setuptools had indeed built
a static library. Looking at the compiler output it looks like none of
the LDSHARED or CCSHARED flags you set got used.

Poking further I found that you are using the dl module to test for
RTLD_NOW, unfortunately dl isn't present on my mac. In fact setup.py
in python's source indicates that it isn't built at all:

if (dl_inc is not None) and (platform not in ['atheos', 'darwin']):
    exts.append( Extension('dl', ['dlmodule.c']) )

I temporarily hacked my copy to always set have_rtld to True and I got
errors when setup.py tried to use the flags. It looks like the
compiler has to be mentioned in the LDSHARED:

	tmp = _config_vars.copy()
	    # XXX Help!  I don't have any idea whether these are right...
	-                _config_vars['LDSHARED'] = "-dynamiclib -undefined
	+                _config_vars['LDSHARED'] = "gcc -Wl,-x -dynamiclib
-undefined dynamic_lookup"
	    _config_vars['CCSHARED'] = " -dynamiclib"
	    _config_vars['SO'] = ".dylib"

(For reference my flags:
$ python -c"import distutils.sysconfig; print
distutils.sysconfig.get_config_vars('LDSHARED', 'CCSHARED', 'SO')"
['gcc -Wl,-x -bundle -undefined dynamic_lookup', '', '.so']

(I've used distutils' get_config_var before to pull out compiler flags
and I remember having to do fun things to build up the LDSHARED across
platforms. I'll see if I have any code.)

Using these flags the module built fine. The otool output doesn't look
quite right though (note how the build directory is mentioned in the
built module, it leads to problems, see below):

$ otool -L libhellolib.dylib
(compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current
version 92.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 88.1.2)

$ otool -L dl-hello.so
(compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current
version 92.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 88.1.2)

When the test tries to use the module it barfs on the use of dl in the
bootstrap code (though it does import without linker errors):

testHelloMsg (test_hello.HelloWorldTest) ... ERROR

ERROR: testHelloMsg (test_hello.HelloWorldTest)
Traceback (most recent call last):
  File "/Users/Shared/Source/svn/python/setuptools/tests/shlib_test/test_hello.py",
line 5, in testHelloMsg
    from hello import hello
  File "./hello.py", line 17, in ?
  File "./hello.py", line 3, in __bootstrap__
ImportError: No module named dl

Ran 1 test in 0.035s

When I comment out the dl stuff in the bootstrap code things look a
lot better, the module imports and runs without problems.

I can move the directory to something_else and it still imports fine.

The only major niggle left is the way the linker has linked in the
build directory locations:

$ mv build/ build.broken
$ python -c "import hello"
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "./hello.py", line 17, in ?
  File "./hello.py", line 13, in __bootstrap__
ImportError: Failure linking new module:
Library not loaded:
  Referenced from:
  Reason: image not found

> The tricky bit -- if I understand correctly -- is that OS X has no concept
> of a runtime library search path, so what this probably *should* be doing
> is specifying something like "-dylib_file
> libhellolib.dylib:./libhellolib.dylib" in the link options for the 'hello'
> extension.  What I'm not clear on is whether this is actually allowed by
> the linker, or if it even does what I think it does in the first
> place.  Even if it's allowed and does the right thing, I have a sneaky
> suspicion that perhaps the path will be converted to an absolute path at
> link time, rather than leaving the './' in place, given the general
> reputation for deviousness possessed by the Mac OS linker.  :)

If I compile and link by hand all in the single directory then I get a
fully relocatable module free of references to the build directory.
It's looking like the linker is building paths relative to where it's
invoked from. Re-ordering the -L flags didn't seem to help either.

> On the other hand, perhaps it's possible to fudge the paths in the dylib
> once it's built, such that the './' is in the file?  Any thoughts or
> suggestions would be welcome.
> In a couple of weeks I'll have a chance to play around with this on a Mac
> myself, but since I'm starting from scratch (zero Mac experience
> whatsoever), I'm hoping maybe someone more experienced could provide some
> degree of guidance by then.  Thanks.

I'll see if I can tweak the flags a bit, I mightn't be able to come up
with anything but I thought you might like the early feedback. I'm
sure someone with more OS X linking experience than me will come up
with better solutions.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_ext.diff
Type: application/octet-stream
Size: 2244 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20060114/a7e86b5e/attachment.obj 

More information about the Distutils-SIG mailing list