[Pythonmac-SIG] dlopenflags and sharing symbols across python extension modules

[Bob Ippolito]

On Mar 5, 2006, at 4:16 PM, Zachary Pincus wrote:

> Thanks for the information, Bob. I appreciate your help.
>
>> Actually it's because Python extensions are bundles.  Bundles are
>> NOT shared libraries and can not do what you want.  Additionally,
>> Python on Mac OS X doesn't use dl to load its extensions, it uses
>> lower-level means... so the RTLD flags aren't going to do anything.
>
> If python isn't using dl to load extensions on the Mac, then I would
> suggest that sys.get/setdlopenflags() should not be made available on
> mac builds (as it is not in windows builds). This looks like a pretty
> easy patch to sysmodule.c.

Eh.  It'd be better to just switch to dl, but that would break 10.2  
support.

>> The "best" you can do is to build shared libraries that all of your
>> extensions link to.
>
> Really? There's no way that different bundles can be loaded such that
> weak external references will be properly shared between the bundles?
> That's a shame, because it really unfits python on the mac for
> modules which use functionality from C++ libraries. It's a double
> shame, since Python on Linux handles this case just fine.
>
> Are bundles in general unable to deal with shared global symbols,
> which are required by C++ for RTTI, or is it just how Python is
> dealing with the bundles?

MH_BUNDLE is not MH_DYLIB and neither are ELF.  Everything you think  
you know about shared libraries isn't quite the same on Mac OS X.  I  
suggest you read up on Mach-O, I don't really have time to answer all  
of your questions about it.

> Anyhow, I assume that there are good reasons for how this all works,
> so I will probably have to go the shared library route. Here's what I
> imagine I would need to do:
> (1) Create shared libraries for each python module
> (2) Write "stub" python modules that only serve to dlopen() the
> shared libraries (with the GLOBAL flag)
> (3) Presumably all the stub would do is declare an initialization
> function that dlopens the shared lib and calls the shared lib's
> initialization function?
>
> Basically, this looks like it would be replicating how Python loads
> modules on Linux. It's too bad that's how it would have to be done,
> unless there's a better way that I can't think of right now. Anyone
> have any thoughts/suggestions/guidance?

It'd probably factor it such that the shared stuff (C++ classes) is  
in shared libraries and the Python C API stuff is in the extensions.

-bob