[Pythonmac-SIG] dlopenflags and sharing symbols across python extension modules

[Zachary Pincus]

Thanks again Bob for the feedback.

> MH_BUNDLE is not MH_DYLIB and neither are ELF.  Everything you  
> think you know about shared libraries isn't quite the same on Mac  
> OS X.  I suggest you read up on Mach-O, I don't really have time to  
> answer all of your questions about it.

My admittedly limited knowledge of shared libraries is primarily mac- 
based -- I only brought up the ELF cases by way of comparison. At any  
rate, regardless of differences between the formats they *all* have a  
way to the support weak external references needed for C++ RTTI with  
templates.

The sections Apple's of documentation about coalesced sections and  
weak definitions discuss how Mach-O supports these features. I think  
it's not really that different from how they are supported on ELF.
http://developer.apple.com/documentation/DeveloperTools/Conceptual/ 
MachORuntime/Reference/reference.html

The problem is that on Macs, Python is not exposing the necessary  
mechanism for loading modules so that weak external references are  
resolved. On Linux and other platforms where Python respects  
sys.setdlopenflags(), this mechanism *is* exposed.

Specifically, the problem is here:
http://svn.python.org/view/python/trunk/Python/dynload_next.c? 
rev=36531&view=auto

Python is using NSCreateObjectFileImageFromFile and NSLinkImage to  
load and link the bundles. The flags passed to NSLinkImage are *hard  
coded* at compile time, so either all bundles are loaded globally or  
none are. Contrast this to the case where dlopen is used, and the  
dlopenflags are looked up at library load time via
'dlopenflags = PyThreadState_GET()->interp->dlopenflags':
http://svn.python.org/view/python/trunk/Python/dynload_shlib.c? 
rev=41910&view=auto

Now, perhaps it is really necessary that on the Mac, the flags be  
hard coded and that no run-time selection will work. If so, then I'll  
have to implement my own dlopen() based solution as I described earlier.

Otherwise, Mac Python's bundle loading code should have a way to  
expose the global symbol loading mechanism (necessary for C++ RTTI  
with templates) to user code. There are two ways that I imagine this  
could be added:

(1) Provide run-time support for setting the NSLinkImage flags. This  
could be as simple as consulting the dlopenflags set in the sys  
module (and just noting in the documentation that on Macs, the  
'dlopenflags' are really 'NSLinkImage flags'). Or there could be some  
mac-specific module where the NSLinkImage flags are defined, and  
where there is a SetNSLinkImageFlags command to call.

(2) Move to using dlopen() to load the libraries instead of  
NSLinkImage. This might need some minor changes to dynload_shlib.c,  
and it would need the configure script to be smart enough to choose  
dynload_next.c on 10.2 and below.

Neither of these seem too hard. I could try to work up a patch for  
one of these two fixes, if people agree that one or the other is a  
better approach. (I might need some guidance, but I would like to  
help get this issue resolved.)


>> Basically, this looks like it would be replicating how Python loads
>> modules on Linux. It's too bad that's how it would have to be done,
>> unless there's a better way that I can't think of right now. Anyone
>> have any thoughts/suggestions/guidance?
>
> It'd probably factor it such that the shared stuff (C++ classes) is  
> in shared libraries and the Python C API stuff is in the extensions.

Agreed. Unfortunately, parts of the first version of these libraries  
are SWIG-generated, and I'm not sure I can teach SWIG to do that  
refactoring. Would the stub/proxy approach I suggested work at all,  
or will I have to figure out a way to get swig to emit the C++ stuff  
in a different place?

Thanks for your suggestions and clarifications!

Zach