[Pythonmac-SIG] dlopenflags and sharing symbols across python extension modules

[Zachary Pincus]

>> [need to share symbols across dso boundaries snipped]
>> Any help or suggestions or thoughts would be greatly appreciated,
>
> Don't do that?
>
> Python uses two-level namespaces on OSX, and for a purpose: to avoid
> accidentally picking up a symbol from another extension. IIRC You  
> should
> build a shared library containing the shared code and link all  
> extensions
> to that.

Hmm, this is not exactly what I was hoping to hear.

About building python with a two-level namespace: isn't the best way  
to not pick up symbols from other extensions to not load the  
extensions with the RTLD_GLOBAL flag? Solving the problem with two- 
level namespaces seems a bit extreme... though perhaps there are  
other good reasons for this?

The python documentation for sys.setdlopenflags() specifically says  
that the sharing of symbols across modules can be enabled by the use  
of the RTLD_GLOBAL flag. It seems like Mac Python is at variance with  
this, and with the fact that modules can be set up on Linux Python to  
share symbols.

As to your first piece of advice: 'Don't do that,' unfortunately that  
is not an option. First, the modules need to support a plugin  
architecture where other modules created in the future will be able  
to pass these objects between them. Having to rebuild a mammoth  
central library each time someone wishes to add a plugin is not  
particularly attractive, and definitely not particularly dynamic.  
Second, the full set of core modules (~10 modules), if all linked  
together to form one central library, would be over 300 megabytes  
already, which totally chokes the loader.

The truth is that I really need to have multiple shared libraries,  
and they need to share the C++ typeinfo symbols between them. And  
*anyone* who wants to implement a plugin architecture with C++ Python  
extensions will need to do the same (or will need to not use C++  
dynamic_casts). Python on Linux supports this fine, and it would be a  
pity if Python on the Mac couldn't.

Does anyone have any additional insight as to (a) why the two-level  
namespace decision was made, (b) how it interacts with plugin  
architectures like I've described, and (c) if there's any way to load  
symbols globally even in the face of the two-level namespace?

Thanks,

Zach