[Pythonmac-SIG] Extensions for static versus framework MachoPython

bill fancher fancher@pacbell.net
Fri, 05 Jul 2002 13:37:16 -0700


On Thursday, July 4, 2002, at 03:12  PM, Jack Jansen wrote:
<snip>
> If you build an extension module for a framework Python it will not use 
> the -bundle_loader option, but in stead link with "-framework Python". If 
> you import this module into a framework Python it'll notice that the 
> Python.framework is incore already, link against that and everything is 
> fine.
>
> If you try to load this module into a static Python, however, it will 
> simply pull in a fresh copy of Python.framework. Even this shouldn't be a 
> problem, because once the module tries to initialize itself it'll call 
> PyModule_Initialize4(), which should notice that the module lives in a 
> fresh, uninitialized interpreter and it should complain bitterly.
>
> However: it doesn't complain. Which means that it must somehow call 
> PyModule_Initialize4() from the already loaded static python (or, at 
> least share its datastructures). But it *does* load Python.framework 
> (checked by turning on DYLD debugging). This makes me very uneasy, is it 
> may be using some stuff from the static Python and some (uninitialized) 
> stuff from the framework it pulled in. That would definitely spell 
> disaster, especially if the two Pythons have been compiled with different 
> options (GC or debugging, for instance).
>
> I'm not sure why it doesn't complain, it may have something to do with 
> Python.framework being linked with -flat_namespace, which it needs 
> because it needs access to _environ (which frameworks or dynamic 
> libraries don't get, sigh, only main programs).

It doesn't complain because the *modules* are linked with -flat_namespace,
  so they know what libraries they "need" but not which library a 
particular symbol comes from. Apparently, dyld finds what it's looking for 
in the statically linked library before getting to the shared lib's 
namespace. I put a breakpoint on the static lib's PyModule_Initialize4 and 
that is in fact what gets called. Module symbols are in a separate 
namespaces and the dylib's symbols are thus segregated, otherwise dyld 
would complain about duplicates.

As for _environ, there's a way around that. From Mach-O Runtime 
Architecture, p.24:

"Shared libraries that need to reference symbols defined in the program 
main
executable, such as the environ variable, must load the symbol dynamically
using a function that does not require a library reference, such as
NSLookupAndBindSymbol (page 83). (In the specific case of environ, you can 
use the
function _NSGetEnviron, embedded as part of the C runtime)."

> I tried specifying Python.framework as the -bundle_loader, but that doesn'
> t work.
>
> I'm out of good ideas now.
>
> And it may be that I'm seeing a problem where there really isn't any: if 
> 100% of the symbols are satisfied from the static Python then everything 
> should be fine. Major differences between versions will be caught by the 
> PY_API_VERSION mechanism (which is the general mechanism that stops you 
> from loading an extension module into a Python that is too old or too new)
> , so it's really only "little differences".

Assuming the PYTHON_API_VERSION	varies with what Apple calls major 
versions (as it does in Python 2.2 vs. 2.3), then it should be safe. I'd 
rather have the module not get linked against the library, but use 
-flat_namespace -undefined suppress. That keeps dyld from looking for a 
library that might not be needed and might not even be there. That patch I 
sent you off list that allows one to build Python as a standalone dynamic 
shared library uses that approach, and doesn't have the problem discussed 
here. It also links the main library without -two_level_namespace, which, 
Apple claims, makes it load faster and also provides protection against 
namespace collisions (though that's probably not much of a practical 
problem.)
>
> Should I stop worrying and just think of this as an advantage ("If you 
> build your extension module with a framework Python you can also use it 
> with a static Python")?

I wouldn't worry too much about it for the nonce, but in the long run, I 
think we'd be better off with a -two_level_namespace shared library in 
/usr/local/lib and modules that are a bit more modular.

FWIW,

--
bill