[Python-Dev] Problems with Python's default dlopen flags

[Martin v. Loewis]

"David Abrahams" <david.abrahams@rcn.com> writes:

> Did you misread my suggestion? I didn't say that RTLD_GLOBAL should be
> the default way to load an extension module, only that there should be a
> way for the module itself to determine how it's loaded.

I dismissed your suggestion as being too complex. There are a number
of questions involved which I cannot answer that may effect usability
of this approach; most of them have to do with dlclosing the library:

1. If the extension module is C++ code, dlopening the module will run
   constructors for global objects. dlclosing it will run destructors.
   So the dlopen/dlclose/dlopen cycle might have side effects; that
   might be confusing.
2. Again, with C++ code, on Linux, with gcc 2.95.x, a block-local
   static object will register its destructor with atexit(3). When
   the module is dlclosed, the code to be called at exit goes away;
   then the program crashes atexit. This is undesirable.
3. If the module is also used as a library that some other module
   links against, I'm not sure what the semantics of dlclose is.  I'd
   feel uncomfortable with such a feature if I don't know precisely
   how it acts in boundary cases.

> And in fact, I expect to ask users to do something special, like
> explicitly linking between extension modules

Indeed, that should also work fine - if users explicitly link
extension modules against each other, they should be able to share
symbols. The need for RTLD_GLOBAL only occurs when they want to share
symbols, but don't want to link the modules against each other.

> However, this is what I didn't expect: the lack of RTLD_GLOBAL flags
> interferes with the ability for ext1.so to catch C++ exceptions
> thrown by libboost_python.so!

That is surprising indeed, and hard to believe. Can you demonstrate
that in a small example?

> Are you suggesting that in order to do this, my users need to add
> yet another .so, a thin layer between Python and the guts of their
> extension modules?

Originally, that's what I suggested. I now think that, for symbol
sharing, linking the modules against each other should be sufficient.

> > Now, people still want to share symbols across modules. For that, you
> > can use CObjects: Export a CObject with an array of function pointers
> > in module A (e.g. as A.API), and import that C object in module B's
> > initialization code. See cStringIO and Numeric for examples.
> 
> Of course you realize that won't help with C++ exception tables...

Actually, I don't: I can't see what C++ exception tables have to do
with it - the exception regions are local in any case.

> ...which leads us back to the fact that the smarts are in the wrong
> place. The extension module writer knows that this particular
> extension needs to share symbols, and once the module is loaded it's
> too late.

The extension module writer can't possibly have this knowledge - to
know whether it is _safe_ to share symbols, you have to know the
complete set of extension modules in the application. If a single
module uses your proposed feature, it would export its symbols to all
other extensions - whether they want those symbols or not. Hence you
might still end up with a situation where you can't use two extensions
in a single application because of module clashes.

> So give setdlopenflags a "force" option which overrides the setting
> designated by the extension module. I realize it's messy (probably too
> messy). If I could think of some non-messy advice for my users that
> avoids a language change, I'd like that just as well.

For that, I'd need to understand the problem of your users first. I'm
unhappy to introduce work-arounds for incompletely-understood
problems.

Regards,
Martin