[Python-Dev] Problems with Python's default dlopen flags

[Martin v. Loewis]

"Mark Hammond" <mhammond@skippinet.com.au> writes:

> I don't understand the issues at all, other than my belief that Linux makes
> it harder than it needs to be.  Eg, in the recent PyXPCOM changes, I found
> that I needed to add the following line of code:
> 
> 	// *sob* - seems necessary to open the .so as RTLD_GLOBAL
> 	dlopen(PYTHON_SO,RTLD_NOW | RTLD_GLOBAL);
> 
> Where PYTHON_SO is the base-name of the Python shared library.  

I have a number of problems following your description, so let me try
to guess. In no released version of Python, libpython is build as a
shared library. However, I'll assume that you have build
libpythonxy.so as a shared library, and that this library provides the
symbols that normally the Python executable provides.

I further assume that the dlopen call is not in the code of the Python
interpreter itself, but somewhere inside Mozilla (I don't know what
PyXPCOM is).

In that case, using RTLD_GLOBAL is one option. It is quite safe to use
here, since it clutters the global namespace primarily with symbols
that all start with Py; those are unlikely to clash with other
symbols. Using RTLD_GLOBAL is needed since extension modules search
the Py symbols in the global namespace; they normally come from the
executable.

The other option is to explicitly link all extension modules against
your libpythonxy.so. If that is done, you can drop the RTLD_GLOBAL.
Of course, the standard build process won't link extension modules
with libpythonxy.so, since that library does not even exist in the
standard build process.

> As I said, I wont even pretend to understand the issues, but I do know that:
> * The line above made my code work.

Yes, see above.

> * Windows works perfectly in this situation, and always has.

This is because on Windows, every extension module is linked against
pythonxy.dll, to find the extension symbols.

> * It shouldn't be this hard.

I don't think it is hard. I feel that the situation on Windows is much
worse - to build an extension module, you need the import library, and
you need to link against it, and you thus hard-code the version of the
interpreter into the extension module, and you thus cannot use
extension modules across Python releases. There are trade-offs on both
sides.

Regards,
Martin