[Distutils] Does anyone understand what's going on with libpython on Linux?

[Nathaniel Smith]

On Sun, Feb 7, 2016 at 7:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 8 February 2016 at 08:33, Matthew Brett <matthew.brett at gmail.com> wrote:
>> It seems reasonable to build to the same compatibility level as most
>> Debian packaged modules.
>
> Right, one of the key things to remember with manylinux1 is that it
> is, *quite deliberately*, only an 80% solution to the cross-distro
> lack-of-ABI-compatibility problem: we want to solve the simple cases
> now, and then move on to figuring out how to solve the more complex
> cases later (and applications that embed their own Python runtimes are
> a whole world of pain, in more ways than one).
>
> Since we know that extensions built against a statically linked
> CPython will run correctly against a dynamically linked one, then it
> probably makes sense to go down that path for the manylinux1 reference
> build environment.
>
> However, there's one particular test case we should investigate before
> committing to that path: loading manylinux1 wheels built against a
> statically linked CPython into a system httpd environment running the
> system mod_wsgi. If I've understood the problem description correctly,
> that *should* work, but if it doesn't, then it would represent a
> significant compatibility concern.

That's actually a great example of the case that "ought" to fail,
because first you have a host program (apache2) that uses dlopen() to
load in the CPython interpreter (implicitly, by dlopen'ing
mod_wsgi.so, which is linked to libpython), and then the CPython
interpreter turns around and tries to use dlopen() to load extension
modules. Normally, this should work if and only if the extension
modules are themselves linked against libpythonX.Y.so, --enable-shared
/ Fedora style.

However, this is not Apache's first rodeo:

  revision 1.12
  date: 1998/07/10 18:29:50; author: rasmus; state: Exp; lines: +2 -2
  Set the RTLD_GLOBAL dlopen mode parameter to allow
  dynamically loaded modules to load their own modules dynamically.
  This improves mod_perl and mod_php3 when these modules are
  loaded dynamically into Apache.

(Confirmation that this is still true:
https://apr.apache.org/docs/apr/2.0/group__apr__dso.html#gaedc8609c2bb76e5c43f2df2281a9d8b6
-- also I ran mod_wsgi-express under LD_DEBUG=scopes and that also
showed libpython2.7.so.1 getting added to the global scope.)

Using RTLD_GLOBAL like this is the "wrong" thing -- it means that
different Apache mods all get loaded into the same global namespace,
and that means that they can have colliding symbols and step on each
other's feet. E.g., this is the sole and entire reason why you can't
load a python2 version of mod_wsgi and a python3 version of mod_wsgi
into the same apache. But OTOH it means that Python extension modules
will work even if they don't explicitly link against libpython.

I also managed to track down two other programs that also follow this
load-a-plugin-that-embeds-python pattern -- LibreOffice and xchat --
and they also both seem to use RTLD_GLOBAL. So even if it's the
"wrong" thing, extension modules that don't explicitly link to
libpython do seem to work reliably here in the world we have, and
they're more compatible with Debian/Ubuntu and their massive market
share, so... the whole manylinux1 strategy is nothing if not
relentlessly pragmatic. I guess we should forbid linking to libpython
in the PEP.

[Note: I did not actually try loading any such modules into mod_wsgi
or these other programs, because I have no idea how to use mod_wsgi or
these other programs :-). The LD_DEBUG output is fairly definitive,
but it wouldn't hurt for someone to double-check if they feel
inspired...]

-n

-- 
Nathaniel J. Smith -- https://vorpus.org