[Distutils] Does anyone understand what's going on with libpython on Linux?

[Matthew Brett]

On Sun, Feb 7, 2016 at 4:38 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 7 Feb 2016 00:25:57 -0800
> "Robert T. McGibbon" <rmcgibbo at gmail.com> wrote:
>> > What are Debian/Ubuntu doing in distutils so that extensions don't link
>> to libpython by default?
>>
>> I don't know exactly, but one way to reproduce this is simply to build the
>> interpreter without `--enable-shared`.
>
> See https://bugs.python.org/issue21536. It would be nice if you could
> lobby for this issue to be resolved... (though that would only be for
> 3.6, presumably)

Just to unpack from that issue - and quoting a nice summary by you (Antoine):

"... the -l flag was added in #832799, for a rather complicated case
where the interpreter is linked with a library dlopened by an
embedding application (I suppose for some kind of plugin system)."

Following the link to https://bugs.python.org/issue832799 - the `-l`
flag (and therefore the dependency on libpython was added at Python
2.3 for the case where an executable A dlopens a library B.so . B.so
has an embedded Python interpreter and is linked to libpython.
However, when the embedded Python interpreter in B.so loads an
extension module mymodule.so , mymodule.so does not inherit a
namespace with the libpython symbols already loaded. See
https://bugs.python.org/msg18810 .

One option we have then is to remove all DT_NEEDED references to
libpython in manylinux wheels. We get instant compatibility for bare
Debian / Ubuntu Python installs, at the cost of causing some puzzling
crash for the case of: dlopened library with embedded Python
interpreter where the embedded Python interpreter imports a manylinux
wheel.

On the other hand, presumably this same crash will occur for nearly
all Debian-packaged Python extension modules (if it is true that they
do not specify a libpython dependency) - so it seems unlikely that
this is a common problem.

Cheers,

Matthew