On Sun, 7 Feb 2016 00:25:57 -0800 "Robert T. McGibbon" wrote:

...

What are Debian/Ubuntu doing in distutils so that extensions don't link to libpython by default?

I don't know exactly, but one way to reproduce this is simply to build the interpreter without `--enable-shared`.

See https://bugs.python.org/issue21536. It would be nice if you could lobby for this issue to be resolved... (though that would only be for 3.6, presumably)

I don't know that their reasons are, but I presume that the Debian maintainers have a well-considered reason for this design.

Actually, shared library builds can be noticeably slower. I did measurements some time ago, and the results are: - shared builds are 5-10% slower on x86 - they can be up to 30% slower on some ARM CPUs! (this is on pystone which is a very crude benchmark, but in this case I think the pattern is more general, since any function call internal to Python is affected by the difference in code generation: shared library builds add an indirection overhead when resolving non-static symbols) Note btw. that Anaconda builds are also shared library builds. Regards Antoine.

On Sun, Feb 7, 2016 at 4:38 AM, Antoine Pitrou wrote:

On Sun, 7 Feb 2016 00:25:57 -0800 "Robert T. McGibbon" wrote:

...

...

What are Debian/Ubuntu doing in distutils so that extensions don't link to libpython by default?

I don't know exactly, but one way to reproduce this is simply to build the interpreter without `--enable-shared`.

See https://bugs.python.org/issue21536. It would be nice if you could lobby for this issue to be resolved... (though that would only be for 3.6, presumably)

Just to unpack from that issue - and quoting a nice summary by you (Antoine): "... the -l flag was added in #832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system)." Following the link to https://bugs.python.org/issue832799 - the `-l` flag (and therefore the dependency on libpython was added at Python 2.3 for the case where an executable A dlopens a library B.so . B.so has an embedded Python interpreter and is linked to libpython. However, when the embedded Python interpreter in B.so loads an extension module mymodule.so , mymodule.so does not inherit a namespace with the libpython symbols already loaded. See https://bugs.python.org/msg18810 . One option we have then is to remove all DT_NEEDED references to libpython in manylinux wheels. We get instant compatibility for bare Debian / Ubuntu Python installs, at the cost of causing some puzzling crash for the case of: dlopened library with embedded Python interpreter where the embedded Python interpreter imports a manylinux wheel. On the other hand, presumably this same crash will occur for nearly all Debian-packaged Python extension modules (if it is true that they do not specify a libpython dependency) - so it seems unlikely that this is a common problem. Cheers, Matthew

On Sun, Feb 7, 2016 at 2:19 PM, Robert T. McGibbon wrote:

...

One option we have then is to remove all DT_NEEDED references to libpython in manylinux wheels. We get instant compatibility for bare Debian / Ubuntu Python installs, at the cost of causing some puzzling crash for the case of: dlopened library with embedded Python interpreter where the embedded Python interpreter imports a manylinux wheel.

I don't think this is acceptable, since it's going to break some packages that depend on dlopen.

...

On the other hand, presumably this same crash will occur for nearly all Debian-packaged Python extension modules (if it is true that they do not specify a libpython dependency) - so it seems unlikely that this is a common problem.

I don't think so. Debian-packaged extensions that require libpython to exist (a minority of them to be sure, but ones that use complex shared library layouts) just declare a dependency on libpython. For example, python-pyside has a Depends on libpython2.7:

``` $ apt-cache depends python-pyside.qtcore python-pyside.qtcore Depends: libc6 Depends: libgcc1 Depends: libpyside1.2 Depends: libpython2.7 Depends: libqtcore4 Depends: libshiboken1.2v5 Depends: libstdc++6 Depends: python Depends: python Conflicts: python-pyside.qtcore:i386 ```

Sure - and this might be because the pyside packager was being especially careful about libpython, or it might be an accident - pyside is hard to build. On the other hand, it looks like almost all the common Debian packages don't declare this dependency - so almost all of the standard scientific Python stack and more would crash in this corner case: apt-cache depends python-numpy | grep libpython apt-cache depends python-scipy | grep libpython apt-cache depends python-yaml | grep libpython apt-cache depends python-regex | grep libpython apt-cache depends python-matplotlib | grep libpython It seems reasonable to build to the same compatibility level as most Debian packaged modules. Matthew

On Sun, Feb 7, 2016 at 7:48 PM, Nick Coghlan wrote:

On 8 February 2016 at 08:33, Matthew Brett wrote:

...

It seems reasonable to build to the same compatibility level as most Debian packaged modules.

Right, one of the key things to remember with manylinux1 is that it is, *quite deliberately*, only an 80% solution to the cross-distro lack-of-ABI-compatibility problem: we want to solve the simple cases now, and then move on to figuring out how to solve the more complex cases later (and applications that embed their own Python runtimes are a whole world of pain, in more ways than one).

Since we know that extensions built against a statically linked CPython will run correctly against a dynamically linked one, then it probably makes sense to go down that path for the manylinux1 reference build environment.

However, there's one particular test case we should investigate before committing to that path: loading manylinux1 wheels built against a statically linked CPython into a system httpd environment running the system mod_wsgi. If I've understood the problem description correctly, that *should* work, but if it doesn't, then it would represent a significant compatibility concern.

That's actually a great example of the case that "ought" to fail, because first you have a host program (apache2) that uses dlopen() to load in the CPython interpreter (implicitly, by dlopen'ing mod_wsgi.so, which is linked to libpython), and then the CPython interpreter turns around and tries to use dlopen() to load extension modules. Normally, this should work if and only if the extension modules are themselves linked against libpythonX.Y.so, --enable-shared / Fedora style. However, this is not Apache's first rodeo: revision 1.12 date: 1998/07/10 18:29:50; author: rasmus; state: Exp; lines: +2 -2 Set the RTLD_GLOBAL dlopen mode parameter to allow dynamically loaded modules to load their own modules dynamically. This improves mod_perl and mod_php3 when these modules are loaded dynamically into Apache. (Confirmation that this is still true: https://apr.apache.org/docs/apr/2.0/group__apr__dso.html#gaedc8609c2bb76e5c4... -- also I ran mod_wsgi-express under LD_DEBUG=scopes and that also showed libpython2.7.so.1 getting added to the global scope.) Using RTLD_GLOBAL like this is the "wrong" thing -- it means that different Apache mods all get loaded into the same global namespace, and that means that they can have colliding symbols and step on each other's feet. E.g., this is the sole and entire reason why you can't load a python2 version of mod_wsgi and a python3 version of mod_wsgi into the same apache. But OTOH it means that Python extension modules will work even if they don't explicitly link against libpython. I also managed to track down two other programs that also follow this load-a-plugin-that-embeds-python pattern -- LibreOffice and xchat -- and they also both seem to use RTLD_GLOBAL. So even if it's the "wrong" thing, extension modules that don't explicitly link to libpython do seem to work reliably here in the world we have, and they're more compatible with Debian/Ubuntu and their massive market share, so... the whole manylinux1 strategy is nothing if not relentlessly pragmatic. I guess we should forbid linking to libpython in the PEP. [Note: I did not actually try loading any such modules into mod_wsgi or these other programs, because I have no idea how to use mod_wsgi or these other programs :-). The LD_DEBUG output is fairly definitive, but it wouldn't hurt for someone to double-check if they feel inspired...] -n -- Nathaniel J. Smith -- https://vorpus.org