[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Victor Stinner]

Hi,

I'm now convinced that C extensions must *not* be linked to libpython on Unix.

I wrote PR 12946:
https://github.com/python/cpython/pull/12946

    bpo-21536: C extensions are no longer linked to libpython

    On Unix, C extensions are no longer linked to libpython.

    It is now possible to load a C extension built using a shared library
    Python with a statically linked Python.

    When Python is embedded, libpython must not be loaded with
    RTLD_LOCAL, but RTDL_GLOBAL instead. Previously, using RTLD_LOCAL, it
    was already not possible to load C extensions which were not linked
    to libpython, like C extensions of the standard library built by the
    "*shared*" section of Modules/Setup.

    distutils, python-config and python-config.py have been modified.

This PR allows to load a C extension built by a shared libpython with
a statically linked Python:
https://bugs.python.org/issue21536#msg340819

It also allows to load C extension built in release mode with a Python
built in debug mode:
https://bugs.python.org/issue21536#msg340821


Le jeu. 25 avr. 2019 à 08:31, Nathaniel Smith <njs at pobox.com> a écrit :
> In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's mod_python, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, *but* it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so.

I'm aware of 2 special use cases of libpython:


(A) Embed Python using RTLD_LOCAL: dlopen("libpython2.7.so.1.0",
RTLD_LOCAL | RTLD_NOW)

Example of issues describing this use case:

* 2003: https://bugs.python.org/issue832799
* 2006: https://bugs.python.org/issue1429775
* 2018: https://bugs.python.org/issue34814 and
https://bugzilla.redhat.com/show_bug.cgi?id=1585201

Python started to link C extensions to libpython in 2006 for this use case.


 (B) Load "libpython2" (Python 2) and "libpython3" (Python 3).

I heard this idea... but I never saw anyone doing it in practice. I
don't understand how it could work in a single address space.


Linking C extensions to libpython is causing different issues:

(1) C extension built by a shared library Python cannot be loaded with
a statically linked Python:
https://bugs.python.org/issue21536

(2) C extension built in release mode cannot be loaded with Python
built in debug mode. That's the issue discussed in this thread ;-)

(3) C extension built by Python 3.6 cannot be loaded in Python 3.7,
even if it has been compiled using the stable ABI (Py_LIMITED_API).

(4) C extensions of the standard library built by "*shared*" of
Modules/Setup are *not* linked to libpython. For example, _struct.so
on Fedora is not linked to libpython, whereas . If libpython is loaded
with RTLD_LOCAL (use case A), import _struct fails.


The use case (A) (RTLD_LOCAL) is trivial to fix: replace RTLD_LOCAL
with RTLD_GLOBAL.

The use case (B) (libpython2 + libpython3) is also easy to workaround:
just use 2 separated processes. Python 2 will reach its end of life at
the end of the year, I'm not sure that we should worry too much about
this use case.

The issue (1) (statically/shared) is a very practical issue.
Fedora/RHEL uses libpython whereas Debian/Ubuntu uses statically
linked Python. C extension compiled on Fedora/RHEL is linked to
libpython and so cannot be loaded on Debian/Ubuntu (their Python
doesn't have libpython!). That's why manylinux forbid link to link to
libpython: be able to load C extensions on all Linux distributions.

IMHO issues (1), (2), (3), (4) are more valuable to be fixed than
supporting use cases (A) and (B).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.