[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Victor Stinner]

Hi,

Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
default in debug build", but I lost myself in details, I forgot the
main purpose of my proposal...

Let me retry from scratch with a more explicit title: I would like to
be able to run C extensions compiled in release mode on a Python
compiled in debug mode ("pydebug"). The use case is to debug bugs in C
extensions thanks to additional runtime checks of a Python debug
build, and more generally get a better debugging experiences on
Python. Even for pure Python, a debug build is useful (to get the
Pyhon traceback in gdb using "py-bt" command).

Currently, using a Python compiled in debug mode means to have to
recompile C extensions in debug mode. Compile a C extension requires a
C compiler, header files, pull dependencies, etc. It can be very
complicated in practical (and pollute your system with all these
additional dependencies). On Linux, it's already hard, but on Windows
it can be even harder.

Just one concrete example: no debug build of numpy is provided at
https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
:-)

--

The first requirement for the use case is that a Python debug build
supports the ABI of a release build. The current blocker issue is that
the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2
extra fields (_ob_prev and _ob_next) which change the offset of all
attributes of all objects and makes the ABI completely incompatible. I
propose to no longer imply Py_TRACE_REFS *by default* (but keep the
code):

https://bugs.python.org/issue36465
https://github.com/python/cpython/pull/12615

(Py_TRACE_REFS would be a different ABI.)

The second issue is that library filenames are different for a debug
build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build
should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then
also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not
possible: a debug build contains many additional functions missing
from a release build.

For Windows, maybe we should provide a Python compiled in debug mode
with the same C Runtime than a Python compiled in release mode.
Otherwise, the debug C Runtime is causing another ABI issue.

Maybe pip could be enhanced to support installing C extensions
compiled in release mode when using a debug mode. But that's more for
convenience, it's not really required, since it is easy to switch the
Python runtime between release and debug build.

Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
structures. I know that the COUNT_ALLOCS define changes the ABI, but
it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I
propose to do the same for Py_TRACE_REFS ;-))

Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory
leaks, only sys.gettotalrefcount().

--

Python debug build has many benefit. If you ignore C extensions, the
debug build is usually compiled with compiler optimization disabled
which makes debugging in gdb a much better experience. If you never
tried: on a release build, most (if not all) variables are "<optimized
out>" and it's really painful to basic debug functions like displaying
the current Python frame.

Assertions are removed in release modes, whereas they can detect a
wide range of bugs way earlier: integer overflow, buffer under- and
overflow, exceptions ignored silently, etc. Nobody likes to see a bug
for the first time in production. For example, I modified Python 3.8
to now logs I/O errors when a file is closed implicitly, but only in
debug or development mode. In release Python silently ignored EBADF
error on such case, whereas it can lead to very nasty bugs causing
Python to call abort() (which creates a coredump on Linux): see
https://bugs.python.org/issue18748 ...

DeprecationWarning and ResourceWarning are shown by default in debug mode :-)

There are too many different additional checks done at runtime: I
cannot list them all here.

--

Being able to switch between Python in release mode and Python in
debug mode is a first step. My long term plan would be to better
separate "Python" from its "runtime". CPython in release mode would be
one runtime, CPython in debug mode would be another runtime, PyPy can
seeen as another runtime, etc. The more general idea is: "compile your
C extension once and use any Python runtime".

https://pythoncapi.readthedocs.io/runtimes.html#runtimes

If you opt-in for the stable ABI, you can already switch between
runtimes of different Python versions (ex: Python 3.6 or Python 3.8).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.