[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Ivan Pozdeev]

On 24.04.2019 2:44, Victor Stinner wrote:
> Hi,
>
> Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
> default in debug build", but I lost myself in details, I forgot the
> main purpose of my proposal...
>
> Let me retry from scratch with a more explicit title: I would like to
> be able to run C extensions compiled in release mode on a Python
> compiled in debug mode ("pydebug").

This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at 
least, in Windows).

> The use case is to debug bugs in C
> extensions thanks to additional runtime checks of a Python debug
> build, and more generally get a better debugging experiences on
> Python. Even for pure Python, a debug build is useful (to get the
> Pyhon traceback in gdb using "py-bt" command).
That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a 
release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build 
extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions.
https://bugs.python.org/issue33637
>
> Currently, using a Python compiled in debug mode means to have to
> recompile C extensions in debug mode. Compile a C extension requires a
> C compiler, header files, pull dependencies, etc. It can be very
> complicated in practical (and pollute your system with all these
> additional dependencies). On Linux, it's already hard, but on Windows
> it can be even harder.
>
> Just one concrete example: no debug build of numpy is provided at
> https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
> manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
> :-)
The above paragraph is probably the reason ;-)
>
> --
>
> The first requirement for the use case is that a Python debug build
> supports the ABI of a release build. The current blocker issue is that
> the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2
> extra fields (_ob_prev and _ob_next) which change the offset of all
> attributes of all objects and makes the ABI completely incompatible. I
> propose to no longer imply Py_TRACE_REFS *by default* (but keep the
> code):
>
> https://bugs.python.org/issue36465
> https://github.com/python/cpython/pull/12615
>
> (Py_TRACE_REFS would be a different ABI.)
>
> The second issue is that library filenames are different for a debug
> build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build
> should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then
> also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not
> possible: a debug build contains many additional functions missing
> from a release build.
>
> For Windows, maybe we should provide a Python compiled in debug mode
> with the same C Runtime than a Python compiled in release mode.
> Otherwise, the debug C Runtime is causing another ABI issue.
>
> Maybe pip could be enhanced to support installing C extensions
> compiled in release mode when using a debug mode. But that's more for
> convenience, it's not really required, since it is easy to switch the
> Python runtime between release and debug build.
>
> Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
> structures. I know that the COUNT_ALLOCS define changes the ABI, but
> it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I
> propose to do the same for Py_TRACE_REFS ;-))
>
> Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory
> leaks, only sys.gettotalrefcount().
>
> --
>
> Python debug build has many benefit. If you ignore C extensions, the
> debug build is usually compiled with compiler optimization disabled
> which makes debugging in gdb a much better experience. If you never
> tried: on a release build, most (if not all) variables are "<optimized
> out>" and it's really painful to basic debug functions like displaying
> the current Python frame.
>
> Assertions are removed in release modes, whereas they can detect a
> wide range of bugs way earlier: integer overflow, buffer under- and
> overflow, exceptions ignored silently, etc. Nobody likes to see a bug
> for the first time in production. For example, I modified Python 3.8
> to now logs I/O errors when a file is closed implicitly, but only in
> debug or development mode. In release Python silently ignored EBADF
> error on such case, whereas it can lead to very nasty bugs causing
> Python to call abort() (which creates a coredump on Linux): see
> https://bugs.python.org/issue18748 ...
>
> DeprecationWarning and ResourceWarning are shown by default in debug mode :-)
>
> There are too many different additional checks done at runtime: I
> cannot list them all here.
>
> --
>
> Being able to switch between Python in release mode and Python in
> debug mode is a first step. My long term plan would be to better
> separate "Python" from its "runtime". CPython in release mode would be
> one runtime, CPython in debug mode would be another runtime, PyPy can
> seeen as another runtime, etc. The more general idea is: "compile your
> C extension once and use any Python runtime".
>
> https://pythoncapi.readthedocs.io/runtimes.html#runtimes
>
> If you opt-in for the stable ABI, you can already switch between
> runtimes of different Python versions (ex: Python 3.6 or Python 3.8).
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan