[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

Wed Apr 24 07:44:35 EDT 2019

On 24.04.2019 3:50, Ivan Pozdeev via Python-Dev wrote:
> On 24.04.2019 2:44, Victor Stinner wrote:
>> Hi,
>>
>> Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
>> default in debug build", but I lost myself in details, I forgot the
>> main purpose of my proposal...
>>
>> Let me retry from scratch with a more explicit title: I would like to
>> be able to run C extensions compiled in release mode on a Python
>> compiled in debug mode ("pydebug").
>
> This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at 
> least, in Windows).

To elaborate:

As per 
https://stackoverflow.com/questions/37541210/whats-the-difference-in-usage-between-shared-libraries-built-in-debug-and-relea/37580323#37580323 ,
Problems will occur if you have two modules that 1. use different versions or binary representations of a type and 2. exchange objects of 
that type

Now, I trust Victor has ensured no discrepancies in explicitly exchanged types.
But I'm not sure if Python and the extension still rely on implicitly sharing some C runtime entities. (In Py2, that would at least be 
descriptor table that MSVCRT maintains privately but Py3 doesn't rely on it AFAIK).

>> The use case is to debug bugs in C
>> extensions thanks to additional runtime checks of a Python debug
>> build, and more generally get a better debugging experiences on
>> Python. Even for pure Python, a debug build is useful (to get the
>> Pyhon traceback in gdb using "py-bt" command).
> That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a 
> release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build 
> extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions.
> https://bugs.python.org/issue33637
>>
>> Currently, using a Python compiled in debug mode means to have to
>> recompile C extensions in debug mode. Compile a C extension requires a
>> C compiler, header files, pull dependencies, etc. It can be very
>> complicated in practical (and pollute your system with all these
>> additional dependencies). On Linux, it's already hard, but on Windows
>> it can be even harder.
>>
>> Just one concrete example: no debug build of numpy is provided at
>> https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
>> manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
>> :-)
> The above paragraph is probably the reason ;-)
>>
>> -- 
>>
>> The first requirement for the use case is that a Python debug build
>> supports the ABI of a release build. The current blocker issue is that
>> the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2
>> extra fields (_ob_prev and _ob_next) which change the offset of all
>> attributes of all objects and makes the ABI completely incompatible. I
>> propose to no longer imply Py_TRACE_REFS *by default* (but keep the
>> code):
>>
>> https://bugs.python.org/issue36465
>> https://github.com/python/cpython/pull/12615
>>
>> (Py_TRACE_REFS would be a different ABI.)
>>
>> The second issue is that library filenames are different for a debug
>> build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build
>> should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then
>> also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not
>> possible: a debug build contains many additional functions missing
>> from a release build.
>>
>> For Windows, maybe we should provide a Python compiled in debug mode
>> with the same C Runtime than a Python compiled in release mode.
>> Otherwise, the debug C Runtime is causing another ABI issue.
>>
>> Maybe pip could be enhanced to support installing C extensions
>> compiled in release mode when using a debug mode. But that's more for
>> convenience, it's not really required, since it is easy to switch the
>> Python runtime between release and debug build.
>>
>> Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in
>> structures. I know that the COUNT_ALLOCS define changes the ABI, but
>> it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I
>> propose to do the same for Py_TRACE_REFS ;-))
>>
>> Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory
>> leaks, only sys.gettotalrefcount().
>>
>> -- 
>>
>> Python debug build has many benefit. If you ignore C extensions, the
>> debug build is usually compiled with compiler optimization disabled
>> which makes debugging in gdb a much better experience. If you never
>> tried: on a release build, most (if not all) variables are "<optimized
>> out>" and it's really painful to basic debug functions like displaying
>> the current Python frame.
>>
>> Assertions are removed in release modes, whereas they can detect a
>> wide range of bugs way earlier: integer overflow, buffer under- and
>> overflow, exceptions ignored silently, etc. Nobody likes to see a bug
>> for the first time in production. For example, I modified Python 3.8
>> to now logs I/O errors when a file is closed implicitly, but only in
>> debug or development mode. In release Python silently ignored EBADF
>> error on such case, whereas it can lead to very nasty bugs causing
>> Python to call abort() (which creates a coredump on Linux): see
>> https://bugs.python.org/issue18748 ...
>>
>> DeprecationWarning and ResourceWarning are shown by default in debug mode :-)
>>
>> There are too many different additional checks done at runtime: I
>> cannot list them all here.
>>
>> -- 
>>
>> Being able to switch between Python in release mode and Python in
>> debug mode is a first step. My long term plan would be to better
>> separate "Python" from its "runtime". CPython in release mode would be
>> one runtime, CPython in debug mode would be another runtime, PyPy can
>> seeen as another runtime, etc. The more general idea is: "compile your
>> C extension once and use any Python runtime".
>>
>> https://pythoncapi.readthedocs.io/runtimes.html#runtimes
>>
>> If you opt-in for the stable ABI, you can already switch between
>> runtimes of different Python versions (ex: Python 3.6 or Python 3.8).
>>
>> Victor
>> -- 
>> Night gathers, and now my watch begins. It shall not end until my death.
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
>
-- 
Regards,
Ivan