[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 25.04.19 13:14, Victor Stinner wrote:
> Le jeu. 25 avr. 2019 à 09:34, Matthias Klose <doko at ubuntu.com> a écrit :
>> there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;)  So
>> depending on the package maintainer, you already have that available, but it is
>> extra maintenance cost.  Simplifying that would be a good idea.
> 
> Fedora provides "debuginfo" for all binarry packages (like numpy), but
> that's different from a debug build. Usually, C code of packages are
> optimized by gcc -O2 or even gcc -O3 which makes the debugging
> experience very painful: gdb fails to read C local variables and just
> say "<optimized out>". To debug internals, you want a debug build
> compiled by gcc -Og or (better IMHO) gcc -O0.
> 
> If you want to inspect *Python* internals but you don't need to
> inspect numpy internals, being able to run a release numpy on a debug
> Python is convenient.

yes, the Debian/Ubuntu packages contain both the debug build, and the debug info
for they normal build, e.g.

/usr/lib/debug/.build-id/3a/8ea2ab6ee85ff68879a48170966873eb8da781.debug
/usr/lib/debug/.build-id/78/5ff95f8d2d06c5990ae4e03cdff99452ca0de9.debug
/usr/lib/debug/.build-id/92/e008cffa3f09106214bfb6b80b7fd02ceab74f.debug
/usr/lib/debug/.build-id/ab/33160518c41acc0488bbc3af878995ef74e07f.debug
/usr/lib/debug/.build-id/bd/65896626a4c6566e96ad008362922cf6a39cd6.debug
/usr/lib/debug/.build-id/f1/e83b14a76dd9564e962dcdd2f70202e6fdb2b1.debug
/usr/lib/debug/.build-id/ff/5eab5fd2d14f4bfa6a1ef2300358efdc7dd800.debug
/usr/lib/python3/dist-packages/lxml/_elementpath.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/builder.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/etree.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/html/clean.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/html/diff.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/objectify.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/sax.cpython-37dm-x86_64-linux-gnu.so

> With an additional change on SOABI (I will open a separated issue for
> that), my PR 12946 (no longer link C extensions to libpython) allows
> to load lxml built in release mode in a Python built in debug mode!
> That's *very* useful for debugging. I show an example of the gdb
> experience with a release Python vs debug Python:
> 
> https://bugs.python.org/issue21536#msg340821
> 
> With a release Python, the basic function "py-bt" works as expected,
> but inspecting Python internals doesn't work: most local C variables
> are "optimized out" :-(
> 
> With a debug Python, the debugging experience is *much* better: it's
> possible to inspect Python internals!
> 
> 
>> However I still
>> would like to be able to have "debug" and "non-debug" builds co-installable at
>> the same time.
> 
> One option is to keep "d" flag in the SOABI so C extensions get a
> different SO filename (no change compared to Python 3.7):
> "NAME.cpython-38-x86_64-linux-gnu.so" for release vs
> "NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d"
> suffix ("cpython-38" vs "cpython-38d").
> 
> *But* modify importlib when Python is compiled in debug mode to look
> also to SO without the "d" suffix: first try load
> "NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there
> is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release:
> no suffix). Since the ABI is now compatible in Python 3.8, it should
> "just work" :-)
> 
> From a Linux packager perspective, nothing changes ;-) We can still
> provide "apt install python3-numpy-dbg" (debug) which can is
> co-installable with "apt install python3-numpy" (release).
> 
> The benefit is that it will be possible to load C extensions which are
> only available in the release flavor with a debug Python ;-)

yes, that sounds good.  Are there use cases where you only want to load *some*
debug extensions, even if more are installed?