Use C extensions compiled in release mode on a Python compiled in debug mode
Hi, Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode ("pydebug"). The use case is to debug bugs in C extensions thanks to additional runtime checks of a Python debug build, and more generally get a better debugging experiences on Python. Even for pure Python, a debug build is useful (to get the Pyhon traceback in gdb using "py-bt" command). Currently, using a Python compiled in debug mode means to have to recompile C extensions in debug mode. Compile a C extension requires a C compiler, header files, pull dependencies, etc. It can be very complicated in practical (and pollute your system with all these additional dependencies). On Linux, it's already hard, but on Windows it can be even harder. Just one concrete example: no debug build of numpy is provided at https://pypi.org/project/numpy/ Good luck to build numpy in debug mode manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.) :-) -- The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code): https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615 (Py_TRACE_REFS would be a different ABI.) The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not possible: a debug build contains many additional functions missing from a release build. For Windows, maybe we should provide a Python compiled in debug mode with the same C Runtime than a Python compiled in release mode. Otherwise, the debug C Runtime is causing another ABI issue. Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build. Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in structures. I know that the COUNT_ALLOCS define changes the ABI, but it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I propose to do the same for Py_TRACE_REFS ;-)) Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory leaks, only sys.gettotalrefcount(). -- Python debug build has many benefit. If you ignore C extensions, the debug build is usually compiled with compiler optimization disabled which makes debugging in gdb a much better experience. If you never tried: on a release build, most (if not all) variables are "<optimized out>" and it's really painful to basic debug functions like displaying the current Python frame. Assertions are removed in release modes, whereas they can detect a wide range of bugs way earlier: integer overflow, buffer under- and overflow, exceptions ignored silently, etc. Nobody likes to see a bug for the first time in production. For example, I modified Python 3.8 to now logs I/O errors when a file is closed implicitly, but only in debug or development mode. In release Python silently ignored EBADF error on such case, whereas it can lead to very nasty bugs causing Python to call abort() (which creates a coredump on Linux): see https://bugs.python.org/issue18748 ... DeprecationWarning and ResourceWarning are shown by default in debug mode :-) There are too many different additional checks done at runtime: I cannot list them all here. -- Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime". CPython in release mode would be one runtime, CPython in debug mode would be another runtime, PyPy can seeen as another runtime, etc. The more general idea is: "compile your C extension once and use any Python runtime". https://pythoncapi.readthedocs.io/runtimes.html#runtimes If you opt-in for the stable ABI, you can already switch between runtimes of different Python versions (ex: Python 3.6 or Python 3.8). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Le mer. 24 avr. 2019 à 01:44, Victor Stinner <vstinner@redhat.com> a écrit :
The first requirement for the use case is that a Python debug build supports the ABI of a release build. (...) I propose to no longer imply Py_TRACE_REFS (...)
Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in structures. (...)
I tested manually: just by disabling Py_TRACE_REFS, the release ABI looks *fully* compatible with a debug build! I modified Python 3.7 to disable Py_TRACE_REFS and to omit "d" from SOABI when build in debug mode. I built Python in debug mode. I ran tests on numpy and lxml.etree: I can use .so libraries from /usr/lib64/python3.7/site-packages (compiled in release mode), it just works! I was very surprised of not getting any crash on such non trivial C extension, so I checked manually that I was running a debug build: yes, sys.gettotalrefcount is present :-) I also wanted to test an even more complex application: I installed gajim, a Jabber client written in Python 3 with PyGTK. It uses many C extensions. Running Gajim with my debug build is slower, well, that's not a surprise, but it works well! (no crash) About the SOABI, maybe we should only keep "d" when Py_TRACE_REFS is used, since technically, the ABI is same between release and debug mode without Py_TRACE_REFS. In that case, pip doesn't need to be modified ;-) If you also want to try, use: PYTHONPATH=/usr/lib64/python3.7/site-packages:/usr/lib/python3.7/site-packages ./python /usr/bin/gajim On a Python compiled with "./configure --with-pydebug && make" and the following patch: diff --git a/Include/object.h b/Include/object.h index bcf78afe6b..4c807981c4 100644 --- a/Include/object.h +++ b/Include/object.h @@ -51,13 +51,8 @@ A standard interface exists for objects that contain an array of items whose size is determined when the object is allocated. */ -/* Py_DEBUG implies Py_TRACE_REFS. */ -#if defined(Py_DEBUG) && !defined(Py_TRACE_REFS) -#define Py_TRACE_REFS -#endif - -/* Py_TRACE_REFS implies Py_REF_DEBUG. */ -#if defined(Py_TRACE_REFS) && !defined(Py_REF_DEBUG) +/* Py_DEBUG implies Py_REF_DEBUG. */ +#if defined(Py_DEBUG) && !defined(Py_REF_DEBUG) #define Py_REF_DEBUG #endif diff --git a/configure b/configure index 2db11e6e86..7271e9de40 100755 --- a/configure +++ b/configure @@ -6365,7 +6365,6 @@ $as_echo "#define Py_DEBUG 1" >>confdefs.h { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 $as_echo "yes" >&6; }; Py_DEBUG='true' - ABIFLAGS="${ABIFLAGS}d" else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 $as_echo "no" >&6; }; Py_DEBUG='false' fi diff --git a/configure.ac b/configure.ac index e5fb7e7b0b..fa4bb1944f 100644 --- a/configure.ac +++ b/configure.ac @@ -1246,7 +1246,6 @@ then [Define if you want to build an interpreter with many run-time checks.]) AC_MSG_RESULT(yes); Py_DEBUG='true' - ABIFLAGS="${ABIFLAGS}d" else AC_MSG_RESULT(no); Py_DEBUG='false' fi], [AC_MSG_RESULT(no)]) Victor
On 24.04.2019 2:44, Victor Stinner wrote:
Hi,
Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal...
Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode ("pydebug").
This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at least, in Windows).
The use case is to debug bugs in C extensions thanks to additional runtime checks of a Python debug build, and more generally get a better debugging experiences on Python. Even for pure Python, a debug build is useful (to get the Pyhon traceback in gdb using "py-bt" command). That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions. https://bugs.python.org/issue33637
Currently, using a Python compiled in debug mode means to have to recompile C extensions in debug mode. Compile a C extension requires a C compiler, header files, pull dependencies, etc. It can be very complicated in practical (and pollute your system with all these additional dependencies). On Linux, it's already hard, but on Windows it can be even harder.
Just one concrete example: no debug build of numpy is provided at https://pypi.org/project/numpy/ Good luck to build numpy in debug mode manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.) :-) The above paragraph is probably the reason ;-)
--
The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code):
https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615
(Py_TRACE_REFS would be a different ABI.)
The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not possible: a debug build contains many additional functions missing from a release build.
For Windows, maybe we should provide a Python compiled in debug mode with the same C Runtime than a Python compiled in release mode. Otherwise, the debug C Runtime is causing another ABI issue.
Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build.
Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in structures. I know that the COUNT_ALLOCS define changes the ABI, but it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I propose to do the same for Py_TRACE_REFS ;-))
Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory leaks, only sys.gettotalrefcount().
--
Python debug build has many benefit. If you ignore C extensions, the debug build is usually compiled with compiler optimization disabled which makes debugging in gdb a much better experience. If you never tried: on a release build, most (if not all) variables are "<optimized out>" and it's really painful to basic debug functions like displaying the current Python frame.
Assertions are removed in release modes, whereas they can detect a wide range of bugs way earlier: integer overflow, buffer under- and overflow, exceptions ignored silently, etc. Nobody likes to see a bug for the first time in production. For example, I modified Python 3.8 to now logs I/O errors when a file is closed implicitly, but only in debug or development mode. In release Python silently ignored EBADF error on such case, whereas it can lead to very nasty bugs causing Python to call abort() (which creates a coredump on Linux): see https://bugs.python.org/issue18748 ...
DeprecationWarning and ResourceWarning are shown by default in debug mode :-)
There are too many different additional checks done at runtime: I cannot list them all here.
--
Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime". CPython in release mode would be one runtime, CPython in debug mode would be another runtime, PyPy can seeen as another runtime, etc. The more general idea is: "compile your C extension once and use any Python runtime".
https://pythoncapi.readthedocs.io/runtimes.html#runtimes
If you opt-in for the stable ABI, you can already switch between runtimes of different Python versions (ex: Python 3.6 or Python 3.8).
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
On 24.04.2019 3:50, Ivan Pozdeev via Python-Dev wrote:
On 24.04.2019 2:44, Victor Stinner wrote:
Hi,
Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal...
Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode ("pydebug").
This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at least, in Windows).
To elaborate: As per https://stackoverflow.com/questions/37541210/whats-the-difference-in-usage-b... , Problems will occur if you have two modules that 1. use different versions or binary representations of a type and 2. exchange objects of that type Now, I trust Victor has ensured no discrepancies in explicitly exchanged types. But I'm not sure if Python and the extension still rely on implicitly sharing some C runtime entities. (In Py2, that would at least be descriptor table that MSVCRT maintains privately but Py3 doesn't rely on it AFAIK).
The use case is to debug bugs in C extensions thanks to additional runtime checks of a Python debug build, and more generally get a better debugging experiences on Python. Even for pure Python, a debug build is useful (to get the Pyhon traceback in gdb using "py-bt" command). That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions. https://bugs.python.org/issue33637
Currently, using a Python compiled in debug mode means to have to recompile C extensions in debug mode. Compile a C extension requires a C compiler, header files, pull dependencies, etc. It can be very complicated in practical (and pollute your system with all these additional dependencies). On Linux, it's already hard, but on Windows it can be even harder.
Just one concrete example: no debug build of numpy is provided at https://pypi.org/project/numpy/ Good luck to build numpy in debug mode manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.) :-) The above paragraph is probably the reason ;-)
--
The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code):
https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615
(Py_TRACE_REFS would be a different ABI.)
The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not possible: a debug build contains many additional functions missing from a release build.
For Windows, maybe we should provide a Python compiled in debug mode with the same C Runtime than a Python compiled in release mode. Otherwise, the debug C Runtime is causing another ABI issue.
Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build.
Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in structures. I know that the COUNT_ALLOCS define changes the ABI, but it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I propose to do the same for Py_TRACE_REFS ;-))
Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory leaks, only sys.gettotalrefcount().
--
Python debug build has many benefit. If you ignore C extensions, the debug build is usually compiled with compiler optimization disabled which makes debugging in gdb a much better experience. If you never tried: on a release build, most (if not all) variables are "<optimized out>" and it's really painful to basic debug functions like displaying the current Python frame.
Assertions are removed in release modes, whereas they can detect a wide range of bugs way earlier: integer overflow, buffer under- and overflow, exceptions ignored silently, etc. Nobody likes to see a bug for the first time in production. For example, I modified Python 3.8 to now logs I/O errors when a file is closed implicitly, but only in debug or development mode. In release Python silently ignored EBADF error on such case, whereas it can lead to very nasty bugs causing Python to call abort() (which creates a coredump on Linux): see https://bugs.python.org/issue18748 ...
DeprecationWarning and ResourceWarning are shown by default in debug mode :-)
There are too many different additional checks done at runtime: I cannot list them all here.
--
Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime". CPython in release mode would be one runtime, CPython in debug mode would be another runtime, PyPy can seeen as another runtime, etc. The more general idea is: "compile your C extension once and use any Python runtime".
https://pythoncapi.readthedocs.io/runtimes.html#runtimes
If you opt-in for the stable ABI, you can already switch between runtimes of different Python versions (ex: Python 3.6 or Python 3.8).
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
Le mer. 24 avr. 2019 à 01:44, Victor Stinner <vstinner@redhat.com> a écrit :
The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define (...):
https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615
I updated my PR: """ Release build and debug build are now ABI compatible: the Py_DEBUG define no longer implies Py_TRACE_REFS define which introduces the only ABI incompatibility. A new "./configure --with-trace-refs" build option is now required to get Py_TRACE_REFS define which adds sys.getobjects() function and PYTHONDUMPREFS environment variable. Changes: * Add ./configure --with-trace-refs * Py_DEBUG no longer implies Py_TRACE_REFS * The "d" flag of SOABI (sys.implementation.cache_tag) is now only added by --with-trace-refs. It is no longer added by --with-pydebug. """
Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode.
In fact, pip doesn't have to be modified. I "fixed" sys.implementation.cache_tag by removing "d" in debug mode instead ;-) By the way, the "m" ABI flag for pymalloc is outdated. I proposed the following change to simply remove it: https://bugs.python.org/issue36707 https://github.com/python/cpython/pull/12931/files With my PR 12931 and my PR 12615, the only remaining ABI flag which be "d" which would only be enabled by ./configure --with-trace-refs, whereas ./configure --with-pydebug has no more effect on SOABI (sys.implementation.cache_tag). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Jeroen Demeyer schrieb am 24.04.19 um 09:24:
On 2019-04-24 01:44, Victor Stinner wrote:
I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode
That seems like a very good idea. I would certainly use the debug mode while developing CPython or C extensions.
+1 Stefan
On Wed, 24 Apr 2019 01:44:17 +0200 Victor Stinner <vstinner@redhat.com> wrote:
The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code):
https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615
+1 from me.
The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m").
Sounds fair (but only on Unix, I guess).
Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build.
Not sure what you mean by "easy to switch the Python runtime". As soon as I want to use pip, I have to use a release build, right? Regards Antoine.
On 24.04.2019 17:03, Antoine Pitrou wrote:
On Wed, 24 Apr 2019 01:44:17 +0200 Victor Stinner <vstinner@redhat.com> wrote:
The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code):
https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615 +1 from me.
The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). Sounds fair (but only on Unix, I guess).
Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build. Not sure what you mean by "easy to switch the Python runtime". As soon as I want to use pip, I have to use a release build, right? No, pip works with a debug Python just as well (python.bat -m ensurepip) and installs modules to `<source tree>/site-packages` IIRC. But building extensions is broken in this case as per https://mail.python.org/pipermail/python-dev/2019-April/157180.html . Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded. When Python is built with --enable-shared, the python3.7 program is linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly linked to libpython3.7m as well: $ python3.7-config --ldflags ... -lpython3.7m ... Example with numpy: $ ldd /usr/lib64/python3.7/site-packages/numpy/core/umath.cpython-37m-x86_64-linux-gnu.so ... libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) ... When Python 3.7 is compiled in debug mode, libpython gets a "d" flag for debug: libpython3.7dm.so.1.0. I see 2 solutions: (1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython. (2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but *implicitly* to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output. The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems. The option (2) is simpler and likely more portable. Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded. I opened https://bugs.python.org/issue34814 to discuss how C extensions of the standard library should be linked but I closed it because we failed to find a consensus and the initial use case became a non-issue. It seems like we should reopen the discussion :-) Victor
On Wed, 24 Apr 2019 18:02:18 +0200 Victor Stinner <vstinner@redhat.com> wrote:
I see 2 solutions:
(1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython.
(2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but *implicitly* to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output.
The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems.
The option (2) is simpler and likely more portable.
Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded.
You can participate in https://bugs.python.org/issue21536 Regards Antoine.
You don't necessarily need rpath actually. The Linux loader has a bug/feature where once it has successfully loaded a library with a given soname, then any future requests for that soname within the same process will automatically return that same library, regardless of rpath settings etc. So as long as the main interpreter has loaded libpython.whatever from the correct directory, then extension modules will all get that same version. The rpath won't matter at all. It is annoying in general that on Linux, we have these two different ways to build extension modules. It definitely violates TOOWTDI :-). It would be nice at some point to get rid of one of them. Note that we can't get rid of the two different ways entirely though – on Windows, extension modules *must* link to libpython.dll, and on macOS, extension modules *can't* link to libpython.dylib. So the best we can hope for is to make Linux consistently do one of these, instead of supporting both. In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's mod_python, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, *but* it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so. In practice, extension modules have never consistently linked to libpython.so, so everybody who loads the interpreter as a plugin has already worked around this. Specifically, they use RTLD_GLOBAL to dump all the interpreter's symbols into the global namespace. This is why you can't have python2 and python3 mod_python at the same time in the same Apache. And since everyone is already working around this, linking to libpython.so currently has zero benefit... in fact manylinux wheels are actually forbidden to link to libpython.so, because this is the only way to get wheels that work on every interpreter. -n On Wed, Apr 24, 2019, 09:54 Victor Stinner <vstinner@redhat.com> wrote:
Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded.
When Python is built with --enable-shared, the python3.7 program is linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly linked to libpython3.7m as well:
$ python3.7-config --ldflags ... -lpython3.7m ...
Example with numpy:
$ ldd /usr/lib64/python3.7/site-packages/numpy/core/ umath.cpython-37m-x86_64-linux-gnu.so ... libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) ...
When Python 3.7 is compiled in debug mode, libpython gets a "d" flag for debug: libpython3.7dm.so.1.0.
I see 2 solutions:
(1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython.
(2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but *implicitly* to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output.
The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems.
The option (2) is simpler and likely more portable.
Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded.
I opened https://bugs.python.org/issue34814 to discuss how C extensions of the standard library should be linked but I closed it because we failed to find a consensus and the initial use case became a non-issue. It seems like we should reopen the discussion :-)
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
On Wed, Apr 24, 2019, 09:54 Victor Stinner <vstinner@redhat.com> wrote:
Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded.
When Python is built with --enable-shared, the python3.7 program is linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly linked to libpython3.7m as well:
$ python3.7-config --ldflags ... -lpython3.7m ...
Example with numpy:
$ ldd /usr/lib64/python3.7/site-packages/numpy/core/ umath.cpython-37m-x86_64-linux-gnu.so ... libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) ...
When Python 3.7 is compiled in debug mode, libpython gets a "d" flag for debug: libpython3.7dm.so.1.0.
I see 2 solutions:
(1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython.
(2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but *implicitly* to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output.
The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems.
The option (2) is simpler and likely more portable.
Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded.
I opened https://bugs.python.org/issue34814 to discuss how C extensions of the standard library should be linked but I closed it because we failed to find a consensus and the initial use case became a non-issue. It seems like we should reopen the discussion :-)
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
On 25.04.19 08:31, Nathaniel Smith wrote:
You don't necessarily need rpath actually. The Linux loader has a bug/feature where once it has successfully loaded a library with a given soname, then any future requests for that soname within the same process will automatically return that same library, regardless of rpath settings etc. So as long as the main interpreter has loaded libpython.whatever from the correct directory, then extension modules will all get that same version. The rpath won't matter at all.
It is annoying in general that on Linux, we have these two different ways to build extension modules. It definitely violates TOOWTDI :-). It would be nice at some point to get rid of one of them.
Note that we can't get rid of the two different ways entirely though – on Windows, extension modules *must* link to libpython.dll, and on macOS, extension modules *can't* link to libpython.dylib. So the best we can hope for is to make Linux consistently do one of these, instead of supporting both.
In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's mod_python, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, *but* it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so.
In practice, extension modules have never consistently linked to libpython.so, so everybody who loads the interpreter as a plugin has already worked around this. Specifically, they use RTLD_GLOBAL to dump all the interpreter's symbols into the global namespace. This is why you can't have python2 and python3 mod_python at the same time in the same Apache. And since everyone is already working around this, linking to libpython.so currently has zero benefit... in fact manylinux wheels are actually forbidden to link to libpython.so, because this is the only way to get wheels that work on every interpreter.
extensions in Debian/Ubuntu packages are not linked against libpython.so, but the main reason here is that sometimes you have to extensions built in transition periods like for 3.6 and 3.7. And this is also the default when not configuring with --enable-shared.
Hi, I'm now convinced that C extensions must *not* be linked to libpython on Unix. I wrote PR 12946: https://github.com/python/cpython/pull/12946 bpo-21536: C extensions are no longer linked to libpython On Unix, C extensions are no longer linked to libpython. It is now possible to load a C extension built using a shared library Python with a statically linked Python. When Python is embedded, libpython must not be loaded with RTLD_LOCAL, but RTDL_GLOBAL instead. Previously, using RTLD_LOCAL, it was already not possible to load C extensions which were not linked to libpython, like C extensions of the standard library built by the "*shared*" section of Modules/Setup. distutils, python-config and python-config.py have been modified. This PR allows to load a C extension built by a shared libpython with a statically linked Python: https://bugs.python.org/issue21536#msg340819 It also allows to load C extension built in release mode with a Python built in debug mode: https://bugs.python.org/issue21536#msg340821 Le jeu. 25 avr. 2019 à 08:31, Nathaniel Smith <njs@pobox.com> a écrit :
In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's mod_python, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, *but* it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so.
I'm aware of 2 special use cases of libpython: (A) Embed Python using RTLD_LOCAL: dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW) Example of issues describing this use case: * 2003: https://bugs.python.org/issue832799 * 2006: https://bugs.python.org/issue1429775 * 2018: https://bugs.python.org/issue34814 and https://bugzilla.redhat.com/show_bug.cgi?id=1585201 Python started to link C extensions to libpython in 2006 for this use case. (B) Load "libpython2" (Python 2) and "libpython3" (Python 3). I heard this idea... but I never saw anyone doing it in practice. I don't understand how it could work in a single address space. Linking C extensions to libpython is causing different issues: (1) C extension built by a shared library Python cannot be loaded with a statically linked Python: https://bugs.python.org/issue21536 (2) C extension built in release mode cannot be loaded with Python built in debug mode. That's the issue discussed in this thread ;-) (3) C extension built by Python 3.6 cannot be loaded in Python 3.7, even if it has been compiled using the stable ABI (Py_LIMITED_API). (4) C extensions of the standard library built by "*shared*" of Modules/Setup are *not* linked to libpython. For example, _struct.so on Fedora is not linked to libpython, whereas . If libpython is loaded with RTLD_LOCAL (use case A), import _struct fails. The use case (A) (RTLD_LOCAL) is trivial to fix: replace RTLD_LOCAL with RTLD_GLOBAL. The use case (B) (libpython2 + libpython3) is also easy to workaround: just use 2 separated processes. Python 2 will reach its end of life at the end of the year, I'm not sure that we should worry too much about this use case. The issue (1) (statically/shared) is a very practical issue. Fedora/RHEL uses libpython whereas Debian/Ubuntu uses statically linked Python. C extension compiled on Fedora/RHEL is linked to libpython and so cannot be loaded on Debian/Ubuntu (their Python doesn't have libpython!). That's why manylinux forbid link to link to libpython: be able to load C extensions on all Linux distributions. IMHO issues (1), (2), (3), (4) are more valuable to be fixed than supporting use cases (A) and (B). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 24.04.19 18:02, Victor Stinner wrote:
Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded.
When Python is built with --enable-shared, the python3.7 program is linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly linked to libpython3.7m as well:
$ python3.7-config --ldflags ... -lpython3.7m ...
Example with numpy:
$ ldd /usr/lib64/python3.7/site-packages/numpy/core/umath.cpython-37m-x86_64-linux-gnu.so ... libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) ...
When Python 3.7 is compiled in debug mode, libpython gets a "d" flag for debug: libpython3.7dm.so.1.0.
I see 2 solutions:
(1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython.
(2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but *implicitly* to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output.
The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems.
The option (2) is simpler and likely more portable.
Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded.
the purpose of python-config here is not clear. Whether it's intended to be used for linking extensions, or embedded interpreters. Currently you are using the same for both use cases.
Le jeu. 25 avr. 2019 à 09:30, Matthias Klose <doko@ubuntu.com> a écrit :
the purpose of python-config here is not clear. Whether it's intended to be used for linking extensions, or embedded interpreters. Currently you are using the same for both use cases.
My PR 12946 removes libpython from distutils, python-config and python-config.py: https://github.com/python/cpython/pull/12946 Do you mean that this change will break the build of applications embedding Python? If yes, what can done to fix that? Provide a different script to the specific case of embedded Python? Or add a new option to specify that you are embedding Python? In Python 3.7, the required linker flag is "-lpython3.7m". It's not trivial to guess the "m" suffix. FYI Python 3.8 it becames just "-lpython3.8": I removed the "m" suffix which was useless. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
I looked how fonforge gets compiler and linker flags to embed Python: it seems like to "pkg-config --libs python-2.7" which returns "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify Misc/python.pc as well... or not? :-) I'm not used to pkg-config. I don't know if it's common that C extensions are built using pkg-config. I guess that distutils is more commonly used to build C extensions. Victor Le jeu. 25 avr. 2019 à 12:53, Victor Stinner <vstinner@redhat.com> a écrit :
Le jeu. 25 avr. 2019 à 09:30, Matthias Klose <doko@ubuntu.com> a écrit :
the purpose of python-config here is not clear. Whether it's intended to be used for linking extensions, or embedded interpreters. Currently you are using the same for both use cases.
My PR 12946 removes libpython from distutils, python-config and python-config.py: https://github.com/python/cpython/pull/12946
Do you mean that this change will break the build of applications embedding Python? If yes, what can done to fix that?
Provide a different script to the specific case of embedded Python? Or add a new option to specify that you are embedding Python?
In Python 3.7, the required linker flag is "-lpython3.7m". It's not trivial to guess the "m" suffix. FYI Python 3.8 it becames just "-lpython3.8": I removed the "m" suffix which was useless.
Victor -- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
On 25.04.19 13:26, Victor Stinner wrote:
I looked how fonforge gets compiler and linker flags to embed Python: it seems like to "pkg-config --libs python-2.7" which returns "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify Misc/python.pc as well... or not? :-) I'm not used to pkg-config. I don't know if it's common that C extensions are built using pkg-config. I guess that distutils is more commonly used to build C extensions.
... except for all the software which is doing some embedding (e.g. vim), or is building some bindings as part of the upstream software. So yes, there is some stuff ... The tendency seems to deprecate your own config helper in favor of pkgconfig. However I'm not sure how this would do with the current MacOS python-config python script. If we want to differentiate between embedding and extensions, then we need two different module names, maybe keeping the current one for extensions, and having a new one for embedding. Not sure about python-config, if we want a new helper for embedding, or add new options for the existing script.
Victor
Le jeu. 25 avr. 2019 à 12:53, Victor Stinner <vstinner@redhat.com> a écrit :
Le jeu. 25 avr. 2019 à 09:30, Matthias Klose <doko@ubuntu.com> a écrit :
the purpose of python-config here is not clear. Whether it's intended to be used for linking extensions, or embedded interpreters. Currently you are using the same for both use cases.
My PR 12946 removes libpython from distutils, python-config and python-config.py: https://github.com/python/cpython/pull/12946
Do you mean that this change will break the build of applications embedding Python? If yes, what can done to fix that?
Provide a different script to the specific case of embedded Python? Or add a new option to specify that you are embedding Python?
In Python 3.7, the required linker flag is "-lpython3.7m". It's not trivial to guess the "m" suffix. FYI Python 3.8 it becames just "-lpython3.8": I removed the "m" suffix which was useless.
Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 2019-04-24, Victor Stinner wrote:
The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define
I think your change to make Py_TRACE_REFS as separate configure flag is fine. I've used the trace fields to debug occasionally but I don't use it often enough to need it enabled by Py_DEBUG.
Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime".
Regarding the Py_TRACE_REFS fields, I think we can't do them without breaking the ABI because of the following. For GC objects, they are always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we can allocate the extra space needed for the GC linked list. For non-GC objects, that's not the case. Extensions can allocate using malloc() directly or their own allocator and then pass that memory to be initialized as a PyObject. I think that's a poor design and I think we should try to make slow progress in fixing it. I think non-GC objects should also get allocated by a Python API. In that case, the Py_TRACE_REFS functionality could be implemented in a way that doesn't break the ABI. It also makes the CPython API more friendly for alternative Python runtimes like PyPy, etc. Note that this change would not prevent an extension from allocating memory with it's own allocator. It just means that memory can't hold a PyObject. The extension PyObject would need to have a pointer that points to this externally allocated memory. I can imagine there could be some situations when people really want a PyObject to reside in a certain memory location. E.g. maybe you have some kind of special shared memory area. In that case, I think we could have specialized APIs to create PyObjects using a specialized allocator. Those APIs would not be supported by some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs would not be used by most extensions. Regards, Neil
Hi Neil, On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer <nas-python@arctrix.com> wrote:
Regarding the Py_TRACE_REFS fields, I think we can't do them without breaking the ABI because of the following. For GC objects, they are always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we can allocate the extra space needed for the GC linked list. For non-GC objects, that's not the case. Extensions can allocate using malloc() directly or their own allocator and then pass that memory to be initialized as a PyObject.
I think that's a poor design and I think we should try to make slow progress in fixing it.
Such progress needs to start with the global static PyTypeObjects that all extensions define. This is going to be impossible to fix without requiring a big fix in of *all* of them. (Unless of course you mean to still allow them, but then Py_TRACE_REF can't be implemented in a way that doesn't break the ABI.) A bientôt, Armin. On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer <nas-python@arctrix.com> wrote:
On 2019-04-24, Victor Stinner wrote:
The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define
I think your change to make Py_TRACE_REFS as separate configure flag is fine. I've used the trace fields to debug occasionally but I don't use it often enough to need it enabled by Py_DEBUG.
Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime".
Regarding the Py_TRACE_REFS fields, I think we can't do them without breaking the ABI because of the following. For GC objects, they are always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we can allocate the extra space needed for the GC linked list. For non-GC objects, that's not the case. Extensions can allocate using malloc() directly or their own allocator and then pass that memory to be initialized as a PyObject.
I think that's a poor design and I think we should try to make slow progress in fixing it. I think non-GC objects should also get allocated by a Python API. In that case, the Py_TRACE_REFS functionality could be implemented in a way that doesn't break the ABI. It also makes the CPython API more friendly for alternative Python runtimes like PyPy, etc.
Note that this change would not prevent an extension from allocating memory with it's own allocator. It just means that memory can't hold a PyObject. The extension PyObject would need to have a pointer that points to this externally allocated memory.
I can imagine there could be some situations when people really want a PyObject to reside in a certain memory location. E.g. maybe you have some kind of special shared memory area. In that case, I think we could have specialized APIs to create PyObjects using a specialized allocator. Those APIs would not be supported by some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs would not be used by most extensions.
Regards,
Neil _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/armin.rigo%40gmail.com
On Sat, Apr 27, 2019, 04:27 Armin Rigo <armin.rigo@gmail.com> wrote:
Hi Neil,
On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer <nas-python@arctrix.com> wrote:
Regarding the Py_TRACE_REFS fields, I think we can't do them without breaking the ABI because of the following. For GC objects, they are always allocated by _PyObject_GC_New/_PyObject_GC_NewVar. So, we can allocate the extra space needed for the GC linked list. For non-GC objects, that's not the case. Extensions can allocate using malloc() directly or their own allocator and then pass that memory to be initialized as a PyObject.
I think that's a poor design and I think we should try to make slow progress in fixing it.
Such progress needs to start with the global static PyTypeObjects that all extensions define. This is going to be impossible to fix without requiring a big fix in of *all* of them. (Unless of course you mean to still allow them, but then Py_TRACE_REF can't be implemented in a way that doesn't break the ABI.)
For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce a list of all live objects on demand. If that's the goal, then static type objects aren't a huge deal. You can't add extra data into the type objects themselves, but since there's a fixed set of them and they're immortal, you can just build a static list of all of them in PyType_Ready. -n
On 2019-04-27, Nathaniel Smith wrote:
For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce a list of all live objects on demand. If that's the goal, then static type objects aren't a huge deal. You can't add extra data into the type objects themselves, but since there's a fixed set of them and they're immortal, you can just build a static list of all of them in PyType_Ready.
As far as I understand, we have a similar problem already for gc.get_objects() because those static type objects don't have a PyGC_Head. My 2-cent proposal for fixing things in the long term would be to introduce a function like PyType_Ready that returns a pointer to the new type. The argument to it would be what is the current static type structure. The function would copy things from the static type structure into a newly allocated type structure. We have a kind of solution already with PyType_FromSpec, etc. However, I think it is harder to convert existing extension module source code to use that API. We want to make it very easy for people to fix source code. If we can remove static types, that would allow us to kill off Py_TYPE(o)->tp_is_gc(o). I understand why that exists but I think it is quite an ugly detail of the current GC implementation. I wonder about the performance impact of it given current memory latencies. When we do a full GC run, we call PyObject_IS_GC() on many objects. I fear having to lookup and call tp_is_gc could be quite expensive. I've been playing with the idea of using memory bitmaps rather then the PyGC_Head. That idea seems to depend on removing static type objects. Initially I was thinking of it as reducing the memory overhead for GC types. Now I think the memory overhead doesn't matter too much but perhaps the bitmaps would be much faster due to memory latency. There is an interesting Youtube video that compares vector traversals vs linked list traversals in C++. Linked lists on modern machines are really terrible. Regards, Neil
You have my support is you work on removing static types :-) Here are my notes on the current C APIs to define a type: https://pythoncapi.readthedocs.io/type_object.html IMHO static types should go away in the long term. They are causing too many practical issues. Victor Le mar. 30 avr. 2019 à 02:01, Neil Schemenauer <nas-python@arctrix.com> a écrit :
On 2019-04-27, Nathaniel Smith wrote:
For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce a list of all live objects on demand. If that's the goal, then static type objects aren't a huge deal. You can't add extra data into the type objects themselves, but since there's a fixed set of them and they're immortal, you can just build a static list of all of them in PyType_Ready.
As far as I understand, we have a similar problem already for gc.get_objects() because those static type objects don't have a PyGC_Head. My 2-cent proposal for fixing things in the long term would be to introduce a function like PyType_Ready that returns a pointer to the new type. The argument to it would be what is the current static type structure. The function would copy things from the static type structure into a newly allocated type structure.
We have a kind of solution already with PyType_FromSpec, etc. However, I think it is harder to convert existing extension module source code to use that API. We want to make it very easy for people to fix source code.
If we can remove static types, that would allow us to kill off Py_TYPE(o)->tp_is_gc(o). I understand why that exists but I think it is quite an ugly detail of the current GC implementation. I wonder about the performance impact of it given current memory latencies. When we do a full GC run, we call PyObject_IS_GC() on many objects. I fear having to lookup and call tp_is_gc could be quite expensive.
I've been playing with the idea of using memory bitmaps rather then the PyGC_Head. That idea seems to depend on removing static type objects. Initially I was thinking of it as reducing the memory overhead for GC types. Now I think the memory overhead doesn't matter too much but perhaps the bitmaps would be much faster due to memory latency. There is an interesting Youtube video that compares vector traversals vs linked list traversals in C++. Linked lists on modern machines are really terrible.
Regards,
Neil
-- Night gathers, and now my watch begins. It shall not end until my death.
On Mon, Apr 29, 2019 at 5:01 PM Neil Schemenauer <nas-python@arctrix.com> wrote:
As far as I understand, we have a similar problem already for gc.get_objects() because those static type objects don't have a PyGC_Head. My 2-cent proposal for fixing things in the long term would be to introduce a function like PyType_Ready that returns a pointer to the new type. The argument to it would be what is the current static type structure. The function would copy things from the static type structure into a newly allocated type structure.
I doubt you'll be able to get rid of static types entirely, due to the usual issues with C API breakage. And I'm guessing that static types make up such a tiny fraction of the address space that merely tweaking the percent up or down won't affect performance. But your proposed new API would make it *way* easier to migrate existing code to the stable ABI. -n -- Nathaniel J. Smith -- https://vorpus.org
On 24.04.19 01:44, Victor Stinner wrote:
Hi,
Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal...
Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode ("pydebug"). The use case is to debug bugs in C extensions thanks to additional runtime checks of a Python debug build, and more generally get a better debugging experiences on Python. Even for pure Python, a debug build is useful (to get the Pyhon traceback in gdb using "py-bt" command).
Currently, using a Python compiled in debug mode means to have to recompile C extensions in debug mode. Compile a C extension requires a C compiler, header files, pull dependencies, etc. It can be very complicated in practical (and pollute your system with all these additional dependencies). On Linux, it's already hard, but on Windows it can be even harder.
Just one concrete example: no debug build of numpy is provided at https://pypi.org/project/numpy/ Good luck to build numpy in debug mode manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.) :-)
there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So depending on the package maintainer, you already have that available, but it is extra maintenance cost. Simplifying that would be a good idea. However I still would like to be able to have "debug" and "non-debug" builds co-installable at the same time.
Le jeu. 25 avr. 2019 à 09:34, Matthias Klose <doko@ubuntu.com> a écrit :
there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So depending on the package maintainer, you already have that available, but it is extra maintenance cost. Simplifying that would be a good idea.
Fedora provides "debuginfo" for all binarry packages (like numpy), but that's different from a debug build. Usually, C code of packages are optimized by gcc -O2 or even gcc -O3 which makes the debugging experience very painful: gdb fails to read C local variables and just say "<optimized out>". To debug internals, you want a debug build compiled by gcc -Og or (better IMHO) gcc -O0. If you want to inspect *Python* internals but you don't need to inspect numpy internals, being able to run a release numpy on a debug Python is convenient. With an additional change on SOABI (I will open a separated issue for that), my PR 12946 (no longer link C extensions to libpython) allows to load lxml built in release mode in a Python built in debug mode! That's *very* useful for debugging. I show an example of the gdb experience with a release Python vs debug Python: https://bugs.python.org/issue21536#msg340821 With a release Python, the basic function "py-bt" works as expected, but inspecting Python internals doesn't work: most local C variables are "optimized out" :-( With a debug Python, the debugging experience is *much* better: it's possible to inspect Python internals!
However I still would like to be able to have "debug" and "non-debug" builds co-installable at the same time.
From a Linux packager perspective, nothing changes ;-) We can still
One option is to keep "d" flag in the SOABI so C extensions get a different SO filename (no change compared to Python 3.7): "NAME.cpython-38-x86_64-linux-gnu.so" for release vs "NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d" suffix ("cpython-38" vs "cpython-38d"). *But* modify importlib when Python is compiled in debug mode to look also to SO without the "d" suffix: first try load "NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release: no suffix). Since the ABI is now compatible in Python 3.8, it should "just work" :-) provide "apt install python3-numpy-dbg" (debug) which can is co-installable with "apt install python3-numpy" (release). The benefit is that it will be possible to load C extensions which are only available in the release flavor with a debug Python ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 25.04.19 13:14, Victor Stinner wrote:
Le jeu. 25 avr. 2019 à 09:34, Matthias Klose <doko@ubuntu.com> a écrit :
there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So depending on the package maintainer, you already have that available, but it is extra maintenance cost. Simplifying that would be a good idea.
Fedora provides "debuginfo" for all binarry packages (like numpy), but that's different from a debug build. Usually, C code of packages are optimized by gcc -O2 or even gcc -O3 which makes the debugging experience very painful: gdb fails to read C local variables and just say "<optimized out>". To debug internals, you want a debug build compiled by gcc -Og or (better IMHO) gcc -O0.
If you want to inspect *Python* internals but you don't need to inspect numpy internals, being able to run a release numpy on a debug Python is convenient.
yes, the Debian/Ubuntu packages contain both the debug build, and the debug info for they normal build, e.g. /usr/lib/debug/.build-id/3a/8ea2ab6ee85ff68879a48170966873eb8da781.debug /usr/lib/debug/.build-id/78/5ff95f8d2d06c5990ae4e03cdff99452ca0de9.debug /usr/lib/debug/.build-id/92/e008cffa3f09106214bfb6b80b7fd02ceab74f.debug /usr/lib/debug/.build-id/ab/33160518c41acc0488bbc3af878995ef74e07f.debug /usr/lib/debug/.build-id/bd/65896626a4c6566e96ad008362922cf6a39cd6.debug /usr/lib/debug/.build-id/f1/e83b14a76dd9564e962dcdd2f70202e6fdb2b1.debug /usr/lib/debug/.build-id/ff/5eab5fd2d14f4bfa6a1ef2300358efdc7dd800.debug /usr/lib/python3/dist-packages/lxml/_elementpath.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/builder.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/etree.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/html/clean.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/html/diff.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/objectify.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/sax.cpython-37dm-x86_64-linux-gnu.so
With an additional change on SOABI (I will open a separated issue for that), my PR 12946 (no longer link C extensions to libpython) allows to load lxml built in release mode in a Python built in debug mode! That's *very* useful for debugging. I show an example of the gdb experience with a release Python vs debug Python:
https://bugs.python.org/issue21536#msg340821
With a release Python, the basic function "py-bt" works as expected, but inspecting Python internals doesn't work: most local C variables are "optimized out" :-(
With a debug Python, the debugging experience is *much* better: it's possible to inspect Python internals!
However I still would like to be able to have "debug" and "non-debug" builds co-installable at the same time.
One option is to keep "d" flag in the SOABI so C extensions get a different SO filename (no change compared to Python 3.7): "NAME.cpython-38-x86_64-linux-gnu.so" for release vs "NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d" suffix ("cpython-38" vs "cpython-38d").
*But* modify importlib when Python is compiled in debug mode to look also to SO without the "d" suffix: first try load "NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release: no suffix). Since the ABI is now compatible in Python 3.8, it should "just work" :-)
From a Linux packager perspective, nothing changes ;-) We can still provide "apt install python3-numpy-dbg" (debug) which can is co-installable with "apt install python3-numpy" (release).
The benefit is that it will be possible to load C extensions which are only available in the release flavor with a debug Python ;-)
yes, that sounds good. Are there use cases where you only want to load *some* debug extensions, even if more are installed?
Matthias Klose schrieb am 25.04.19 um 13:48:
Are there use cases where you only want to load *some* debug extensions, even if more are installed?
Not sure if there are _important_ use cases (that could justify certain design decisions), but I can certainly imagine using a non-debug (and therefore faster) Pandas or NumPy for preparing some data that I need to debug my own code. More generally, whenever I can avoid using a debug version of a *dependency* that I don't need to include in my debug analysis, it's probably a good idea to not use the debug version. Even given venvs and virtualisation techniques, it would probably be nice if users could install debug+nondebug versions of libraries once and then import the right one at need, rather than having to set up a new environment (while they're on a train in the middle of nowhere without fast access to PyPI). Stefan
FYI I pushed my 3 changes to implement my idea. It is now possible to install some extensions in release mode and some others in debug mode. Python in debug mode prefers debug extensions. I documented changes here: https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-... The library filename has to be different in debug mode, so it can be co-installable with release build of a C extension. Victor Le samedi 27 avril 2019, Stefan Behnel <stefan_ml@behnel.de> a écrit :
Matthias Klose schrieb am 25.04.19 um 13:48:
Are there use cases where you only want to load *some* debug extensions, even if more are installed?
Not sure if there are _important_ use cases (that could justify certain design decisions), but I can certainly imagine using a non-debug (and therefore faster) Pandas or NumPy for preparing some data that I need to debug my own code. More generally, whenever I can avoid using a debug version of a *dependency* that I don't need to include in my debug analysis, it's probably a good idea to not use the debug version.
Even given venvs and virtualisation techniques, it would probably be nice if users could install debug+nondebug versions of libraries once and then import the right one at need, rather than having to set up a new environment (while they're on a train in the middle of nowhere without fast access to PyPI).
Stefan
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
-- Night gathers, and now my watch begins. It shall not end until my death.
participants (9)
-
Antoine Pitrou -
Armin Rigo -
Ivan Pozdeev -
Jeroen Demeyer -
Matthias Klose -
Nathaniel Smith -
Neil Schemenauer -
Stefan Behnel -
Victor Stinner