On 03. 06. 21 1:43, Victor Stinner wrote:
Hi,
What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10?
Should this project be moved under the GitHub psf organization to have a more "future proof" URL?
If you ask me, no. I think we should aim to not break the C API as often. Rather than promoting a tool to solve problems, I would much rather not create the problems in the first place. Sure, supporting HPy or per-interpreter GIL will be good when they're ready to use. But at this point, they are *experiments*. I do not think it's worth breaking the API for existing users, which get no benefit from the changes, to run these experiments.
I would like to promote this project to prepare C extensions maintainers for the following incompatible C API change (Py_TYPE) that I would like to push into Python 3.11: https://github.com/python/cpython/pull/26493 (Py_REFCNT was already converted to a static inline function in Python 3.10.)
I already made this Py_TYPE change in Python 3.10, but I had to revert it since it broke too many projects. Since last year, I upgraded most of these broken projects, I created the pythoncapi_compat project, and I succeeded to use upgrade_pythoncapi.py script and copy the pythoncapi_compat.h header file in multiple C extensions.
C extensions written with Cython are not affected. I already fixed Cython last year to emit code compatible with my incoming incompatible change. If it's not done yet, you only have to regenerate the C files using a recent Cython version.
--
I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 2.7 support: https://github.com/pythoncapi/pythoncapi_compat
To add Python 3.10 support to your C extension, go to its source directory and run:
/path/to/upgrade_pythoncapi.py .
It upgrades all C files (.c) in the current directory and subdirectories. For example, it replaces "op->ob_type" with "Py_TYPE(op)". It creates an ".old" copy of patched files.
Use the -o option to select operations:
* -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)". * -o all,-PyMem_MALLOC: run all operations, but don't replace PyMem_MALLOC(...) with PyMem_Malloc(...).
--
The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h header file that I wrote to provide recent Python 3.9-3.11 C functions on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). Functions are implemented as simple static inline functions to avoid requiring to link your extension to a dynamic library.
You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 functions in your projects without losing support for Python 2.7-3.9!
--
The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference).
--
This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions.
Article on the creation of the pythoncapi project: https://vstinner.github.io/pythoncapi_compat.html
The main drawback of this project is that it uses regular expressions to parse C code. Such "parser" can miss C code which has to be patched manually. In my experience, additional manual changes are really rare and take less than 1 minute on a very large C extension like numpy. > --
This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-)
* https://cython.org/ * https://hpy.readthedocs.io/
I hope that my script will facilitate migration of C extensions to HPy.
Victor