New script: add Python 3.10 support to your C extensions without losing Python 3.6 support
Hi,
I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 3.6 support: https://github.com/pythoncapi/pythoncapi_compat
For example, it replaces "op->ob_type" with "Py_TYPE(op)" and replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
It relies on the pythoncapi_compat.h header file that I wrote to implement Python 3.9 and Python 3.10 on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame().
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference). The PyFrame_GetBack() function was added to Python 3.9 and pythoncapi_compat.h provides it on older Python versions.
This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions.
This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-)
I hope that my script will facilitate migration of C extensions to HPy.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 2020-12-04, Victor Stinner wrote:
I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 3.6 support:
A compatibility shim like this is a good idea. Thanks for making it.
Would it make sense to have a recipe to add this to setup.py? That way, an extension could automatically work with a newly released version of CPython (e.g. by getting pythoncapi_compat at build time).
Hi Neil,
Update: I added Python 2.7, 3.4 and 3.5 support to pythoncapi_api.h ;-) I did that for Mercurial which still supports Python 2.7.
Le ven. 4 déc. 2020 à 23:56, Neil Schemenauer <nas-python@arctrix.com> a écrit :
Would it make sense to have a recipe to add this to setup.py? That way, an extension could automatically work with a newly released version of CPython (e.g. by getting pythoncapi_compat at build time).
Currently, the implementation has multiple limitations, and so I would suggest to not run it automatically when building a C extension. It might be possible if the tool is enhanced :-)
The current implementation is written in Python and uses simple regular expressions.
The tool replaces patterns which should not, especially the "PyThreadState_GetInterpreter" operation which replaces "tstate->interp" with "PyThreadState_GetInterpreter(tstate)". It replaces blindly the text pattern without checking the variable type.
My regular expressions are strict and miss some patterns. For example, "Py_TYPE(obj) = type" pattern is not replaced with "Py_SET_TYPE(obj, type)" because the tool expects a ";" terminator.
I started with a simple Python implementation with no external dependencies to ease its installation on any machine.
A better implementation should use Coccinelle. For example, the old 2to3c tool written by David Malcolm used Coccinelle: https://github.com/davidmalcolm/2to3c and https://dmalcolm.livejournal.com/3935.html
Well, I tested my tool on 10 C extensions and in practice, there had no issue. It's only when I tested the tool on Python itself that I got bad surprises, since the Python code base is large (500K lines of C code) and complex.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
participants (2)
-
Neil Schemenauer -
Victor Stinner