New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support
Hi,
What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10?
Should this project be moved under the GitHub psf organization to have a more "future proof" URL?
I would like to promote this project to prepare C extensions maintainers for the following incompatible C API change (Py_TYPE) that I would like to push into Python 3.11: https://github.com/python/cpython/pull/26493 (Py_REFCNT was already converted to a static inline function in Python 3.10.)
I already made this Py_TYPE change in Python 3.10, but I had to revert it since it broke too many projects. Since last year, I upgraded most of these broken projects, I created the pythoncapi_compat project, and I succeeded to use upgrade_pythoncapi.py script and copy the pythoncapi_compat.h header file in multiple C extensions.
C extensions written with Cython are not affected. I already fixed Cython last year to emit code compatible with my incoming incompatible change. If it's not done yet, you only have to regenerate the C files using a recent Cython version.
--
I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 2.7 support: https://github.com/pythoncapi/pythoncapi_compat
To add Python 3.10 support to your C extension, go to its source directory and run:
/path/to/upgrade_pythoncapi.py .
It upgrades all C files (.c) in the current directory and subdirectories. For example, it replaces "op->ob_type" with "Py_TYPE(op)". It creates an ".old" copy of patched files.
Use the -o option to select operations:
* -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)". * -o all,-PyMem_MALLOC: run all operations, but don't replace PyMem_MALLOC(...) with PyMem_Malloc(...).
--
The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h header file that I wrote to provide recent Python 3.9-3.11 C functions on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). Functions are implemented as simple static inline functions to avoid requiring to link your extension to a dynamic library.
You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 functions in your projects without losing support for Python 2.7-3.9!
--
The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference).
--
This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions.
Article on the creation of the pythoncapi project: https://vstinner.github.io/pythoncapi_compat.html
The main drawback of this project is that it uses regular expressions to parse C code. Such "parser" can miss C code which has to be patched manually. In my experience, additional manual changes are really rare and take less than 1 minute on a very large C extension like numpy.
--
This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-)
* https://cython.org/ * https://hpy.readthedocs.io/
I hope that my script will facilitate migration of C extensions to HPy.
Victor
I like the idea but I don't understand the full impact yet. That's my job to assess.
For my research, I was also trying to have a similar compatibility hack for the changes I have had to do to allow GC integration.
This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-)
Is HPy ready yet given IIRC, there is no even first release yet? I stand to be corrected. I reckon it will still go through a period of incompatible changes for some time/months too.
On Wed, Jun 2, 2021 at 8:47 PM Victor Stinner vstinner@python.org wrote:
Hi,
What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10?
Should this project be moved under the GitHub psf organization to have a more "future proof" URL?
I would like to promote this project to prepare C extensions maintainers for the following incompatible C API change (Py_TYPE) that I would like to push into Python 3.11: https://github.com/python/cpython/pull/26493 (Py_REFCNT was already converted to a static inline function in Python 3.10.)
I already made this Py_TYPE change in Python 3.10, but I had to revert it since it broke too many projects. Since last year, I upgraded most of these broken projects, I created the pythoncapi_compat project, and I succeeded to use upgrade_pythoncapi.py script and copy the pythoncapi_compat.h header file in multiple C extensions.
C extensions written with Cython are not affected. I already fixed Cython last year to emit code compatible with my incoming incompatible change. If it's not done yet, you only have to regenerate the C files using a recent Cython version.
--
I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 2.7 support: https://github.com/pythoncapi/pythoncapi_compat
To add Python 3.10 support to your C extension, go to its source directory and run:
/path/to/upgrade_pythoncapi.py .
It upgrades all C files (.c) in the current directory and subdirectories. For example, it replaces "op->ob_type" with "Py_TYPE(op)". It creates an ".old" copy of patched files.
Use the -o option to select operations:
- -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)".
- -o all,-PyMem_MALLOC: run all operations, but don't replace
PyMem_MALLOC(...) with PyMem_Malloc(...).
--
The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h header file that I wrote to provide recent Python 3.9-3.11 C functions on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). Functions are implemented as simple static inline functions to avoid requiring to link your extension to a dynamic library.
You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 functions in your projects without losing support for Python 2.7-3.9!
--
The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference).
--
This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions.
Article on the creation of the pythoncapi project: https://vstinner.github.io/pythoncapi_compat.html
The main drawback of this project is that it uses regular expressions to parse C code. Such "parser" can miss C code which has to be patched manually. In my experience, additional manual changes are really rare and take less than 1 minute on a very large C extension like numpy.
--
This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-)
I hope that my script will facilitate migration of C extensions to HPy.
Victor
Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KHDZGCNO... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Jun 3, 2021 at 2:30 AM Joannah Nanjekye nanjekyejoannah@gmail.com wrote:
Is HPy ready yet given IIRC, there is no even first release yet? I stand to be corrected. I reckon it will still go through a period of incompatible changes for some time/months too.
HPy is a great project, but even if 90% of top 4000 PyPI extensions are converted to it, CPython will likely still want to support the long tail of old C extensions written directly with the Python C API ("Python.h"). My PEP 620 addresses the specific case of CPython which wants to maximize its backward compatibility. It's a trade-off between optimizations/major code rewrite and backward compatibility. Sadly, you have to break eggs to make an omelet (french expression) :-)
I suggest to use Cython or HPy to write *new* C extensions, since they don't depend (directly) on the C API. At a new Python version, just update Cython/HPy, and you're good. I hope that tomorrow, Cython/HPy will even support the limited C API and the stable ABI to not even have to rebuild wheel packages! Cython already has an (experimental?) option to restrict emitted C code to the limited C API.
--
I would say that HPy is not complete nor mature yet. IMO it's already worth it to start playing with it, try to convert your small C extensions, and send back your early feedback to HPy developers!
HPy links:
* Website: https://hpyproject.org/ * Blog: https://hpyproject.org/blog/ * Documentation: https://docs.hpyproject.org/ * GitHub: https://github.com/hpyproject/hpy/
Status:
* May 2021 (blog): https://hpyproject.org/blog/posts/2021/05/may-status-update/ * April 2021 (doc): https://docs.hpyproject.org/en/latest/overview.html#current-status-and-roadm... * March 2021 (blog): https://hpyproject.org/blog/posts/2021/03/hello-hpy/
There is a 0.0.1 release on GitHub (Git tag created last January): https://github.com/hpyproject/hpy/releases/tag/0.0.1
On PyPI, the 0.0.1 tarball doesn't contain any source: it was only created to reserve the name on PyPI.
Victor
On Thu, Jun 3, 2021 at 4:16 AM Victor Stinner vstinner@python.org wrote:
I would say that HPy is not complete nor mature yet. IMO it's already worth it to start playing with it, try to convert your small C extensions, and send back your early feedback to HPy developers!
This is my view of the state of HPy too. It's a really good time to try things out and give feedback. Once we start doing more official releases, changing fundamental things about the HPy API will become harder (or at least more work).
On 03. 06. 21 1:43, Victor Stinner wrote:
Hi,
What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10?
Should this project be moved under the GitHub psf organization to have a more "future proof" URL?
If you ask me, no.
I think we should aim to not break the C API as often. Rather than promoting a tool to solve problems, I would much rather not create the problems in the first place.
Sure, supporting HPy or per-interpreter GIL will be good when they're ready to use. But at this point, they are *experiments*. I do not think it's worth breaking the API for existing users, which get no benefit from the changes, to run these experiments.
I would like to promote this project to prepare C extensions maintainers for the following incompatible C API change (Py_TYPE) that I would like to push into Python 3.11: https://github.com/python/cpython/pull/26493 (Py_REFCNT was already converted to a static inline function in Python 3.10.)
I already made this Py_TYPE change in Python 3.10, but I had to revert it since it broke too many projects. Since last year, I upgraded most of these broken projects, I created the pythoncapi_compat project, and I succeeded to use upgrade_pythoncapi.py script and copy the pythoncapi_compat.h header file in multiple C extensions.
C extensions written with Cython are not affected. I already fixed Cython last year to emit code compatible with my incoming incompatible change. If it's not done yet, you only have to regenerate the C files using a recent Cython version.
--
I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 2.7 support: https://github.com/pythoncapi/pythoncapi_compat
To add Python 3.10 support to your C extension, go to its source directory and run:
/path/to/upgrade_pythoncapi.py .
It upgrades all C files (.c) in the current directory and subdirectories. For example, it replaces "op->ob_type" with "Py_TYPE(op)". It creates an ".old" copy of patched files.
Use the -o option to select operations:
- -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)".
- -o all,-PyMem_MALLOC: run all operations, but don't replace
PyMem_MALLOC(...) with PyMem_Malloc(...).
--
The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h header file that I wrote to provide recent Python 3.9-3.11 C functions on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). Functions are implemented as simple static inline functions to avoid requiring to link your extension to a dynamic library.
You can already use the new Py_NewRef() and Py_IsNone() Python 3.10 functions in your projects without losing support for Python 2.7-3.9!
--
The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference).
--
This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions.
Article on the creation of the pythoncapi project: https://vstinner.github.io/pythoncapi_compat.html
The main drawback of this project is that it uses regular expressions to parse C code. Such "parser" can miss C code which has to be patched manually. In my experience, additional manual changes are really rare and take less than 1 minute on a very large C extension like numpy. > --
This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-)
I hope that my script will facilitate migration of C extensions to HPy.
Victor
participants (4)
-
Joannah Nanjekye -
Petr Viktorin -
Simon Cross -
Victor Stinner