It became more and more common that a C API incompatible change introduced in Python breaks Cython and all Python projects using Cython (ex: numpy). Hopefully, usually only some projects using Cython are broken, not all of them.
Some of you may remind the PEP 590 (vectorcall) implementation which removed the tp_print member of PyTypeObject in Python 3.8. This change broke a large number of projects using Cython. Cython was updated, the member removal was reverted, etc. It took a few weeks to solve the issue. It was unpleasant to have to revert the change (add again tp_print). The tp_print member was removed again in Python 3.9.
IMO the PEP 570 (positional-only arguments) implementation went even worse when it added a parameter to PyCode_New(). Cython was modified and released with a change, and *then* the PyCode_New() change was reverted in Python (if I recall correctly)! It was very confusing. There were multiple communication issues between Python and Cython. The first Cython fix was incorrect, etc.
Last December, another Python change related to exceptions (bpo-45711) broke Cython on purpose. A commit message says "Add to what's new, because this change breaks things like Cython".
It's kind of annoying that the each time we have a long period of time (several weeks) when Cython is unusable and there is pressure on Cython get a fix and then release a new version.
At Red Hat, we are rebuilding Python frequently with an up to date Python 3.11: right now, numpy fails to build because of that.
I would prefer to introduce C API incompatible changes differently: first fix Cython, and *then* introduce the change.
- (1) Propose a Cython PR and get it merged - (2) Wait until a new Cython version is released - (3) If possible, wait until numpy is released with regenerated Cython code - (4) Introduce the incompatible change in Python
Note: Fedora doesn't need (3) since we always regenerated Cython code in numpy.
None of these C API incompatible changes are wrong. For each change, there are good reasons to introduce them. I'm only proposing to change *how* we introduce them to avoid a long period of time when Cython and/or numpy are not usable on the development version of Python.
If you want to test if a C API change is going to break Cython or numpy, you can try my tool: https://github.com/vstinner/pythonci/
Usage: build a modified Python with your change, and then run pythonci with it. pythonci uses pinned version of all dependencies to try to be more reproducible. I update them manually infrequently (see pythonci/requirements.txt).
Right now, I propose to revert the incompatible change related to exceptions until Cython is prepared for it: https://bugs.python.org/issue45711#msg412264
Victor -- Night gathers, and now my watch begins. It shall not end until my death.