On Thu, Sep 9, 2021 at 4:59 PM Marc-Andre Lemburg <mal@egenix.com> wrote:
The problem with C API changes is that extensions need to be adapted. If that has to be done for every single Python release, this causes a lot of churn.
Minor C API incompatible changes have already been done in Python 3.7, 3.8 and 3.9. I'm keeping an eye on the Python bug tracker, capi-sig and python-dev lists, and I only saw very complains. Usually, it's just one project, and I help them to fix their issue.
I reverted one or two C API changes (like Py_TYPE, reverted in 3.10) which caused too many issues. I'm always volunteer to help projects to help them to adapt to C API changes.
Only a *minority* of C extensions are impacted by these C API changes.
So far, only a few C extensions had to be adapted, and this work has been done. But maybe you are exaggerating the number of impacted projects?
It may also cause issues for packages to support multiple Python versions (e.g. PyData packages typically support 3.7 - 3.9). Victor's package can help with the latter, but it doesn't solve the former issue.
All C extensions that I saw support a large number of Python versions, usually Python 3.6-3.9, up to 2.7-3.11.
I'm not sure of which technical issue you're talking about. If you want to get new C API functions on old Python versions, use pythoncapi_compat.h that I talked about.
Cython, numpy and pybind11 chose to have their own compatibility layer. They are doing that for many years, it's not something else. Obviously, if you use Cython or pybind11, you don't have to handle differences between the different Python versions.
If we want to officially push for Cython as a way to provide a layer between the Python C API and Python C extensions, the PSF needs to coordinate some serious and sustained funding for that project. Otherwise, we end up with a single point of failure in our extension stack, which isn't even under SC control.
Cython is already recommended for many years directly in the official C API documentation, aside with "cffi, SWIG and Numba": https://docs.python.org/dev/extending/index.html?highlight=cython#recommende...
Maybe it should be advertized in more C API pages.
I agree that Cython lacks funding, but that's a different topic, no?
At least, I'm trying to prepare Cython for incompatible C API changes *before* pushing them into Python. Contributions is a help to help Cython. My team at Red Hat helped to fix multiple compatibility issues in Cython caused by changes in new Python versions.
Victor
Night gathers, and now my watch begins. It shall not end until my death.