[Python-Dev] Re: Cython and incompatible C API changes

On 2020-06-17 12:03, Victor Stinner wrote:

re: [Python-Dev] When can we remove wchar_t* cache from string?

Le mar. 16 juin 2020 à 21:19, Steve Dower a écrit :

...

On 16Jun2020 1641, Inada Naoki wrote:

...

* This change doesn't affect to pure Python packages. * Most of the rest uses Cython. Since I already report an issue to Cython, regenerating with new Cython release fixes them.

The precedent set in our last release with tp_print was that regenerating Cython releases was too much to ask.

Unless we're going to overrule that immediately, we should leave everything there and give users/developers a full release cycle with updated Cython version to make new releases without causing any breakage.

I already made changes in Python 3.10 which require again to regenerate C code generated by Cython: https://docs.python.org/dev/whatsnew/3.10.html#id2

Py_TYPE(), Py_REFCNT() and Py_SIZE() can no longer be used as l-value. These changes also broke numpy. I helped to fix Cython and numpy (and they are already fixed).

Those are not all the projects that were broken by the change -- they're just the most popular ones. Right?

You can expect further incompatible changes in the C API. For example, I would like to make the PyThreadState structure opaque, whereas Cython currently accesses directly to PyThreadState members.

There is an ongoing discussion about always requiring to run Cython when installing a C extension which uses Cython.

Do you have a link to that discussion?

Maybe we can find a way to use pre-generated C files for Python up to version N, but require to run Cython for Python newer than version N? It would prevent to require running Cython on stable Python versions, but help to upgrade to newer Python and also test the "next Python" (current master branch).

Note: if the Py_TYPE() & cie changes are causing too many issues, we can also reconsider to postpone/revert these changes. IMO it's important that we remain able to push incompatible changes to the C API, because there are multiple known flaws in the C API.

If PEP 387 (Backwards Compatibility Policy) is accepted, all the incompatible changes changes will require a two-year deprecation period. Right?