On Mon, Feb 1, 2021 at 4:17 PM M.-A. Lemburg <mal@egenix.com> wrote:
I have a feeling that this will drive away extension writers from Python.
The maintenance burden of having to constantly update the code base to accommodate for changes in the Python C API will be too much for them to handle (I know it's too much for me, which is why I stopped doing extension development).
https://docs.python.org/dev/extending/ starts with:
"This guide only covers the basic tools for creating extensions provided as part of this version of CPython. Third party tools like Cython, cffi, SWIG and Numba offer both simpler and more sophisticated approaches to creating C and C++ extensions for Python."
I suggest to stop writing new extensions directly using the C API. The problem is more about maintaining existing extensions written with the C API.
On top of that, those projects will have to keep compatibility with multiple Python versions, which makes things even more difficult.
Cython does that for you.
Other vendors introduce shims to handle breaking changes (e.g. MS on Windows) or even provide complete emulations (e.g. Apple for macOS).
Some people discussed writing a "cpyext"-like (similar to the PyPy module) for CPython, to provide backward compatibility. But so far, nobody came up with an implementation, or even a design.
I believe that's the only reasonable way to avoid another Python 2 to 3 15+ year transition process or losing the crowd to other languages. (*)
The Python 2 => Python 3 migration requires tons of changes at once and it took something like 5 years to have convenient solutions to have a single code base compatible with Python 2 and Python 3.
I'm proposing something different. From the start, providing concrete solutions to have a single code base compatible with the old and the new Python versions. Also, I propose to only make tiny incompatible changes. I started around Python 3.7. If you didn't notice, it means that my incompatible changes only impact a minority of users ;-) It's nothing like Python str type becoming Unicode in Python 3.
Your tool is already going in that direction, but rather than having to run the tool over and over again for every release, which adds testing and verification overhead every single time, it would be better to have something like HPy readily working and then use a tool like pythoncapi_compat to migrate over the code once, inserting a shim between the extension using the old style Python C API and the new HPy Python C API.
I agree that HPy is a better solution for the long term. But HPy is not mature yet: https://hpy.readthedocs.io/en/latest/overview.html#current-status-and-roadma...
I don't want to have to wait years until HPy is mature. We can already make a few changes which only impact a minority of users.
That shim could also be had at a higher level by e.g. using Cython as layer between the extension and Python; and using a tool to help migrate towards this approach.
Right. Don't use the C API, but Cython ;-) Cython can be modified to emit HPy code.
What I want to say is that we can take our time to avoid disruption :-)
In short, you are against CPython evolutions and consider that CPython is good as it is (it's not worth it to optimize CPython). Well, that's one of the two choices in my previous email.
Victor
Night gathers, and now my watch begins. It shall not end until my death.