On Thu, Sep 9, 2021 at 12:30 PM Victor Stinner <vstinner@python.org> wrote:
We cannot evolve Python (fix bugs, optimize it) without this annoying work of fixing the C API: add more abstractions on top of C structures.
I agree with this and I think that it's the core of the problem. Currently, the C API leaks too many implementation details and makes it almost impossible to change anything in CPython.
The first thing to acknowledge is that CPython is stuck with design decisions which were taken in 1990 and that cannot be changed without breaking (some) backwards compatibility. That's a fact.
Another fact is that there is no single breaking change and/or abstraction which by itself enables big refactorings/experiments/optimizations, because many of them have multiple blockers. So if you take any breaking change on its own, it's never worthwhile because it breaks compatibility for no immediate gain. On the other hand, every such small change decreases a tiny bit of the technical debt which has been accumulated over the years.
IMHO, the correct question that the core CPython developers need to ask themselves is how much technical debt they are willing to carry over forever for the sake of backwards compatibility at all costs. I think that the tone of my email makes it clear what my personal opinion is :), but again this is something which should be decided by CPython developers.