On 09.09.2021 16:18, Antonio Cuni wrote:
On Thu, Sep 9, 2021 at 12:30 PM Victor Stinner vstinner@python.org wrote:
We cannot evolve Python (fix bugs, optimize it) without this annoying work of fixing the C API: add more abstractions on top of C structures.
I agree with this and I think that it's the core of the problem. Currently, the C API leaks too many implementation details and makes it almost impossible to change anything in CPython.
The first thing to acknowledge is that CPython is stuck with design decisions which were taken in 1990 and that cannot be changed without breaking (some) backwards compatibility. That's a fact.
Another fact is that there is no single breaking change and/or abstraction which by itself enables big refactorings/experiments/optimizations, because many of them have multiple blockers. So if you take any breaking change on its own, it's never worthwhile because it breaks compatibility for no immediate gain. On the other hand, every such small change decreases a tiny bit of the technical debt which has been accumulated over the years.
IMHO, the correct question that the core CPython developers need to ask themselves is how much technical debt they are willing to carry over forever for the sake of backwards compatibility at all costs. I think that the tone of my email makes it clear what my personal opinion is :), but again this is something which should be decided by CPython developers.
I think there's overall agreement that the C API needs to be tweaked to make it easier to add optimizations to the internals. I also think there's little push back against the particular change which triggered the discussion - getters/setters do make things easier.
Where I see need for discussion is the strategy on how this should be applied. Some possibilities:
a) Many small changes throughout the next few years, each release with a certain smaller number of them b) One larger change every few releases c) One huge change for Python 4.0
The problem with C API changes is that extensions need to be adapted. If that has to be done for every single Python release, this causes a lot of churn. It may also cause issues for packages to support multiple Python versions (e.g. PyData packages typically support 3.7 - 3.9). Victor's package can help with the latter, but it doesn't solve the former issue.
This is something the SC should provide some guidance on, since the current approach (which is close to a)) makes life hard for the community.
I'm also missing an overarching strategy of where the C API should be heading from the SC in the coming years, so that everyone is on the same page.
I put the SC on CC to get this on their radar.
Related to all this, since it's come up often in these change discussions:
If we want to officially push for Cython as a way to provide a layer between the Python C API and Python C extensions, the PSF needs to coordinate some serious and sustained funding for that project. Otherwise, we end up with a single point of failure in our extension stack, which isn't even under SC control.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 09 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/