[capi-sig] Re: Pre-PEP: Maintaining the Stable ABI

Hi Petr,

On Mon, Feb 1, 2021 at 11:06 AM Petr Viktorin <encukou@gmail.com> wrote:

I think "introducing incompatible changes on purpose" makes Python worse without much benefit. Has any of the recent incompatible changes actually made Python faster?

Optimizing Python is a hard problem. Look back at the 30 years of CPython optimizations. There were some nice optimizations, but the CPython design is the same for 30 years. If it would be easy, it would already be done. There are big companies would need a faster CPython. Significant speedup (ex: 2x faster) require to deeply change CPython design.

It will only become possible to *start* *experimenting* optimizing CPython when most of the PEP 620 will be implemented. Some people disagree, but so far, I didn't see any concrete working optimization proposed and implemented in CPython (without the PEP 620 being implemented). So everyone is free to have their own "beliefs" in term of optimization possibilities (with or without incompatible C API changes) ;-)

To be clear, the PEP 620 brings zero optimization. It should be done separately. The PEP 620 only prepares CPython for future optimizations.

Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API and stable ABI as global symbols. This prevents making the GIL per-interpreter. Here's how stuff like that can be changed:

1. Formalize, test and document the stable ABI
2. Introduce new API (e.g. PyList_GetListType())
3. Deprecate the old API
4. Create abi4, a new, incompatible version of the stable ABI, which doesn't include deprecated stuff. Create a flag to mark extension modules that use it.
5. Make it possible to create subinterpreters with a separate GIL. These can only load abi4 extension modules.

Step 4 introduces a backward incompatible change without much benefit :-)

Step 5 will only be safe once all static types will be removed, which is an incompatible change.

For major CPython design changes (like running multiple interpreters in parallel), we *need* to introduce incompatible changes.

There are two options: consider that CPython must no longer evolve (and so die slowly as a dead language ;-)), or we should find a way to minimize the number of unhappy people when we introduce incompatible changes :-)

6. Work with various third-party projects to switch to abi4 so we can start getting its benefits, and improve abi4 along the way.

Over the last years, I proposed different PEP drafts with an opt-in choice for a faster python which is backward incompatible. Nobody liked this option (having two Python). Look at the Python 2 => Python 3 migration and "CPython vs PyPy". "Faster Python" is not enough to migrate the critical mass of users to the "new" Python, the old Python never goes away, and then you have to maintain two Python runtimes instead of one.

My new approach is to do all changes directly in Python master branch (single Python) and slowly introduce incompatible changes. When too many people complain, I revert the change and help to migrate extensions to the new API. I'm doing that for Py_SET_TYPE() and Py_SET_SIZE() using my pythoncapi_compat project for example (I'm waiting for my Mercurial change to be merged upstream ;-)).

https://github.com/pythoncapi/pythoncapi_compat allows to make existing extensions compatibile with the future incompatible changes, without dropping support for old Python versions. The general plan is:

(a) Introduce a new API and deprecate the old one (b) Run pythoncapi_compat on enough extension modules (and get changes accepted) (c) Once enough extensions are ported, drop the old API

I suggest to have at least one Python release between (a) and (c). For example, I introduced Py_SET_REFCNT() in Python 3.9 and I disallowed "Py_REFCNT(obj) = refcnt;" syntax in Python 3.10.

During (b) phase, we can communicate on the future incompatible changes in What's New in Python X.Y, communicate on capi-sig list and/or python-dev, contact extensions maintainers, etc.

I tried to explain my failures and my new approach on C API incompatible changes in an article: https://vstinner.github.io/hide-implementation-details-python-c-api.html

?. In Python 4.0, drop abi3.

I don't see how you can run interpreters in parallel if static types are still in use. In my experience, it immediately crash on various ways: https://bugs.python.org/issue40512#msg383830

PEP 620 has many things, some of which conflict with the accepted PEP 387. I don't think all of it is a good plan (and I guess that's why it wasn't accepted).

The PEP 387 is a generic guidelines how to introduce incompatible changes. It remains possible to introduce incompatible changes differently (ex: without deprecation warning or with no deprecation period) with a dedicated PEP. That's why I wrote PEP 620.

Victor

Night gathers, and now my watch begins. It shall not end until my death.