On 2/1/21 3:41 PM, Victor Stinner wrote:
Hi Petr,
On Mon, Feb 1, 2021 at 11:06 AM Petr Viktorin <encukou@gmail.com> wrote:
I think "introducing incompatible changes on purpose" makes Python worse without much benefit. Has any of the recent incompatible changes actually made Python faster?
Optimizing Python is a hard problem. Look back at the 30 years of CPython optimizations. There were some nice optimizations, but the CPython design is the same for 30 years. If it would be easy, it would already be done. There are big companies would need a faster CPython. Significant speedup (ex: 2x faster) require to deeply change CPython design.
It will only become possible to *start* *experimenting* optimizing CPython when most of the PEP 620 will be implemented. Some people disagree, but so far, I didn't see any concrete working optimization proposed and implemented in CPython (without the PEP 620 being implemented). So everyone is free to have their own "beliefs" in term of optimization possibilities (with or without incompatible C API changes) ;-)
To be clear, the PEP 620 brings zero optimization. It should be done separately. The PEP 620 only prepares CPython for future optimizations.
Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API and stable ABI as global symbols. This prevents making the GIL per-interpreter. Here's how stuff like that can be changed:
- Formalize, test and document the stable ABI
- Introduce new API (e.g. PyList_GetListType())
- Deprecate the old API
- Create
abi4, a new, incompatible version of the stable ABI, which doesn't include deprecated stuff. Create a flag to mark extension modules that use it.- Make it possible to create subinterpreters with a separate GIL. These can only load
abi4extension modules.Step 4 introduces a backward incompatible change without much benefit :-)
It does not. All existing modules would continue to work.
Step 5 will only be safe once all static types will be removed, which is an incompatible change.
Again, no – all existing code would continue to work.
I see that I didn't explain one aspect of the idea: there would be two kinds of subinterpreters – one that shares the GIL with the "main" interpreter, and a new opt-in kind that does not share the GIL but only allows loading abi4 modules. I don't think this would be hard to do in practice: GIL must already be shared across thread states belonging to a single interpreter; it doesn't seem hard to have some interpreters sharing the GIL and some not.
For major CPython design changes (like running multiple interpreters in parallel), we *need* to introduce incompatible changes.
Sure, and we have PEP 387 for that. A backwards-incompatible change can be done after at least two years of warnings (and if warnings are not possible, then with an accepted PEP). Incompatible changes should be the last resort, and there needs to be enough time to think about approaches that don't break backwards compatibility.
There are two options: consider that CPython must no longer evolve (and so die slowly as a dead language ;-)), or we should find a way to minimize the number of unhappy people when we introduce incompatible changes :-)
This is a false dichotomy. It is also possible to make progress without breaking so much existing code. It's harder and takes more time, but it's much better for existing software.
- Work with various third-party projects to switch to abi4 so we can start getting its benefits, and improve abi4 along the way.
Over the last years, I proposed different PEP drafts with an opt-in choice for a faster python which is backward incompatible. Nobody liked this option (having two Python). Look at the Python 2 => Python 3 migration and "CPython vs PyPy". "Faster Python" is not enough to migrate the critical mass of users to the "new" Python, the old Python never goes away, and then you have to maintain two Python runtimes instead of one.
My new approach is to do all changes directly in Python master branch (single Python) and slowly introduce incompatible changes. When too many people complain, I revert the change and help to migrate extensions to the new API. I'm doing that for Py_SET_TYPE() and Py_SET_SIZE() using my pythoncapi_compat project for example (I'm waiting for my Mercurial change to be merged upstream ;-)).
https://github.com/pythoncapi/pythoncapi_compat allows to make existing extensions compatibile with the future incompatible changes, without dropping support for old Python versions. The general plan is:
(a) Introduce a new API and deprecate the old one (b) Run pythoncapi_compat on enough extension modules (and get changes accepted) (c) Once enough extensions are ported, drop the old API
I suggest to have at least one Python release between (a) and (c). For example, I introduced Py_SET_REFCNT() in Python 3.9 and I disallowed "Py_REFCNT(obj) = refcnt;" syntax in Python 3.10.
PEP 387 is very clear that this should be at least *two* releases. And only if the old behavior raises warnings for those two releases.
On top of that, I don't plans to drop some API should be made until the old API is *actually* incompatible with a real improvement (e.g. speedup). Until then, any experiments can be done on a subset of extension modules, without requiring everyone in the ecosystem to make changes. If people end up needing to run pythoncapi_compat, it would be much nicer to let them run
During (b) phase, we can communicate on the future incompatible changes in What's New in Python X.Y, communicate on capi-sig list and/or python-dev, contact extensions maintainers, etc.
I tried to explain my failures and my new approach on C API incompatible changes in an article: https://vstinner.github.io/hide-implementation-details-python-c-api.html
?. In Python 4.0, drop
abi3.I don't see how you can run interpreters in parallel if static types are still in use. In my experience, it immediately crash on various ways: https://bugs.python.org/issue40512#msg383830
PEP 620 has many things, some of which conflict with the accepted PEP 387. I don't think all of it is a good plan (and I guess that's why it wasn't accepted).
The PEP 387 is a generic guidelines how to introduce incompatible changes. It remains possible to introduce incompatible changes differently (ex: without deprecation warning or with no deprecation period) with a dedicated PEP. That's why I wrote PEP 620.
The worrying thing is that PEP 620 is still a Draft, even though most action points from it are "Completed". Why do we even have a PEP approval process if it's not being used before changes are done?
On the items from PEP 620 that are not completed yet:
Make structures opaque
Avoid functions returning PyObject** I support these ideas, but again, the old API should stay available until the underlying implementation actually changes to make them impossible. Improvements that these functions are preventing can still be tested – they just need to be tested with modules that avoid the old functions.
pythoncapi_compat.h header file I don't think this is necessary if we're more careful about backwards compatibility.