Hi,

Recently I had started some effort to make sure the symbols are listed somewhere and that they are exported (as some of them were mistakenly removed in the past). You can see my initial PR here:

https://github.com/python/cpython/pull/23616

This is currently active in the CPython CI.

Have a great start of 2021!

On Thu, 31 Dec 2020, 18:18 Petr Viktorin, encukou@gmail.com wrote:

Hello. I am writing a PEP to describe how I'd like to "resurrect" and maintain the Stable ABI & Limited API going forward. I assume this'll be my main focus for CPython in 2021. I'm not as far along as I hoped to be at the end of 2020, but I guess it's time to request comments.

If you have any thoughts, arguments or improvements I'd be happy to hear them!

I'm including the text below; a rendered and forkable/pull-request-able version is at https://github.com/encukou/abi3/blob/main/PEP.rst

Have a great start of 2021!

---

PEP: 9999 Title: Maintaining the Stable ABI Author: Petr Viktorin encukou@gmail.com Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 08-Dec-2020

Abstract

[A short (~200 word) description of the technical issue being addressed.]

XXX: Abstract should be written last

Motivation

:pep:384 defined a limited API and stable ABI, which allows extenders and embedders of CPython to compile extension modules that are binary-compatible with any subsequent version of 3.x. In theory, this brings many advantages:

• A module can be built only once per platform and support multiple versions of Python, reducing time, power and maintainer attention needed for builds.
• Binary wheels using the stable ABI work with new versions of CPython throughout the pre-release period, and can be tested in environments where building from source is not practical.

As a welcome side effect of the stable ABI's hiding of implementation details is that it is becoming a viable target for alternate Python implementations that need to implement (parts of) the C API.

However, in hindsignt, PEP 384 and its implementation has several issues:

• There is no process keep the ABI up to date.
• Contents of the limited API are not listed explicitly, making it unclear if a particular member (e.g. function, structure) is a part of it.
• There is no way to deprecate parts of the limited API.

This PEP defines the limited API more clearly and introducess process designed to make the API more useful.

Additionally, PEP 384 defines a *limited API* as a way to build against the stable ABI. This PEP defines the limited API more robustly.

Rationale

This PEP contains a lot of clarifications and definitions, but just one big technical change: the stable ABI will be explicitly listed in a human-maintained “manifest” file.

There have been efforts to collect such lists automatically, e.g. by scanning the symbols exported from Python. This might seem to be easier to maintain by our volunteer team.

However, designing a future-proof API is not a trivial task. The cost of updating an explicit manifest is small compared to the overall work that should go into changing API that will need to be suppported forever (or until Python 3 reaches end of life, if that comes sooner).

This PEP proposes automatically generating things *from* the manifest: initially documentation and DLL contents, with later possibilities for also automating tests.

Stable ABI vs. Limited API

:pep:384 and this document deal with the *Limited API* and the *Stable ABI*, two related but distinct concepts. This section clarifies what they mean and defines some of their semantics (either pre-existing or newly proposed here).

The word “Extensions” is used as a shorthand for all code that uses the Python API, e.g. extension modules or software that embeds Python.

Stable ABI

The CPython *Stable ABI* is a promise that extensions built with a specific Cpython version will be usable with any newer interpreter of the same major version, on the same platform and with the same compiler & settings. For example, a extension built with CPython 3.10 Stable ABI will be usable with CPython 3.11, 3.12, and so on, but not necessarily with 4.0.

The Stable ABI is not generally forward-compatible: an extension built and tested with CPython 3.10 will not generally be compatible with CPython 3.9.

.. note:: For example, starting in Python 3.10, the Py_tp_doc slot may be set to NULL, while in older versions, a NULL value will likely crash the interpreter.

The Stable ABI trades performance for its stability. For example, many functions in the stable ABI are available as faster macros to extensions that are built for a specific CPython version.

Future Python sversions may deprecate some members of the Stable ABI. Such members will still work, but may suffer from issues like reduced performance or, in the most extreme cases, memory/resource leaks.

Limited API

Stable ABI guarantee holds for extensions compiled from code that restricts itself to the *Limited API*, a subset of CPython's C API.

The limited API is used when preprocessor macro Py_LIMITED_API is defined to either 3 or the current PYTHON_API_VERSION.

The Limited API is not guaranteed to be *stable*. In the future, parts of the limited API may be deprecated. They may even be removed, as long as the *stable ABI* is kept stable and Python's general backwards compatibility policy, :pep:387, is followed.

.. note::

For example, a function declaration might be removed from public header
files but kept in the library.
This is currently a possibility for the future; this PEP does not to

propose a concrete process for deprecations and removals.

The goal is for the limited API to cover everything needed to interact with the interpreter. There main reasons to not include a public API in the limited subset should be that it needs implementation details that change between CPython versions, like struct memory layouts, for performance reasons.

The limited API is not limited to CPython; other implementations are encouraged to implement it and help drive its design.

Specification

To make the stable ABI more useful and stable, the following changes are proposed.

Stable ABI Manifest

All members of the stable ABI – functions, typedefs, structs, struct fields, data values etc. – will be explicitly listed in a single "manifest" file, along with the Limited API version they were added in. Struct fields that users of the stable ABI are allowed to access will be listed explicitly. Members that are not part of the Limited API, but are part of the Stable ABI (e.g. PyObject.ob_type, which is accessible by the Py_TYPE macro), will be annotated as such.

Notes saying “Part of the stable ABI” will be added to Python's documentation automatically, in a way similar to the notes on functions that return borrowed references.

Source for the Windows shared library python3.dll will be generated from the stable ABI definition.

The format of the manifest will be subject to change whenever needed. It should be consumed only by scripts in the CPython repository. If a more public list is needed, a script can be added to generate it.

Contents of the Stable ABI

The initial stable ABI manifest will include:

• The Stable ABI specified in :pep:384.
• All functions listed in PC/python3dll.c.
• All structs (struct typedefs) which these functions return or take as arguments. (Fields of such structs will not necessarily be added.)
• New type slots, such as Py_am_aiter.
• The type flags Py_TPFLAGS_DEFAULT, Py_TPFLAGS_BASETYPE, Py_TPFLAGS_HAVE_GC, Py_TPFLAGS_METHOD_DESCRIPTOR.
• The calling conventions METH_* (except deprecated ones).
• All API needed by macros is the stable ABI (usually annotated as not being part of the limited API).

Additional items may be aded to the initial manifest according to the checklist below.

Testing the Stable ABI

An automatically generated test module will be added to ensure that all members of the stable ABI are available at compile time

For each function in the stable ABI, a test will be added that calls the function using ctypes. (Where calling is not practical, such as with functions related to intepreter initialization and shutdown, the test will only look the function up.) This should prevent regressions when a function is converted to a macro, which keeps the same API but breaks the ABI. An check will be added to ensure all functions in the stable ABI are tested this way.

Changing the Limited API

A checklist for changing the limited API, including new members (structs, functions or values), will be added to the Devguide_. The checklist will 1) mention best practices and common pitfalls in Python C API design and 2) guide the developer around the files that need changing and scripts that need running when the limited API is changed.

Below is the initial proposal for the checklist. After the PEP is accepted, see the Devguide for the current version.

Note that the checklist applies to new additions; not the existing limited API.

Design considerations:

• Make sure the change does not break the Stable ABI of any version of Python since 3.5.

• Make sure no exposed names are private (i.e. begin with an underscore).

• Make sure the new API is well documented.

• Make sure the types of all parameters and return values of the added function(s) and all fields of the added struct(s) are be part of the limited API (or standard C).

• Make sure the new API and its intended use follows standard C, not just features of currently suppoerted platforms.

  • Do not cast a function pointer to void* (a data pointer) or vice versa.

• Make sure the new API follows reference counting conventions. (Following them makes the API easier to reason about, and easier use in other Python implementations.)

  • Do not return borrowed references from functions.
  • Do not steal references to function arguments.

• Make sure the ownership rules and lifetimes of all applicable struct fields, arguments and return values are well defined.

• Think about ease of use for the user. (In C, ease of use itself is not very important; what *is* important is reducing boilerplate code needed to use the API. Bugs like to hide in boiler plates.)

  • If a function will be often called with specific value for an argument, consider making it default (assumed when NULL is passed in).

• Think about future extensions: for example, if it's possible that future Python versions will need to add a new field to your struct, how will that be done?

• Make as few assumptions as possible about details that might change in future CPython versions or differ across C API implementations:

  • The GIL
  • Garbage collection
  • Layout of PyObject and other structs

If following these guidelines would hurt performance, add a fast function (or macro) to the non-limited API and a stable equivalent to the limited API.

If anything is unclear, or you have a good reason to break the guidelines, consider discussing the change at the capi-sig_ mailing list.

.. _capi-sig: https://mail.python.org/mailman3/lists/capi-sig.python.org/

Procedure:

• Move the declaration to a header file directly under Include/ and #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000 (with the yy corresponding to Python version).
• Make an entry in the stable ABI list. (XXX: mention filename)
• For functions, add a test that calls the function using ctypes (XXX: mention filename).
• Regenerate the autogenerated files. (XXX: specific instructions)

Advice for Extenders and Embedders

The following notes will be added to documentation.

Extension authors should test with all Python versions they support, and preferably build with the lowest such version.

Compiling with Py_LIMITED_API defined is *not* a guarantee that your code conforms to the limited API or the stable ABI. It only covers definitions, but an API also includes other issues, such as expected semantics.

Examples of issues that Py_LIMITED_API does not guard against are:

• Calling a function with invalid arguments
• An function that started accepting NULL values for an argument in Python 3.9 will fail if NULL is passed to it under Python 3.8. Only testing with 3.8 (or lower versions) will uncover this issue.
• Some structs include a few fields that are part of the stable ABI and other fields that aren't. Py_LIMITED_API does not filter out such “private” fields.
• Using something that is not documented as part of the stable ABI, but exposed even with Py_LIMITED_API defined. Despite the team's best efforts, such issues may happen.

Backwards Compatibility

The PEP aims at full compatibility with the existing stable ABI and limited API, but defines them terms more explicitly. It might not be consistent with some interpretations of what the existing stable ABI/limited API is.

Security Implications

None known.

How to Teach This

Technical documentation will be provided. It will be aimed at experienced users familiar with C.

Reference Implementation

None so far.

Rejected Ideas

While this PEP acknowledges that parts of the limited API might be deprecated or removed in the future, a process to do this is not in scope, and is left to a possible future PEP.

Open Issues

None so far.

References

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

.. _Devguide: https://devguide.python.org/

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

---

capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: pablogsal@gmail.com

Thanks for putting the proposal together. I will read it carefully as well to see if I can be of help.

One note regarding the current checks: the current script was mainly aimed at avoiding symbols to be removed mistakenly because that has happened 3 or 4 times already but there are many things that are not checked currently. For instance we don't check currently that the API or the ABI are fully preserved so changing the type of a field in a struct or a parameter in a function will not be detected with this initial check. For the CI to fail you need to remove an entire symbol. That's why I think your proposal of getting something automated using ctypes is much better and will certainly be the future of these checks.

On Thu, 31 Dec 2020, 18:29 Petr Viktorin, encukou@gmail.com wrote:

Thanks! I'll mention it in the next draft.

A big reason I started writing this is to get everyone on the same and coordinate, as I'm having trouble keeping track of everything happening around the API. I'm glad to learn that you're one step ahead of me in the automated checking!

...

Hi,

Recently I had started some effort to make sure the symbols are listed somewhere and that they are exported (as some of them were mistakenly removed in the past). You can see my initial PR here:

https://github.com/python/cpython/pull/23616 https://github.com/python/cpython/pull/23616

This is currently active in the CPython CI.

Have a great start of 2021!

On Thu, 31 Dec 2020, 18:18 Petr Viktorin, mailto:encukou@gmail.com> wrote:

Hello.
I am writing a PEP to describe how I'd like to "resurrect" and

On 12/31/20 7:23 PM, Pablo Galindo Salgado wrote: maintain

...

the Stable ABI & Limited API going forward. I assume this'll be my

main

...

focus for CPython in 2021.
I'm not as far along as I hoped to be at the end of 2020, but I guess
it's time to request comments.

If you have any thoughts, arguments or improvements I'd be happy to
hear
them!

I'm including the text below; a rendered and

forkable/pull-request-able

...

version is at https://github.com/encukou/abi3/blob/main/PEP.rst
<https://github.com/encukou/abi3/blob/main/PEP.rst>

Have a great start of 2021!


---

PEP: 9999
Title: Maintaining the Stable ABI
Author: Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>>
Discussions-To:
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Dec-2020


Abstract
========

[A short (~200 word) description of the technical issue being
addressed.]

XXX: Abstract should be written last


Motivation
==========

:pep:`384` defined a limited API and stable ABI, which allows
extenders and
embedders of CPython to compile extension modules that are
binary-compatible
with any subsequent version of 3.x.
In theory, this brings many advantages:

* A module can be built only once per platform and support multiple
versions
    of Python, reducing time, power and maintainer attention needed

for

...

builds.
* Binary wheels using the stable ABI work with new versions of

CPython

...

    throughout the pre-release period, and can be tested in
environments
where
    building from source is not practical.

As a welcome side effect of the stable ABI's hiding of implementation
details
is that it is becoming a viable target for alternate Python
implementations
that need to implement (parts of) the C API.

However, in hindsignt, PEP 384 and its implementation has several
issues:

* There is no process keep the ABI up to date.
* Contents of the limited API are not listed explicitly, making it
unclear
    if a particular member (e.g. function, structure) is a part of

it.

...

* There is no way to deprecate parts of the limited API.

This PEP defines the limited API more clearly and introducess process
designed to make the API more useful.

Additionally, PEP 384 defines a *limited API* as a way to build
against the
stable ABI.
This PEP defines the limited API more robustly.


Rationale
=========

This PEP contains a lot of clarifications and definitions, but just
one big
technical change: the stable ABI will be explicitly listed in
a human-maintained “manifest” file.

There have been efforts to collect such lists automatically, e.g. by
scanning
the symbols exported from Python.
This might seem to be easier to maintain by our volunteer team.

However, designing a future-proof API is not a trivial task.
The cost of updating an explicit manifest is small compared
to the overall work that should go into changing API that will need

to

...

be suppported forever (or until Python 3 reaches
end of life, if that comes sooner).

This PEP proposes automatically generating things *from* the

manifest:

...

initially documentation and DLL contents, with later possibilities
for also automating tests.


Stable ABI vs. Limited API
==========================

:pep:`384` and this document deal with the *Limited API* and the
*Stable
ABI*,
two related but distinct concepts.
This section clarifies what they mean and defines some of their
semantics
(either pre-existing or newly proposed here).

The word “Extensions” is used as a shorthand for all code that uses

the

...

Python API, e.g. extension modules or software that embeds Python.


Stable ABI
----------

The CPython *Stable ABI* is a promise that extensions built with a
specific
Cpython version will be usable with any newer interpreter of the
same major
version, on the same platform and with the same compiler & settings.
For example, a extension built with CPython 3.10 Stable ABI will be
usable with
CPython 3.11, 3.12, and so on, but not necessarily with 4.0.

The Stable ABI is not generally forward-compatible: an extension
built and
tested with CPython 3.10 will not generally be compatible with
CPython 3.9.

.. note::
     For example, starting in Python 3.10, the `Py_tp_doc` slot may
be set to
     `NULL`, while in older versions, a `NULL` value will likely
crash the
     interpreter.

The Stable ABI trades performance for its stability.
For example, many functions in the stable ABI are available as
faster macros
to extensions that are built for a specific CPython version.

Future Python sversions may deprecate some members of the Stable ABI.
Such members will still work, but may suffer from issues like reduced
performance or, in the most extreme cases, memory/resource leaks.


Limited API
-----------

Stable ABI guarantee holds for extensions compiled from code that
restricts
itself to the *Limited API*, a subset of CPython's C API.

The limited API is used when preprocessor macro `Py_LIMITED_API` is
defined
to either `3` or the current `PYTHON_API_VERSION`.

The Limited API is not guaranteed to be *stable*.
In the future, parts of the limited API may be deprecated.
They may even be removed, as long as the *stable ABI* is kept
stable and Python's general backwards compatibility policy,

:pep:387,

...

is followed.

.. note::

     For example, a function declaration might be removed from
public header
     files but kept in the library.
     This is currently a possibility for the future; this PEP does
not to
propose
     a concrete process for deprecations and removals.

The goal is for the limited API to cover everything needed to

interact

...

with the interpreter.
There main reasons to not include a public API in the limited subset
should be that it needs implementation details that change between
CPython
versions, like struct memory layouts, for performance reasons.

The limited API is not limited to CPython; other implementations are
encouraged to implement it and help drive its design.


Specification
=============

To make the stable ABI more useful and stable, the following changes
are proposed.


Stable ABI Manifest
-------------------

All members of the stable ABI – functions, typedefs, structs, struct
fields,
data values etc. – will be explicitly listed in a single "manifest"
file,
along with the Limited API version they were added in.
Struct fields that users of the stable ABI are allowed to access

will be

...

listed explicitly.
Members that are not part of the Limited API, but are part of the
Stable ABI
(e.g. ``PyObject.ob_type``, which is accessible by the ``Py_TYPE``
macro),
will be annotated as such.

Notes saying “Part of the stable ABI” will be added to Python's
documentation
automatically, in a way similar to the notes on functions that return
borrowed
references.

Source for the Windows shared library `python3.dll` will be generated
from the
stable ABI definition.

The format of the manifest will be subject to change whenever needed.
It should be consumed only by scripts in the CPython repository.
If a more public list is needed, a script can be added to generate

it.

...

Contents of the Stable ABI
--------------------------

The initial stable ABI manifest will include:

* The Stable ABI specified in :pep:`384`.
* All functions listed in ``PC/python3dll.c``.
* All structs (struct typedefs) which these functions return or take

as

...

    arguments. (Fields of such structs will not necessarily be

added.)

...

* New type slots, such as ``Py_am_aiter``.
* The type flags  ``Py_TPFLAGS_DEFAULT``, ``Py_TPFLAGS_BASETYPE``,
    ``Py_TPFLAGS_HAVE_GC``, ``Py_TPFLAGS_METHOD_DESCRIPTOR``.
* The calling conventions ``METH_*`` (except deprecated ones).
* All API needed by macros is the stable ABI (usually annotated as
not being
    part of the limited API).

Additional items may be aded to the initial manifest according to
the checklist below.


Testing the Stable ABI
----------------------

An automatically generated test module will be added to ensure that

all

...

members
of the stable ABI are available at compile time

For each function in the stable ABI, a test will be added that calls

the

...

function using `ctypes`. (Where calling is not practical, such as

with

...

functions related to intepreter initialization and shutdown, the
test will
only look the function up.)
This should prevent regressions when a function is converted to a

macro,

...

which keeps the same API but breaks the ABI.
An check will be added to ensure all functions in the stable ABI are
tested
this way.


Changing the Limited API
------------------------

A checklist for changing the limited API, including new members
(structs,
functions or values), will be added to the `Devguide`_.
The checklist will 1) mention best practices and common pitfalls in
Python
C API design and 2) guide the developer around the files that need
changing and
scripts that need running when the limited API is changed.

Below is the initial proposal for the checklist. After the PEP is
accepted,
see the Devguide for the current version.

Note that the checklist applies to new additions; not the existing
limited API.

Design considerations:

* Make sure the change does not break the Stable ABI of any version

of

...

Python
    since 3.5.
* Make sure no exposed names are private (i.e. begin with an
underscore).
* Make sure the new API is well documented.
* Make sure the types of all parameters and return values of the

added

...

    function(s) and all fields of the added struct(s) are be part of

the

...

    limited API (or standard C).
* Make sure the new API and its intended use follows standard C, not
just
    features of currently suppoerted platforms.

    * Do not cast a function pointer to ``void*`` (a data pointer) or
vice versa.

* Make sure the new API follows reference counting conventions.
(Following them
    makes the API easier to reason about, and easier use in other

Python

...

    implementations.)

    * Do not return borrowed references from functions.
    * Do not steal references to function arguments.

* Make sure the ownership rules and lifetimes of all applicable

struct

...

fields,
    arguments and return values are well defined.
* Think about ease of use for the user. (In C, ease of use itself is
not
very
    important; what *is* important is reducing boilerplate code
needed to
use the
    API. Bugs like to hide in boiler plates.)

    * If a function will be often called with specific value for an
argument,
      consider making it default (assumed when ``NULL`` is passed

in).

...

* Think about future extensions: for example, if it's possible that
future
    Python versions will need  to add a new field to your struct,
    how will that be done?

* Make as few assumptions as possible about details that might

change in

...

    future CPython versions or differ across C API implementations:

      * The GIL
      * Garbage collection
      * Layout of PyObject and other structs

If following these guidelines would hurt performance, add a fast
function
(or macro) to the non-limited API and a stable equivalent to the
limited
API.

If anything is unclear, or you have a good reason to break the
guidelines,
consider discussing the change at the `capi-sig`_ mailing list.

.. _capi-sig:
https://mail.python.org/mailman3/lists/capi-sig.python.org/
<https://mail.python.org/mailman3/lists/capi-sig.python.org/>

Procedure:

* Move the declaration to a header file directly under ``Include/``

and

...

    ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >=

0x03yy0000``

...

    (with the ``yy`` corresponding to Python version).
* Make an entry in the stable ABI list. (XXX: mention filename)
* For functions, add a test that calls the function using ctypes
    (XXX: mention filename).
* Regenerate the autogenerated files. (XXX: specific instructions)


Advice for Extenders and Embedders
----------------------------------

The following notes will be added to documentation.

Extension authors should test with all Python versions they support,
and preferably build with the lowest such version.

Compiling with ``Py_LIMITED_API`` defined is *not* a guarantee that
your
code
conforms to the limited API or the stable ABI.
It only covers definitions, but an API also includes other issues,
such as expected semantics.

Examples of issues that ``Py_LIMITED_API`` does not guard against

are:

...

* Calling a function with invalid arguments
* An function that started accepting ``NULL`` values for an argument
    in Python 3.9 will fail if ``NULL`` is passed to it under Python
3.8.
    Only testing with 3.8 (or lower versions) will uncover this

issue.

...

* Some structs include a few fields that are part of the stable ABI

and

...

other
    fields that aren't.
    ``Py_LIMITED_API`` does not filter out such “private” fields.
* Using something that is not documented as part of the stable ABI,
    but exposed even with ``Py_LIMITED_API`` defined.
    Despite the team's best efforts, such issues may happen.


Backwards Compatibility
=======================

The PEP aims at full compatibility with the existing stable ABI and
limited
API, but defines them terms more explicitly.
It might not be consistent with some interpretations of what the
existing
stable ABI/limited API is.


Security Implications
=====================

None known.


How to Teach This
=================

Technical documentation will be provided.
It will be aimed at experienced users familiar with C.


Reference Implementation
========================

None so far.


Rejected Ideas
==============

While this PEP acknowledges that parts of the limited API might be
deprecated
or removed in the future, a process to do this is not in scope, and
is left
to a possible future PEP.


Open Issues
===========

None so far.


References
==========


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


.. _Devguide: https://devguide.python.org/
<https://devguide.python.org/>

..
      Local Variables:
      mode: indented-text
      indent-tabs-mode: nil
      sentence-end-double-space: t
      fill-column: 70
      coding: utf-8
      End:
_______________________________________________
capi-sig mailing list -- capi-sig@python.org
<mailto:capi-sig@python.org>
To unsubscribe send an email to capi-sig-leave@python.org
<mailto:capi-sig-leave@python.org>
https://mail.python.org/mailman3/lists/capi-sig.python.org/
<https://mail.python.org/mailman3/lists/capi-sig.python.org/>
Member address: pablogsal@gmail.com <mailto:pablogsal@gmail.com>

Hi,

Yeah, the stable ABI is not tested, maintained manually and errors are very common :-( Something should be done to simply its maintenance and reduce the risk of manual errors. This PEP looks like a great step forward!

If we want to optimize CPython (which remains an open question for 5 years at least!), the limited C API and the stable ABI must evolve. I tried to write down technical reasons to explain why: https://pythoncapi.readthedocs.io/optimize_python.html

For example, the ability to dereference a PyObject* to access PyObject members prevents implementing a moving garbage collector.

"Evolving" here means "introducing incompatible changes on purpose" which goes against the principle of a "stable" API :-( If we break the API once, maybe we should try take in account all currently known issues and try to address all of them at once. The problem is that the current C API is based on "PyObject*". Moving away from PyObject* would be a radical change (more or less rewrite the whole C API). So it doesn't seem possible to fix the API "at once". (I'm not talking about the stable ABI here, but the API.)

The HPy project tries to design a new C API from scratch which doesn't inherit these design issues. Extensions written with HPy are faster on PyPy than the same existing written with the Python C API, and HPy is as fast as the Python C API on CPython. HPy C API is incompatible with the existing Python C API but the migration should not be "too hard" (I didn't try, so I cannot testify). HPy design should also provide a stable ABI (but the project is not mature yet). One advantage is that HPy doesn't require to touch CPython, it's developed externally.

HPy is still young since it's uncertain to only bet on it. That's why I'm modifying the C API in CPython (hide implementation details: PEP 620) in the meanwhile to make it "less broken" (prepare it for future optimizations).

It would be great if optimizing Python and making the stable ABI would not be two "incompatible" goals. But the practical question is more "do we want to fix a few issues with minor incompatible changes in the short term?" or "do we want to fix all issues with major incompatible changes at once for the long term?". Since HPy is developed externally, it doesn't seem incompatible to fix the limited C API and the stable ABI in CPython and develop HPy in parallel.

About the PEP, maybe it would be interesting to explain that it doesn't try to solve all issues at once for practical reasons, and that projects like HPy can be solutions for that.

Victor

Hello!

I'm replying to clarifying a few things about HPy since it was mentioned:

On Mon, Feb 1, 2021 at 12:07 PM Petr Viktorin encukou@gmail.com wrote:

Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API

There are the two core things that HPy addresses. Making PyObject* opaque is the bigger of the two changes, because it affects the majority of API methods. HPy attempts to address both of these issues in a "minimalist" way in the sense that although it is a whole new API, it tries to be the obvious conversion from the old API.

PEP 620 has many things, some of which conflict with the accepted PEP 387. I don't think all of it is a good plan (and I guess that's why it wasn't accepted).

HPy and PEP 620 are largely unrelated. They are inspired by some of the same issues, but are completely different things. HPy can (and in the case of CPython is being) implemented on top of the existing C API, and so doesn't require a PEP.

Perhaps once HPy is more mature, CPython's API will evolve to support HPy a little better or to make porting from the old API to HPy easier, but it's likely still slightly premature to be trying to imagine what those changes are, although generally hiding parts of PyObject* or global state like static types is likely to be helpful.

Yours sincerely, Simon Cross

On Mon, Feb 1, 2021 at 11:06 AM Petr Viktorin encukou@gmail.com wrote:

Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API and stable ABI as global symbols. This prevents making the GIL per-interpreter.

I created https://bugs.python.org/issue40601 to discuss the problem of static types "exposed" (indirectly) in the limited C API.

I wrote an experimental PR to see how we could do that: https://github.com/python/cpython/pull/24146

• Add Py_GetXXXType()
• Replace &XXX_Type with Py_GetXXXType()

These heap types must be created as early as possible, and destroyed as late as possible.

Py_GetXXXType() functions access the current interpreter. Currently, _PyInterpreterState_GET() returns NULL when the GIL is released. It means that accessing types is no longer possible when the GIL is released. I consider that it was never officially supported and so it's not an incompatible change, but some extensions do that and will crash with these changes (we can fix them and document the issue).

Here's how stuff like that can be changed:

1. Formalize, test and document the stable ABI
2. Introduce new API (e.g. PyList_GetListType())
3. Deprecate the old API

I suggest to use something like https://github.com/pythoncapi/pythoncapi_compat to provide new "get" functions to old Python functions (they would access static types) and prepare extension modules for the future incompatible changes, without losing support with old Python versions.

For example: static inline PyTypeObject* Py_GetListType(void) { return &PyList_Type; }

Flag day migration as we did with Python 2 => Python 3 is usually a disaster and annoy everybody.

We should automate the migration as much as we can.

Victor

Night gathers, and now my watch begins. It shall not end until my death.

On Mon, Feb 1, 2021 at 3:41 PM Victor Stinner vstinner@python.org wrote:

...

4. Create abi4, a new, incompatible version of the stable ABI, which doesn't include deprecated stuff. Create a flag to mark extension modules that use it.
5. Make it possible to create subinterpreters with a separate GIL. These can only load abi4 extension modules.

Step 4 introduces a backward incompatible change without much benefit :-)

Step 5 will only be safe once all static types will be removed, which is an incompatible change.

(...)

...

?. In Python 4.0, drop abi3.

I don't see how you can run interpreters in parallel if static types are still in use. In my experience, it immediately crash on various ways: https://bugs.python.org/issue40512#msg383830

Ah I forgot to mention that removing static types from the C API is not enough to run multiple interpreters in parallel. Removing static types would be an incompatible changes with no benefit in the short term. It's the same for the PEP 620.

Other incompatible changes are required. For example, reject imports of extensions which don't implement PEP 489 multi-phase initialization API.

See https://bugs.python.org/issue40512 "Meta issue: per-interpreter GIL" for the current status and future required changes.

I also wrote an article about this project progress and what should be done next: https://vstinner.github.io/isolate-subinterpreters.html

Victor

Night gathers, and now my watch begins. It shall not end until my death.

On 2/1/2021 10:17 AM, M.-A. Lemburg wrote:

On 01.02.2021 15:41, Victor Stinner wrote:

...

My new approach is to do all changes directly in Python master branch (single Python) and slowly introduce incompatible changes. I have a feeling that this will drive away extension writers from Python.

The maintenance burden of having to constantly update the code base to accommodate for changes in the Python C API will be too much for them to handle (I know it's too much for me, which is why I stopped doing extension development).

On top of that, those projects will have to keep compatibility with multiple Python versions, which makes things even more difficult.

...

When too many people complain, I revert the change and help to migrate extensions to the new API. I'm doing that for Py_SET_TYPE() and Py_SET_SIZE() using my pythoncapi_compat project for example (I'm waiting for my Mercurial change to be merged upstream ;-)).

https://github.com/pythoncapi/pythoncapi_compat allows to make existing extensions compatibile with the future incompatible changes, without dropping support for old Python versions. The general plan is:

(a) Introduce a new API and deprecate the old one (b) Run pythoncapi_compat on enough extension modules (and get changes accepted) (c) Once enough extensions are ported, drop the old API

I suggest to have at least one Python release between (a) and (c). For example, I introduced Py_SET_REFCNT() in Python 3.9 and I disallowed "Py_REFCNT(obj) = refcnt;" syntax in Python 3.10.

During (b) phase, we can communicate on the future incompatible changes in What's New in Python X.Y, communicate on capi-sig list and/or python-dev, contact extensions maintainers, etc. Other vendors introduce shims to handle breaking changes (e.g. MS on Windows) or even provide complete emulations (e.g. Apple for macOS).

I believe that's the only reasonable way to avoid another Python 2 to 3 15+ year transition process or losing the crowd to other languages. (*)

Your tool is already going in that direction, but rather than having to run the tool over and over again for every release, which adds testing and verification overhead every single time, it would be better to have something like HPy readily working and then use a tool like pythoncapi_compat to migrate over the code once, inserting a shim between the extension using the old style Python C API and the new HPy Python C API.

That shim could also be had at a higher level by e.g. using Cython as layer between the extension and Python; and using a tool to help migrate towards this approach.

The latter is what I'm currently considering, but have not checked the performance hit such an approach would have for low level data type implementations. Could be that it's not feasible.

I agree with MAL here.

I guess my question is: if we go with "slowly introduce incompatible changes", is there some point (let's call it an inflection point) that we think we can get some performance improvements before we drop the C-API entirely? If there's not, I don't think there's any benefit to a slowly introducing incompatible changes. And if there is, let's identify that inflection point now, so we can sell the benefits. And also let's consider not doing slowly incompatible changes, but rather just save all of the changes up for this inflection point.

Eric

...

I tried to explain my failures and my new approach on C API incompatible changes in an article: https://vstinner.github.io/hide-implementation-details-python-c-api.html (*) Also note that the Python GIL scalability problem is not really that urgent anymore these days. People are moving to distributed computing, splitting workloads across processes, containers, VMs, GPUs and other specialized hardware. Together with async code, the GIL no longer prevents Python applications from scaling easily.

It would still be nice to move away from a global lock, but such a change will also require changes in the extensions, since many are not written in a re-entrant way, so the Python C API modifications to remove the GIL are only part of the solution to free-threading.

What I want to say is that we can take our time to avoid disruption :-)

On 01.02.2021 17:40, Victor Stinner wrote:

...

What I want to say is that we can take our time to avoid disruption :-)

In short, you are against CPython evolutions and consider that CPython is good as it is (it's not worth it to optimize CPython). Well, that's one of the two choices in my previous email.

Sorry, but that's a gross misrepresentation of what I said.

I said that we do have the time to wait for HPy to mature and can then make the switch-over to the new API an easy one step solution, rather than introducing breaking changes with every single Python release. The latter causes churn and harms Python's adoption more than it helps.

Python's success is in large parts built on the ease of using it's C API and the thousands of extensions out there to integrate with other software. People use Python to make other software accessible. That's the key advantage we have over other languages.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Feb 01 2021)

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

On 01.02.2021 18:16, Victor Stinner wrote:

On Mon, Feb 1, 2021 at 6:08 PM M.-A. Lemburg mal@egenix.com wrote:

...

I said that we do have the time to wait for HPy to mature and can then make the switch-over to the new API an easy one step solution, rather than introducing breaking changes with every single Python release. The latter causes churn and harms Python's adoption more than it helps.

As I wrote in a previous email, HPy by itself doesn't solve any CPython design issue, since we want to continue supporting C extension modules written with the C API.

For example, are you ready to drop support for numpy? There is a project to start rewriting some parts of numpy with HPy, but a full conversion will take months or years.

As I said: we have those years to wait. The GIL is not causing people to run away from Python anymore, since compute has moved on and no longer relies on threads for scaling up.

I think that's a very positive development and one we should use for our benefit.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Feb 01 2021)

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

On Mon, Feb 01, 2021 at 06:34:04PM +0100, M.-A. Lemburg wrote:

As I said: we have those years to wait. The GIL is not causing people to run away from Python anymore, since compute has moved on and no longer relies on threads for scaling up.

Where has it moved to? I see people choosing Go and Rust for performance reasons.

-- Senthil

Le 02/02/2021 à 10:14, M.-A. Lemburg a écrit :

On 02.02.2021 00:37, Senthil Kumaran wrote:

...

On Mon, Feb 01, 2021 at 06:34:04PM +0100, M.-A. Lemburg wrote:

...

As I said: we have those years to wait. The GIL is not causing people to run away from Python anymore, since compute has moved on and no longer relies on threads for scaling up.

Where has it moved to? I see people choosing Go and Rust for performance reasons.

Compute has moved on to distributed computing, quoting an earlier email:

""" Also note that the Python GIL scalability problem is not really that urgent anymore these days. People are moving to distributed computing, splitting workloads across processes, containers, VMs, GPUs and other specialized hardware. Together with async code, the GIL no longer prevents Python applications from scaling easily. """

Have a look at Python Dask for example.

Speaking as own of the Dask core developers, the Python GIL is very much a problem for some parts of Dask (ironically, for the part that distributes work over multiple computers, since distributed scheduling is a difficult task that takes a lot of CPU).

See for example https://github.com/dask/distributed/issues/4443

Regards

Antoine.

Le 02/02/2021 à 10:50, M.-A. Lemburg a écrit :

On 02.02.2021 10:27, Antoine Pitrou wrote:

...

Le 02/02/2021 à 10:14, M.-A. Lemburg a écrit :

...

On 02.02.2021 00:37, Senthil Kumaran wrote:

...

On Mon, Feb 01, 2021 at 06:34:04PM +0100, M.-A. Lemburg wrote:

...

As I said: we have those years to wait. The GIL is not causing people to run away from Python anymore, since compute has moved on and no longer relies on threads for scaling up.

Where has it moved to? I see people choosing Go and Rust for performance reasons.

Compute has moved on to distributed computing, quoting an earlier email:

""" Also note that the Python GIL scalability problem is not really that urgent anymore these days. People are moving to distributed computing, splitting workloads across processes, containers, VMs, GPUs and other specialized hardware. Together with async code, the GIL no longer prevents Python applications from scaling easily. """

Have a look at Python Dask for example.

Speaking as own of the Dask core developers, the Python GIL is very much a problem for some parts of Dask (ironically, for the part that distributes work over multiple computers, since distributed scheduling is a difficult task that takes a lot of CPU).

See for example https://github.com/dask/distributed/issues/4443

Reading through that ticket, it looks more like a socket problem than a GIL problem:

https://github.com/dask/distributed/issues/4443#issuecomment-765241186

The concerns in that issue a bit commingled. But the underlying issue is that the main scheduler thread competes for CPU resources with other threads where other tasks are offloaded. The scheduler by design works on shared state and cannot be rewritten to use several processes.

Regards

Antoine.

On 2/1/21 3:41 PM, Victor Stinner wrote:

Hi Petr,

On Mon, Feb 1, 2021 at 11:06 AM Petr Viktorin encukou@gmail.com wrote:

...

I think "introducing incompatible changes on purpose" makes Python worse without much benefit. Has any of the recent incompatible changes actually made Python faster?

Optimizing Python is a hard problem. Look back at the 30 years of CPython optimizations. There were some nice optimizations, but the CPython design is the same for 30 years. If it would be easy, it would already be done. There are big companies would need a faster CPython. Significant speedup (ex: 2x faster) require to deeply change CPython design.

It will only become possible to *start* *experimenting* optimizing CPython when most of the PEP 620 will be implemented. Some people disagree, but so far, I didn't see any concrete working optimization proposed and implemented in CPython (without the PEP 620 being implemented). So everyone is free to have their own "beliefs" in term of optimization possibilities (with or without incompatible C API changes) ;-)

To be clear, the PEP 620 brings zero optimization. It should be done separately. The PEP 620 only prepares CPython for future optimizations.

...

Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API and stable ABI as global symbols. This prevents making the GIL per-interpreter. Here's how stuff like that can be changed:

1. Formalize, test and document the stable ABI
2. Introduce new API (e.g. PyList_GetListType())
3. Deprecate the old API
4. Create abi4, a new, incompatible version of the stable ABI, which doesn't include deprecated stuff. Create a flag to mark extension modules that use it.
5. Make it possible to create subinterpreters with a separate GIL. These can only load abi4 extension modules.

Step 4 introduces a backward incompatible change without much benefit :-)

It does not. All existing modules would continue to work.

Step 5 will only be safe once all static types will be removed, which is an incompatible change.

Again, no – all existing code would continue to work.

I see that I didn't explain one aspect of the idea: there would be two kinds of subinterpreters – one that shares the GIL with the "main" interpreter, and a new opt-in kind that does not share the GIL but only allows loading abi4 modules. I don't think this would be hard to do in practice: GIL must already be shared across thread states belonging to a single interpreter; it doesn't seem hard to have some interpreters sharing the GIL and some not.

For major CPython design changes (like running multiple interpreters in parallel), we *need* to introduce incompatible changes.

Sure, and we have PEP 387 for that. A backwards-incompatible change can be done after at least two years of warnings (and if warnings are not possible, then with an accepted PEP). Incompatible changes should be the last resort, and there needs to be enough time to think about approaches that don't break backwards compatibility.

There are two options: consider that CPython must no longer evolve (and so die slowly as a dead language ;-)), or we should find a way to minimize the number of unhappy people when we introduce incompatible changes :-)

This is a false dichotomy. It is also possible to make progress without breaking so much existing code. It's harder and takes more time, but it's much better for existing software.

...

6. Work with various third-party projects to switch to abi4 so we can start getting its benefits, and improve abi4 along the way.

Over the last years, I proposed different PEP drafts with an opt-in choice for a faster python which is backward incompatible. Nobody liked this option (having two Python). Look at the Python 2 => Python 3 migration and "CPython vs PyPy". "Faster Python" is not enough to migrate the critical mass of users to the "new" Python, the old Python never goes away, and then you have to maintain two Python runtimes instead of one.

My new approach is to do all changes directly in Python master branch (single Python) and slowly introduce incompatible changes. When too many people complain, I revert the change and help to migrate extensions to the new API. I'm doing that for Py_SET_TYPE() and Py_SET_SIZE() using my pythoncapi_compat project for example (I'm waiting for my Mercurial change to be merged upstream ;-)).

https://github.com/pythoncapi/pythoncapi_compat allows to make existing extensions compatibile with the future incompatible changes, without dropping support for old Python versions. The general plan is:

(a) Introduce a new API and deprecate the old one (b) Run pythoncapi_compat on enough extension modules (and get changes accepted) (c) Once enough extensions are ported, drop the old API

I suggest to have at least one Python release between (a) and (c). For example, I introduced Py_SET_REFCNT() in Python 3.9 and I disallowed "Py_REFCNT(obj) = refcnt;" syntax in Python 3.10.

PEP 387 is very clear that this should be at least *two* releases. And only if the old behavior raises warnings for those two releases.

On top of that, I don't plans to drop some API should be made until the old API is *actually* incompatible with a real improvement (e.g. speedup). Until then, any experiments can be done on a subset of extension modules, without requiring everyone in the ecosystem to make changes. If people end up needing to run pythoncapi_compat, it would be much nicer to let them run

During (b) phase, we can communicate on the future incompatible changes in What's New in Python X.Y, communicate on capi-sig list and/or python-dev, contact extensions maintainers, etc.

I tried to explain my failures and my new approach on C API incompatible changes in an article: https://vstinner.github.io/hide-implementation-details-python-c-api.html

...

?. In Python 4.0, drop abi3.

I don't see how you can run interpreters in parallel if static types are still in use. In my experience, it immediately crash on various ways: https://bugs.python.org/issue40512#msg383830

...

PEP 620 has many things, some of which conflict with the accepted PEP 387. I don't think all of it is a good plan (and I guess that's why it wasn't accepted).

The PEP 387 is a generic guidelines how to introduce incompatible changes. It remains possible to introduce incompatible changes differently (ex: without deprecation warning or with no deprecation period) with a dedicated PEP. That's why I wrote PEP 620.

The worrying thing is that PEP 620 is still a Draft, even though most action points from it are "Completed". Why do we even have a PEP approval process if it's not being used before changes are done?

On the items from PEP 620 that are not completed yet:

1. Make structures opaque

2. Avoid functions returning PyObject** I support these ideas, but again, the old API should stay available until the underlying implementation actually changes to make them impossible. Improvements that these functions are preventing can still be tested – they just need to be tested with modules that avoid the old functions.

3. pythoncapi_compat.h header file I don't think this is necessary if we're more careful about backwards compatibility.