On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin wrote:

On 06. 12. 21 20:29, Guido van Rossum wrote:

...

Hi Petr,

In PEP 384 it is written that no functions starting with an underscore are part of the stable ABI:

PEP 384 -- Defining a Stable ABI | Python.org https://www.python.org/dev/peps/pep-0384/#excluded-functions

...

All functions starting with _Py are not available to applications

OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists many functions starting with _Py_, for example _PyObject_GC_Malloc. Then again, that function is not listed in Doc/data/stable_abi.dat. (I didn't check other functions, but maybe there are others.)

So is Misc/stable_abi.txt just out of date? Or how can the discrepancy be explained?

These are not part of the limited API, so extension authors can't use them in the C source. But they typically are (or have been) called by macros from the limited API. So, they are part of the stable ABI; they need to be exported.

Misc/stable_abi.txt says "abi_only" for all of these. They don't show up in the user-facing docs.

Thanks, that helps. It's too bad that there's no comment at the top explaining the format (in fact it appears to discourage reading the file?). Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt in https://github.com/python/cpython/pull/29879 Is that allowed? If it's being used by a macro it means code using that macro will fail unless recompiled for 3.11. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On 06. 12. 21 21:50, Guido van Rossum wrote:

On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin mailto:encukou@gmail.com> wrote:

On 06. 12. 21 20:29, Guido van Rossum wrote: > Hi Petr, > > In PEP 384 it is written that no functions starting with an underscore > are part of the stable ABI: > > PEP 384 -- Defining a Stable ABI | Python.org > <https://www.python.org/dev/peps/pep-0384/#excluded-functions https://www.python.org/dev/peps/pep-0384/#excluded-functions> >  > All functions starting with _Py are not available to applications > > OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists > many functions starting with _Py_, for example _PyObject_GC_Malloc. Then > again, that function is not listed in Doc/data/stable_abi.dat. (I didn't > check other functions, but maybe there are others.) > > So is Misc/stable_abi.txt just out of date? Or how can the discrepancy > be explained?

These are not part of the limited API, so extension authors can't use them in the C source. But they typically are (or have been) called by macros from the limited API. So, they are part of the stable ABI; they need to be exported.

Misc/stable_abi.txt says "abi_only" for all of these. They don't show up in the user-facing docs.

Thanks, that helps. It's too bad that there's no comment at the top explaining the format (in fact it appears to discourage reading the file?).

You can read it, but I want to discourage people from relying on the format: Tools/scripts/stable_abi.py should be the only consumer. I will add a comment though.

Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt in https://github.com/python/cpython/pull/29879 https://github.com/python/cpython/pull/29879 Is that allowed? If it's being used by a macro it means code using that macro will fail unless recompiled for 3.11.

Generally, that's not allowed. In this particular case, Victor's analysis is right: if you trawl through the history from 3.2 on, you can see that you can't call _PyObject_GC_Malloc via macros in the limited API. So yes, this one can be removed. I'll also note that removing things that are "allowed" to go is not nice to people who relied on PEP 384, which says that defining Py_LIMITED_API "will hide all definitions that are not part of the ABI" -- even though that's incompatible with the part where it says "All functions starting with _Py are not available to applications". PEP 384 is a historical document, but before 3.10 it was the best available documentation. PEP 652 sort of changed the rules mid-course (ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility). But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it. Go ahead and remove it.

On 07/12/2021 19.28, Guido van Rossum wrote:

I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.

The stable ABI is useful for Python packages that ship binary wheels. Take PyCA cryptography [1] as an example. Alex and Paul already build, upload, and ship 12 abi3 wheels for each release and combinations of CPU arch, platform, and libc ABI. Without a stable ABI they would have to create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The number will only increase over time. Python 3.6 is very common on LTS/Enterprise Linux distros. If the current stable ABI makes performance improvements too complex then we should consider to define a new stable ABI with less symbols. Christian [1] https://pypi.org/project/cryptography/#files

On 07. 12. 21 20:58, Guido van Rossum wrote:

On Tue, Dec 7, 2021 at 11:02 AM Christian Heimes mailto:christian@python.org> wrote:

On 07/12/2021 19.28, Guido van Rossum wrote: > I assume it would be insensitive to ask whether we could just get rid of > the stable ABI altogether and focus on the limited API? Just tell > everyone they have to rebuild binary wheels for every Python feature > release. Presumably the deprecation of the stable ABI itself would > require a two-release waiting period. But maybe it would be worth it, > given how subtle it is to do the historical research about even a single > function.

The stable ABI is useful for Python packages that ship binary wheels.

Take PyCA cryptography [1] as an example. Alex and Paul already build, upload, and ship 12 abi3 wheels for each release and combinations of CPU arch, platform, and libc ABI. Without a stable ABI they would have to create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The number will only increase over time. Python 3.6 is very common on LTS/Enterprise Linux distros.

Thanks, that's a very useful example.

If the current stable ABI makes performance improvements too complex then we should consider to define a new stable ABI with less symbols.

But then we will run into backwards compatibility concerns. Suppose we want to delete *one* functions from the stable ABI. How many releases do we have to wait before we can actually delete (as opposed to just deprecate) it? It sounds like you're saying it would take 5 releases, i.e. if we deprecate it in 3.11, we can delete it in 3.16. It would probably be easier to just not bother with the deprecation.

That really depends on what function we'd want to remove. There are usually alternatives to deleting things, but the options depend on the function. If we run out of other options we can make the function always fail or make it leak memory. And the regular backwards compatibility policy gives us 2 years to figure something out :) It is possible that we'll need a new stable ABI for nogil, though, since refcounting is one of the few areas where even the stable ABI uses direct struct access rather than functions.

On 08. 12. 21 18:06, Eric Snow wrote:

On Wed, Dec 8, 2021 at 2:23 AM Petr Viktorin wrote:

...

That really depends on what function we'd want to remove. There are usually alternatives to deleting things, but the options depend on the function. If we run out of other options we can make the function always fail or make it leak memory. And the regular backwards compatibility policy gives us 2 years to figure something out :)

What about the various symbols listed in Misc/stable_abi.txt that were accidentally added to the limited API? Can we move toward dropping them from the stable ABI?

Most notably, there are quite a few functions listed there that are in the stable ABI but no longer in the limited API. This implies that either they were already deprecated in the limited API (and removed) or they were just removed. At least in some cases they were moved to header files in Include/cpython or Include/internal. So I would not expect extensions to be using them. This subset of those symbols seems entirely appropriate to remove from the stable ABI. Is that okay? Do we even need to bother deprecating them? What about just the "private" ones?

For example, I went to change/remove _PyThreadState_Init() (internal API declared in Include/internal/pycore_pystate.h) and found that it is in the stable ABI but not the limited API. It's highly unlikely anyone is using it and plan on double-checking. As far as I can tell, the function was accidentally exposed in the limited API and stable ABI and later removed from the limited API.

It's possible to remove them just like _PyObject_GC_Malloc was removed, but check that it was unusable (e.g. not called from public macros) in all versions of Python from 3.2 up to now. Could you check if this PR makes things clear? https://github.com/python/devguide/pull/778

On 07. 12. 21 19:28, Guido van Rossum wrote:

On Tue, Dec 7, 2021 at 12:58 AM Petr Viktorin mailto:encukou@gmail.com> wrote:

On 06. 12. 21 21:50, Guido van Rossum wrote: [...] > Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc > from stable_abi.txt in https://github.com/python/cpython/pull/29879 https://github.com/python/cpython/pull/29879 > <https://github.com/python/cpython/pull/29879 https://github.com/python/cpython/pull/29879> Is that allowed? If it's > being used by a macro it means code using that macro will fail unless > recompiled for 3.11.

Generally, that's not allowed. In this particular case, Victor's analysis is right: if you trawl through the history from 3.2 on, you can see that you can't call _PyObject_GC_Malloc via macros in the limited API. So yes, this one can be removed.

Okay, that's very subtle, so thanks for confirming.

I'll also note that removing things that are "allowed" to go is not nice to people who relied on PEP 384, which says that defining Py_LIMITED_API "will hide all definitions that are not part of the ABI" -- even though that's incompatible with the part where it says "All functions starting with _Py are not available to applications".

I don't actually really follow what you are trying to say here. Probably because I've never paid much attention to PEP 384. I guess the API is confusing because the "right" way to do it (having to define some symbol to *expose* extra stuff rather than to *hide* stuff) was not possible for backwards compatibility reasons. But the extra negative will forever make this confusing. Also, "All functions starting with _Py are not available" sounds like a clumsy way to say "No functions starting with _Py are available" (and you left out whether Py_LIMITED_API affects that availability, whether it was intended to affect it, whether it did in practice affect it in all cases, etc.

It's hard to say what PEP 384 was meant to say. My interpretation, PEP 652, is hopefully more consistent. But someone who had a different interpretation of PEP 384 might feel that it broke some promise.

I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.

A honest question wouldn't be insensitive. Thanks for asking! The part where you don't need to rebuild extensions (not just wheels) is the main reason for both Stable ABI and the Limited API. Without it, there might be some reduced API to focus on, but it wouldn't be this feature.