Clarification regarding Stable ABI and _Py_*
Hi Petr, In PEP 384 it is written that no functions starting with an underscore are part of the stable ABI: PEP 384 -- Defining a Stable ABI | Python.org <https://www.python.org/dev/peps/pep-0384/#excluded-functions>
All functions starting with _Py are not available to applications
OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists many functions starting with _Py_, for example _PyObject_GC_Malloc. Then again, that function is not listed in Doc/data/stable_abi.dat. (I didn't check other functions, but maybe there are others.) So is Misc/stable_abi.txt just out of date? Or how can the discrepancy be explained? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 06. 12. 21 20:29, Guido van Rossum wrote:
Hi Petr,
In PEP 384 it is written that no functions starting with an underscore are part of the stable ABI:
PEP 384 -- Defining a Stable ABI | Python.org <https://www.python.org/dev/peps/pep-0384/#excluded-functions>
All functions starting with _Py are not available to applications
OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists many functions starting with _Py_, for example _PyObject_GC_Malloc. Then again, that function is not listed in Doc/data/stable_abi.dat. (I didn't check other functions, but maybe there are others.)
So is Misc/stable_abi.txt just out of date? Or how can the discrepancy be explained?
These are not part of the limited API, so extension authors can't use them in the C source. But they typically are (or have been) called by macros from the limited API. So, they are part of the stable ABI; they need to be exported. Misc/stable_abi.txt says "abi_only" for all of these. They don't show up in the user-facing docs.
On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin <encukou@gmail.com> wrote:
On 06. 12. 21 20:29, Guido van Rossum wrote:
Hi Petr,
In PEP 384 it is written that no functions starting with an underscore are part of the stable ABI:
PEP 384 -- Defining a Stable ABI | Python.org <https://www.python.org/dev/peps/pep-0384/#excluded-functions>
All functions starting with _Py are not available to applications
OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists many functions starting with _Py_, for example _PyObject_GC_Malloc. Then again, that function is not listed in Doc/data/stable_abi.dat. (I didn't check other functions, but maybe there are others.)
So is Misc/stable_abi.txt just out of date? Or how can the discrepancy be explained?
These are not part of the limited API, so extension authors can't use them in the C source. But they typically are (or have been) called by macros from the limited API. So, they are part of the stable ABI; they need to be exported.
Misc/stable_abi.txt says "abi_only" for all of these. They don't show up in the user-facing docs.
Thanks, that helps. It's too bad that there's no comment at the top explaining the format (in fact it appears to discourage reading the file?). Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt in https://github.com/python/cpython/pull/29879 Is that allowed? If it's being used by a macro it means code using that macro will fail unless recompiled for 3.11. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Mon, Dec 6, 2021 at 9:54 PM Guido van Rossum <guido@python.org> wrote:
Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt
In Python 3.2, _PyObject_GC_Malloc() is implemented as a function. PyObject_GC_New() macro calls _PyObject_GC_New() function. Internally, PyType_GenericAlloc() and _PyObject_GC_New() call _PyObject_GC_Malloc(). => IMO _PyObject_GC_Malloc() doesn't need to be part of Python 3.2 stable ABI In Python 3.11, _PyObject_GC_Malloc() is still implemented as a function. It's not used in the C API header files, it's only used in the implementation of other functions. => IMO again, IMO _PyObject_GC_Malloc() doesn't need to be part of Python 3.11 stable ABI It would be nice if someone could double check my analysis. In Python 3.7, there was still "#undef _PyObject_GC_Malloc" in Modules/gcmodule.c with the comment: /* extension modules might be compiled with GC support so these functions must always be available */ This code comes from an old commit made for ABI compatibility with... Python 2.2! https://github.com/python/cpython/commit/fec4eb1be1bb193ab3db0645a2c084c3486... --- commit fec4eb1be1bb193ab3db0645a2c084c34868401c Author: Neil Schemenauer <nascheme@enme.ucalgary.ca> Date: Fri Apr 12 02:41:03 2002 +0000 Allow PyObject_Del to be used as a function designator. Provide binary compatibility function. Make PyObject_GC_Track and PyObject_GC_UnTrack functions instead of trivial macros wrapping functions. Provide binary compatibility functions. --- I don't think that Python 3.2 needs to provide ABI compatibility with Python 2.2. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 06. 12. 21 21:50, Guido van Rossum wrote:
On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 06. 12. 21 20:29, Guido van Rossum wrote: > Hi Petr, > > In PEP 384 it is written that no functions starting with an underscore > are part of the stable ABI: > > PEP 384 -- Defining a Stable ABI | Python.org > <https://www.python.org/dev/peps/pep-0384/#excluded-functions <https://www.python.org/dev/peps/pep-0384/#excluded-functions>> > > All functions starting with _Py are not available to applications > > OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists > many functions starting with _Py_, for example _PyObject_GC_Malloc. Then > again, that function is not listed in Doc/data/stable_abi.dat. (I didn't > check other functions, but maybe there are others.) > > So is Misc/stable_abi.txt just out of date? Or how can the discrepancy > be explained?
These are not part of the limited API, so extension authors can't use them in the C source. But they typically are (or have been) called by macros from the limited API. So, they are part of the stable ABI; they need to be exported.
Misc/stable_abi.txt says "abi_only" for all of these. They don't show up in the user-facing docs.
Thanks, that helps. It's too bad that there's no comment at the top explaining the format (in fact it appears to discourage reading the file?).
You can read it, but I want to discourage people from relying on the format: Tools/scripts/stable_abi.py should be the only consumer. I will add a comment though.
Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt in https://github.com/python/cpython/pull/29879 <https://github.com/python/cpython/pull/29879> Is that allowed? If it's being used by a macro it means code using that macro will fail unless recompiled for 3.11.
Generally, that's not allowed. In this particular case, Victor's analysis is right: if you trawl through the history from 3.2 on, you can see that you can't call _PyObject_GC_Malloc via macros in the limited API. So yes, this one can be removed. I'll also note that removing things that are "allowed" to go is not nice to people who relied on PEP 384, which says that defining Py_LIMITED_API "will hide all definitions that are not part of the ABI" -- even though that's incompatible with the part where it says "All functions starting with _Py are not available to applications". PEP 384 is a historical document, but before 3.10 it was the best available documentation. PEP 652 sort of changed the rules mid-course (ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility). But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it. Go ahead and remove it.
On Tue, Dec 7, 2021 at 12:58 AM Petr Viktorin <encukou@gmail.com> wrote:
On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 06. 12. 21 20:29, Guido van Rossum wrote: > Hi Petr, > > In PEP 384 it is written that no functions starting with an underscore > are part of the stable ABI: > > PEP 384 -- Defining a Stable ABI | Python.org > <https://www.python.org/dev/peps/pep-0384/#excluded-functions <https://www.python.org/dev/peps/pep-0384/#excluded-functions>> > > All functions starting with _Py are not available to applications > > OTOH there's a data file in the repo, Misc/stabe_abi.txt, which lists > many functions starting with _Py_, for example _PyObject_GC_Malloc. Then > again, that function is not listed in Doc/data/stable_abi.dat. (I didn't > check other functions, but maybe there are others.) > > So is Misc/stable_abi.txt just out of date? Or how can the discrepancy > be explained?
These are not part of the limited API, so extension authors can't use them in the C source. But they typically are (or have been) called by macros from the limited API. So, they are part of the stable ABI;
On 06. 12. 21 21:50, Guido van Rossum wrote: they
need to be exported.
Misc/stable_abi.txt says "abi_only" for all of these. They don't show up in the user-facing docs.
Thanks, that helps. It's too bad that there's no comment at the top explaining the format (in fact it appears to discourage reading the
file?).
You can read it, but I want to discourage people from relying on the format: Tools/scripts/stable_abi.py should be the only consumer. I will add a comment though.
I don't mind that the format might change. But I feel that I should be able to understand the current format so I can know how to maintain the document without using cargo-culting (and trying to understand what the tool does is not an option). So thanks for adding the comment.
Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc from stable_abi.txt in https://github.com/python/cpython/pull/29879 <https://github.com/python/cpython/pull/29879> Is that allowed? If it's being used by a macro it means code using that macro will fail unless recompiled for 3.11.
Generally, that's not allowed. In this particular case, Victor's analysis is right: if you trawl through the history from 3.2 on, you can see that you can't call _PyObject_GC_Malloc via macros in the limited API. So yes, this one can be removed.
Okay, that's very subtle, so thanks for confirming.
I'll also note that removing things that are "allowed" to go is not nice to people who relied on PEP 384, which says that defining Py_LIMITED_API "will hide all definitions that are not part of the ABI" -- even though that's incompatible with the part where it says "All functions starting with _Py are not available to applications".
I don't actually really follow what you are trying to say here. Probably because I've never paid much attention to PEP 384. I guess the API is confusing because the "right" way to do it (having to define some symbol to *expose* extra stuff rather than to *hide* stuff) was not possible for backwards compatibility reasons. But the extra negative will forever make this confusing. Also, "All functions starting with _Py are not available" sounds like a clumsy way to say "No functions starting with _Py are available" (and you left out whether Py_LIMITED_API affects that availability, whether it was intended to affect it, whether it did in practice affect it in all cases, etc. I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.
PEP 384 is a historical document, but before 3.10 it was the best available documentation. PEP 652 sort of changed the rules mid-course (ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility).
But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it. Go ahead and remove it.
It's gone. Mark landed his change earlier today. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 07/12/2021 19.28, Guido van Rossum wrote:
I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.
The stable ABI is useful for Python packages that ship binary wheels. Take PyCA cryptography [1] as an example. Alex and Paul already build, upload, and ship 12 abi3 wheels for each release and combinations of CPU arch, platform, and libc ABI. Without a stable ABI they would have to create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The number will only increase over time. Python 3.6 is very common on LTS/Enterprise Linux distros. If the current stable ABI makes performance improvements too complex then we should consider to define a new stable ABI with less symbols. Christian [1] https://pypi.org/project/cryptography/#files
On Tue, Dec 7, 2021 at 11:02 AM Christian Heimes <christian@python.org> wrote:
On 07/12/2021 19.28, Guido van Rossum wrote:
I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.
The stable ABI is useful for Python packages that ship binary wheels.
Take PyCA cryptography [1] as an example. Alex and Paul already build, upload, and ship 12 abi3 wheels for each release and combinations of CPU arch, platform, and libc ABI. Without a stable ABI they would have to create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The number will only increase over time. Python 3.6 is very common on LTS/Enterprise Linux distros.
Thanks, that's a very useful example.
If the current stable ABI makes performance improvements too complex then we should consider to define a new stable ABI with less symbols.
But then we will run into backwards compatibility concerns. Suppose we want to delete *one* functions from the stable ABI. How many releases do we have to wait before we can actually delete (as opposed to just deprecate) it? It sounds like you're saying it would take 5 releases, i.e. if we deprecate it in 3.11, we can delete it in 3.16. It would probably be easier to just not bother with the deprecation. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 07. 12. 21 20:58, Guido van Rossum wrote:
On Tue, Dec 7, 2021 at 11:02 AM Christian Heimes <christian@python.org <mailto:christian@python.org>> wrote:
On 07/12/2021 19.28, Guido van Rossum wrote: > I assume it would be insensitive to ask whether we could just get rid of > the stable ABI altogether and focus on the limited API? Just tell > everyone they have to rebuild binary wheels for every Python feature > release. Presumably the deprecation of the stable ABI itself would > require a two-release waiting period. But maybe it would be worth it, > given how subtle it is to do the historical research about even a single > function.
The stable ABI is useful for Python packages that ship binary wheels.
Take PyCA cryptography [1] as an example. Alex and Paul already build, upload, and ship 12 abi3 wheels for each release and combinations of CPU arch, platform, and libc ABI. Without a stable ABI they would have to create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The number will only increase over time. Python 3.6 is very common on LTS/Enterprise Linux distros.
Thanks, that's a very useful example.
If the current stable ABI makes performance improvements too complex then we should consider to define a new stable ABI with less symbols.
But then we will run into backwards compatibility concerns. Suppose we want to delete *one* functions from the stable ABI. How many releases do we have to wait before we can actually delete (as opposed to just deprecate) it? It sounds like you're saying it would take 5 releases, i.e. if we deprecate it in 3.11, we can delete it in 3.16. It would probably be easier to just not bother with the deprecation.
That really depends on what function we'd want to remove. There are usually alternatives to deleting things, but the options depend on the function. If we run out of other options we can make the function always fail or make it leak memory. And the regular backwards compatibility policy gives us 2 years to figure something out :) It is possible that we'll need a new stable ABI for nogil, though, since refcounting is one of the few areas where even the stable ABI uses direct struct access rather than functions.
On Wed, Dec 8, 2021 at 2:23 AM Petr Viktorin <encukou@gmail.com> wrote:
That really depends on what function we'd want to remove. There are usually alternatives to deleting things, but the options depend on the function. If we run out of other options we can make the function always fail or make it leak memory. And the regular backwards compatibility policy gives us 2 years to figure something out :)
What about the various symbols listed in Misc/stable_abi.txt that were accidentally added to the limited API? Can we move toward dropping them from the stable ABI? Most notably, there are quite a few functions listed there that are in the stable ABI but no longer in the limited API. This implies that either they were already deprecated in the limited API (and removed) or they were just removed. At least in some cases they were moved to header files in Include/cpython or Include/internal. So I would not expect extensions to be using them. This subset of those symbols seems entirely appropriate to remove from the stable ABI. Is that okay? Do we even need to bother deprecating them? What about just the "private" ones? For example, I went to change/remove _PyThreadState_Init() (internal API declared in Include/internal/pycore_pystate.h) and found that it is in the stable ABI but not the limited API. It's highly unlikely anyone is using it and plan on double-checking. As far as I can tell, the function was accidentally exposed in the limited API and stable ABI and later removed from the limited API. -eric
On 08. 12. 21 18:06, Eric Snow wrote:
On Wed, Dec 8, 2021 at 2:23 AM Petr Viktorin <encukou@gmail.com> wrote:
That really depends on what function we'd want to remove. There are usually alternatives to deleting things, but the options depend on the function. If we run out of other options we can make the function always fail or make it leak memory. And the regular backwards compatibility policy gives us 2 years to figure something out :)
What about the various symbols listed in Misc/stable_abi.txt that were accidentally added to the limited API? Can we move toward dropping them from the stable ABI?
Most notably, there are quite a few functions listed there that are in the stable ABI but no longer in the limited API. This implies that either they were already deprecated in the limited API (and removed) or they were just removed. At least in some cases they were moved to header files in Include/cpython or Include/internal. So I would not expect extensions to be using them. This subset of those symbols seems entirely appropriate to remove from the stable ABI. Is that okay? Do we even need to bother deprecating them? What about just the "private" ones?
For example, I went to change/remove _PyThreadState_Init() (internal API declared in Include/internal/pycore_pystate.h) and found that it is in the stable ABI but not the limited API. It's highly unlikely anyone is using it and plan on double-checking. As far as I can tell, the function was accidentally exposed in the limited API and stable ABI and later removed from the limited API.
It's possible to remove them just like _PyObject_GC_Malloc was removed, but check that it was unusable (e.g. not called from public macros) in all versions of Python from 3.2 up to now. Could you check if this PR makes things clear? https://github.com/python/devguide/pull/778
On Thu, Dec 9, 2021 at 1:56 AM Petr Viktorin <encukou@gmail.com> wrote:
It's possible to remove them just like _PyObject_GC_Malloc was removed, but check that it was unusable (e.g. not called from public macros) in all versions of Python from 3.2 up to now.
That's what I expected. Thanks.
Could you check if this PR makes things clear? https://github.com/python/devguide/pull/778
Yeah, that text is super helpful. -eric
On 07. 12. 21 19:28, Guido van Rossum wrote:
On Tue, Dec 7, 2021 at 12:58 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 06. 12. 21 21:50, Guido van Rossum wrote: [...] > Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc > from stable_abi.txt in https://github.com/python/cpython/pull/29879 <https://github.com/python/cpython/pull/29879> > <https://github.com/python/cpython/pull/29879 <https://github.com/python/cpython/pull/29879>> Is that allowed? If it's > being used by a macro it means code using that macro will fail unless > recompiled for 3.11.
Generally, that's not allowed. In this particular case, Victor's analysis is right: if you trawl through the history from 3.2 on, you can see that you can't call _PyObject_GC_Malloc via macros in the limited API. So yes, this one can be removed.
Okay, that's very subtle, so thanks for confirming.
I'll also note that removing things that are "allowed" to go is not nice to people who relied on PEP 384, which says that defining Py_LIMITED_API "will hide all definitions that are not part of the ABI" -- even though that's incompatible with the part where it says "All functions starting with _Py are not available to applications".
I don't actually really follow what you are trying to say here. Probably because I've never paid much attention to PEP 384. I guess the API is confusing because the "right" way to do it (having to define some symbol to *expose* extra stuff rather than to *hide* stuff) was not possible for backwards compatibility reasons. But the extra negative will forever make this confusing. Also, "All functions starting with _Py are not available" sounds like a clumsy way to say "No functions starting with _Py are available" (and you left out whether Py_LIMITED_API affects that availability, whether it was intended to affect it, whether it did in practice affect it in all cases, etc.
It's hard to say what PEP 384 was meant to say. My interpretation, PEP 652, is hopefully more consistent. But someone who had a different interpretation of PEP 384 might feel that it broke some promise.
I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.
A honest question wouldn't be insensitive. Thanks for asking! The part where you don't need to rebuild extensions (not just wheels) is the main reason for both Stable ABI and the Limited API. Without it, there might be some reduced API to focus on, but it wouldn't be this feature.
On Wed, 8 Dec 2021 at 12:42, Petr Viktorin <encukou@gmail.com> wrote:
I assume it would be insensitive to ask whether we could just get rid of the stable ABI altogether and focus on the limited API? Just tell everyone they have to rebuild binary wheels for every Python feature release. Presumably the deprecation of the stable ABI itself would require a two-release waiting period. But maybe it would be worth it, given how subtle it is to do the historical research about even a single function.
A honest question wouldn't be insensitive. Thanks for asking!
The part where you don't need to rebuild extensions (not just wheels) is the main reason for both Stable ABI and the Limited API. Without it, there might be some reduced API to focus on, but it wouldn't be this feature.
This I think is the critical point here. I don't follow the API/ABI debate closely, but I do know that the number varieties of C extension builds some projects have to make is a serious overhead. Being able to target the stable ABI and say "this will work for all (essentially) versions of CPython" is a huge benefit, as it reduces the number (and hence the burden) significantly. Not having to amend your source for each CPython release feels to me like a subset of this, and loses a major advantage, so my personal view is that the stable ABI is far more beneficial than the limited API (the limited API is what you fall back to if you're unable to use the stable ABI). Paul
participants (6)
-
Christian Heimes -
Eric Snow -
Guido van Rossum -
Paul Moore -
Petr Viktorin -
Victor Stinner