New public C API functions must not steal references or return borrowed references
Hi, A new Include/README.rst file was just added to document the 3 C API provided by CPython: * Include/: Limited C API * Include/cpython/: CPython implementation details * Include/internal/: The internal API I would like to note that *new* public C API functions must no longer steal references or return borrowed references. Don't worry, there is no plan to deprecate or remove existing functions which do that, like PyModule_AddObject() (streal a reference) or PyDict_GetItem() (return a borrowed reference). The policy is only to *add* new functions. IMO for the *internal* C API, it's fine to continue doing that for best performances. Moreover, the limited C API must not expose "implementation details". For example, structure members must not be accessed directly, because most structures are excluded from the limited C API. A function call hiding implementation details is usually better. Here is a copy of the current Include/README.rst file: The Python C API ================ The C API is divided into three sections: 1. ``Include/`` 2. ``Include/cpython/`` 3. ``Include/internal/`` Include: Limited API ==================== ``Include/``, excluding the ``cpython`` and ``internal`` subdirectories, contains the public Limited API (Application Programming Interface). The Limited API is a subset of the C API, designed to guarantee ABI stability across Python 3 versions, and is defined in :pep:`384`. Guidelines for expanding the Limited API: - Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference - Macros should not expose implementation details - Please start a public discussion before expanding the API - Functions or macros with a ``_Py`` prefix do not belong in ``Include/``. It is possible to add a function or macro to the Limited API from a given Python version. For example, to add a function to the Limited API from Python 3.10 and onwards, wrap it with ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``. Include/cpython: CPython implementation details =============================================== ``Include/cpython/`` contains the public API that is excluded from the Limited API and the Stable ABI. Guidelines for expanding the public API: - Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference Include/internal: The internal API ================================== With PyAPI_FUNC or PyAPI_DATA ----------------------------- Functions or structures in ``Include/internal/`` defined with ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are exposed only for specific use cases like debuggers and profilers. With the extern keyword ----------------------- Functions in ``Include/internal/`` defined with the ``extern`` keyword *must not and can not* be used outside the CPython code base. Only built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN`` macro defined) can use such functions. When in doubt, new internal C functions should be defined in ``Include/internal`` using the ``extern`` keyword. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
I’m glad there is more clarity here. And thanks to Erlend Egeberg Aasland for championing the PR! —Guido On Thu, Mar 25, 2021 at 09:30 Victor Stinner <vstinner@python.org> wrote:
Hi,
A new Include/README.rst file was just added to document the 3 C API provided by CPython:
* Include/: Limited C API * Include/cpython/: CPython implementation details * Include/internal/: The internal API
I would like to note that *new* public C API functions must no longer steal references or return borrowed references.
Don't worry, there is no plan to deprecate or remove existing functions which do that, like PyModule_AddObject() (streal a reference) or PyDict_GetItem() (return a borrowed reference). The policy is only to *add* new functions.
IMO for the *internal* C API, it's fine to continue doing that for best performances.
Moreover, the limited C API must not expose "implementation details". For example, structure members must not be accessed directly, because most structures are excluded from the limited C API. A function call hiding implementation details is usually better.
Here is a copy of the current Include/README.rst file:
The Python C API ================
The C API is divided into three sections:
1. ``Include/`` 2. ``Include/cpython/`` 3. ``Include/internal/``
Include: Limited API ====================
``Include/``, excluding the ``cpython`` and ``internal`` subdirectories, contains the public Limited API (Application Programming Interface). The Limited API is a subset of the C API, designed to guarantee ABI stability across Python 3 versions, and is defined in :pep:`384`.
Guidelines for expanding the Limited API:
- Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference - Macros should not expose implementation details - Please start a public discussion before expanding the API - Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.
It is possible to add a function or macro to the Limited API from a given Python version. For example, to add a function to the Limited API from Python 3.10 and onwards, wrap it with ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``.
Include/cpython: CPython implementation details ===============================================
``Include/cpython/`` contains the public API that is excluded from the Limited API and the Stable ABI.
Guidelines for expanding the public API:
- Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference
Include/internal: The internal API ==================================
With PyAPI_FUNC or PyAPI_DATA -----------------------------
Functions or structures in ``Include/internal/`` defined with ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are exposed only for specific use cases like debuggers and profilers.
With the extern keyword -----------------------
Functions in ``Include/internal/`` defined with the ``extern`` keyword *must not and can not* be used outside the CPython code base. Only built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN`` macro defined) can use such functions.
When in doubt, new internal C functions should be defined in ``Include/internal`` using the ``extern`` keyword.
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QKFDGIAF... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Glad to help! Thank you Guido, Antoine, Serhiy, and Victor for your insights and scrutinising :) E On 25 Mar 2021, at 18:15, Guido van Rossum <guido@python.org<mailto:guido@python.org>> wrote: I’m glad there is more clarity here. And thanks to Erlend Egeberg Aasland for championing the PR! —Guido On Thu, Mar 25, 2021 at 09:30 Victor Stinner <vstinner@python.org<mailto:vstinner@python.org>> wrote: Hi, A new Include/README.rst file was just added to document the 3 C API provided by CPython: * Include/: Limited C API * Include/cpython/: CPython implementation details * Include/internal/: The internal API I would like to note that *new* public C API functions must no longer steal references or return borrowed references. Don't worry, there is no plan to deprecate or remove existing functions which do that, like PyModule_AddObject() (streal a reference) or PyDict_GetItem() (return a borrowed reference). The policy is only to *add* new functions. IMO for the *internal* C API, it's fine to continue doing that for best performances. Moreover, the limited C API must not expose "implementation details". For example, structure members must not be accessed directly, because most structures are excluded from the limited C API. A function call hiding implementation details is usually better. Here is a copy of the current Include/README.rst file: The Python C API ================ The C API is divided into three sections: 1. ``Include/`` 2. ``Include/cpython/`` 3. ``Include/internal/`` Include: Limited API ==================== ``Include/``, excluding the ``cpython`` and ``internal`` subdirectories, contains the public Limited API (Application Programming Interface). The Limited API is a subset of the C API, designed to guarantee ABI stability across Python 3 versions, and is defined in :pep:`384`. Guidelines for expanding the Limited API: - Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference - Macros should not expose implementation details - Please start a public discussion before expanding the API - Functions or macros with a ``_Py`` prefix do not belong in ``Include/``. It is possible to add a function or macro to the Limited API from a given Python version. For example, to add a function to the Limited API from Python 3.10 and onwards, wrap it with ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``. Include/cpython: CPython implementation details =============================================== ``Include/cpython/`` contains the public API that is excluded from the Limited API and the Stable ABI. Guidelines for expanding the public API: - Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference Include/internal: The internal API ================================== With PyAPI_FUNC or PyAPI_DATA ----------------------------- Functions or structures in ``Include/internal/`` defined with ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are exposed only for specific use cases like debuggers and profilers. With the extern keyword ----------------------- Functions in ``Include/internal/`` defined with the ``extern`` keyword *must not and can not* be used outside the CPython code base. Only built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN`` macro defined) can use such functions. When in doubt, new internal C functions should be defined in ``Include/internal`` using the ``extern`` keyword. Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org<mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org<mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QKFDGIAF... Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido (mobile) _______________________________________________ Python-Dev mailing list -- python-dev@python.org<mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org<mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6TEXVP6C... Code of Conduct: http://python.org/psf/codeofconduct/
Hi Victor, I'm with you 100% on not returning borrowed references, doing so is just plain dangerous. However, is a blanket ban on stealing references the right thing? Maybe the problem is the term "stealing". The caller is transferring the reference to the callee. In some circumstances it can make a lot of sense to do so, since the caller has probably finished with the reference and the callee needs a new one. Cheers, Mark. On 25/03/2021 4:27 pm, Victor Stinner wrote:
Hi,
A new Include/README.rst file was just added to document the 3 C API provided by CPython:
* Include/: Limited C API * Include/cpython/: CPython implementation details * Include/internal/: The internal API
I would like to note that *new* public C API functions must no longer steal references or return borrowed references.
Don't worry, there is no plan to deprecate or remove existing functions which do that, like PyModule_AddObject() (streal a reference) or PyDict_GetItem() (return a borrowed reference). The policy is only to *add* new functions.
IMO for the *internal* C API, it's fine to continue doing that for best performances.
Moreover, the limited C API must not expose "implementation details". For example, structure members must not be accessed directly, because most structures are excluded from the limited C API. A function call hiding implementation details is usually better.
Here is a copy of the current Include/README.rst file:
The Python C API ================
The C API is divided into three sections:
1. ``Include/`` 2. ``Include/cpython/`` 3. ``Include/internal/``
Include: Limited API ====================
``Include/``, excluding the ``cpython`` and ``internal`` subdirectories, contains the public Limited API (Application Programming Interface). The Limited API is a subset of the C API, designed to guarantee ABI stability across Python 3 versions, and is defined in :pep:`384`.
Guidelines for expanding the Limited API:
- Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference - Macros should not expose implementation details - Please start a public discussion before expanding the API - Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.
It is possible to add a function or macro to the Limited API from a given Python version. For example, to add a function to the Limited API from Python 3.10 and onwards, wrap it with ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``.
Include/cpython: CPython implementation details ===============================================
``Include/cpython/`` contains the public API that is excluded from the Limited API and the Stable ABI.
Guidelines for expanding the public API:
- Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference
Include/internal: The internal API ==================================
With PyAPI_FUNC or PyAPI_DATA -----------------------------
Functions or structures in ``Include/internal/`` defined with ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are exposed only for specific use cases like debuggers and profilers.
With the extern keyword -----------------------
Functions in ``Include/internal/`` defined with the ``extern`` keyword *must not and can not* be used outside the CPython code base. Only built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN`` macro defined) can use such functions.
When in doubt, new internal C functions should be defined in ``Include/internal`` using the ``extern`` keyword.
Victor
On Thu, Mar 25, 2021 at 11:58 AM Mark Shannon <mark@hotpy.org> wrote:
Hi Victor,
I'm with you 100% on not returning borrowed references, doing so is just plain dangerous.
However, is a blanket ban on stealing references the right thing?
Maybe the problem is the term "stealing". The caller is transferring the reference to the callee. In some circumstances it can make a lot of sense to do so, since the caller has probably finished with the reference and the callee needs a new one.
Cheers, Mark.
When was the last time a non-internal API that transferred references added? I suggest keeping the restriction on new APIs in place until we actually find a situation where we think we "need" one outside of Include/internal/ to help force the discussion as to why that needs to be public. -gps On 25/03/2021 4:27 pm, Victor Stinner wrote:
Hi,
A new Include/README.rst file was just added to document the 3 C API provided by CPython:
* Include/: Limited C API * Include/cpython/: CPython implementation details * Include/internal/: The internal API
I would like to note that *new* public C API functions must no longer steal references or return borrowed references.
Don't worry, there is no plan to deprecate or remove existing functions which do that, like PyModule_AddObject() (streal a reference) or PyDict_GetItem() (return a borrowed reference). The policy is only to *add* new functions.
IMO for the *internal* C API, it's fine to continue doing that for best performances.
Moreover, the limited C API must not expose "implementation details". For example, structure members must not be accessed directly, because most structures are excluded from the limited C API. A function call hiding implementation details is usually better.
Here is a copy of the current Include/README.rst file:
The Python C API ================
The C API is divided into three sections:
1. ``Include/`` 2. ``Include/cpython/`` 3. ``Include/internal/``
Include: Limited API ====================
``Include/``, excluding the ``cpython`` and ``internal`` subdirectories, contains the public Limited API (Application Programming Interface). The Limited API is a subset of the C API, designed to guarantee ABI stability across Python 3 versions, and is defined in :pep:`384`.
Guidelines for expanding the Limited API:
- Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference - Macros should not expose implementation details - Please start a public discussion before expanding the API - Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.
It is possible to add a function or macro to the Limited API from a given Python version. For example, to add a function to the Limited API from Python 3.10 and onwards, wrap it with ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000``.
Include/cpython: CPython implementation details ===============================================
``Include/cpython/`` contains the public API that is excluded from the Limited API and the Stable ABI.
Guidelines for expanding the public API:
- Functions *must not* steal references - Functions *must not* return borrowed references - Functions returning references *must* return a strong reference
Include/internal: The internal API ==================================
With PyAPI_FUNC or PyAPI_DATA -----------------------------
Functions or structures in ``Include/internal/`` defined with ``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are exposed only for specific use cases like debuggers and profilers.
With the extern keyword -----------------------
Functions in ``Include/internal/`` defined with the ``extern`` keyword *must not and can not* be used outside the CPython code base. Only built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN`` macro defined) can use such functions.
When in doubt, new internal C functions should be defined in ``Include/internal`` using the ``extern`` keyword.
Victor
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5UURMDSQ... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, 25 Mar 2021 23:37:46 -0700 "Gregory P. Smith" <greg@krypto.org> wrote:
On Thu, Mar 25, 2021 at 11:58 AM Mark Shannon <mark@hotpy.org> wrote:
Hi Victor,
I'm with you 100% on not returning borrowed references, doing so is just plain dangerous.
However, is a blanket ban on stealing references the right thing?
Maybe the problem is the term "stealing". The caller is transferring the reference to the callee. In some circumstances it can make a lot of sense to do so, since the caller has probably finished with the reference and the callee needs a new one.
Cheers, Mark.
When was the last time a non-internal API that transferred references added?
I suggest keeping the restriction on new APIs in place until we actually find a situation where we think we "need" one outside of Include/internal/ to help force the discussion as to why that needs to be public.
+1. Regards Antoine.
On Thu, Mar 25, 2021 at 9:01 PM Mark Shannon <mark@hotpy.org> wrote:
Maybe the problem is the term "stealing". The caller is transferring the reference to the callee. In some circumstances it can make a lot of sense to do so, since the caller has probably finished with the reference and the callee needs a new one.
The problem with both borrowed and stolen references is that they implicitly expose the lifetime of the reference as part of the API. I agree that borrowed references are worse in some sense (there is no way to indicate that one is done with a borrowed reference at all). Stolen references are not great though. As you mention, in theory one can treat the stolen reference as having been DECREF'ed and never use it again, but in practice that's hard to do. When reviewing or writing code, one has to remember precisely which API functions steal a reference. One can no longer reason about whether the reference counting is correct by looking for appropriate DECREF's. It's very hard for tools to help pick up mistakes because the reference was freed -- just not necessarily at the right time. Of course it is hard to avoid these kinds of problems as long as pointers to PyObjects and reference counting are part of the API, but let's not make it more difficult to write correct code unless there is a really big advantage to be had by doing so. I also don't see the rules that Victor is proposing as absolutes. More of a "let's have a big conversation before doing this" kind of rule. - Simon
I (with a lot of help and input from other people) wrote up some of the thinking behind these kinds of API issues for the HPy project -- https://github.com/hpyproject/hpy/wiki/c-api-next-level-changes#what-needs-t... It's written from an HPy point of view and if you've already thought about these things a lot none of it may be new to you and you may have your own opinions, but hopefully it's a good introduction to the issues if one is new to them.
participants (7)
-
Antoine Pitrou -
Erlend Aasland -
Gregory P. Smith -
Guido van Rossum -
Mark Shannon -
Simon Cross -
Victor Stinner