PEP 689 – Semi-stable C API tier
Hello, Victor and others are organizing the C-API to make it clearer what's public and what's private (see Victor's file-based stats: https://pythoncapi.readthedocs.io/stats.html) It became clear that we need a tier between public and private C-API. PEP 689 proposes such API, with an opt-in macro and transition period. Please discuss. And if you can think of a better name, that would be great :) The PEP is at: https://peps.python.org/pep-0689/ Thread where this started: https://mail.python.org/archives/list/python-dev@python.org/thread/MA4FQ7G6F... Implementation notes: https://github.com/python/cpython/issues/91744 Here's the current version for easy quoting: ---------- PEP: 689 Title: Semi-stable C API tier Author: Petr Viktorin <encukou@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 523 Created: 22-Apr-2022 Python-Version: 3.11 Abstract ======== Some functions and types of the C-API are designated *semi-stable*, meaning that they will not change in patch (bugfix/security) releases, but may change between minor releases (e.g. between 3.11 and 3.12) without deprecation warnings. Motivation & Rationale ====================== The Python C-API is currently divided into `three tiers <https://devguide.python.org/c-api/>`__: - Limited API, with high compatibility expectations - Public API, which follows the :pep:`backwards compatibility policy <387>`, and requires deprecation warnings before changes - Private (internal) API, which can change at any time. We need a tier between Public and Private to accommodate tools like advanced debuggers and JIT compilers, which need access to CPython internals but assume the C-API they use does not change in patch releases. Setting Stability Expectations ------------------------------ Currently, there are no guarantees for the internal API -- that is, anything that requires ``Py_BUILD_CORE`` or is named with a leading underscore. This API can change without warning at any time, and code that uses it is pinned to a specific build of Python. However, in practice, even the internal API usually happens to be stable in patch releases: - Some CPython core developers take this as an an unwritten rule. - Patch releases only contain bugfixes, which are unlikely to change the API. Semi-stable API will make the stability expectations more explicit. It will also hopefully encourage existing users of the private API to reach out to python-dev, so we can expose, standardize and test an API for some of their use cases. Reserving underscores for Private API ------------------------------------- :pep:`523` introduced functions for use by debuggers and JIT compilers, which are stable only across minor releases. The functions names have leading underscores to suggest their limited stability. However, leading underscores usually mark *fully private* API. CPython developers familiar with the “underscore means internal” convention are unlikely to check if underscored functions they are changing are documented and used outside CPython itself. This proposal brings us a bit closer to reserving underscores only for truly private, unstable, hands-off API. Warning about API that is changed often --------------------------------------- The ``PyCode_New()`` family is an example of functions that are documented as unstable, and also often change in practice. Moving it to the semi-stable tier will make its status obvious even to people who don't read the docs carefully enough, and will make it hard to use accidentally. Changes during the Beta period ------------------------------ Since the API itself can change continuously up until Beta 1 (feature freeze) of a minor version, major users of this API are unlikely to test Alpha releases and provide feedback. It is very difficult to determine what needs to be exposed as semi-stable. Additions to the semi-stable tier will count as *stabilization*, and will be allowed up to Release Candidate 1. Specification ============= Several functions and types (“APIs”) will be moved to a new *semi-stable* tier. They will be expected to stay stable across patch releases, but may change or be removed without warning in minor releases (3.x.0), including Alpha and Beta releases of 3.x.0. When they change significantly, code that uses them should no longer compile (e.g. arguments should be added/removed, or a function should be renamed, but the semantic meaning of an argument should not change). Their definitions will be moved to a new directory, ``Include/semistable/``, and will be included from ``Python.h``. From Python 3.12 on, these APIs will only be usable when the ``Py_USING_SEMI_STABLE_API`` macro is defined. CPython will only define the macro for building CPython itself (``Py_BUILD_CORE``). To make transition to semi-stable API easier, in Python 3.11 the APIs will be available without ``Py_USING_SEMI_STABLE_API`` defined. In this case, using them will generate a deprecation warning on compilers that support ``Py_DEPRECATED``. A similar deprecation period will be used when making more APIs semi-stable in the future: - When moving from public API, the deprecation period should follow Python's backwards compatibility policy (currently, it should last at least two releases). - When moving from public API that is documented as unstable, the deprecation period can only last one release. - When moving from private API or adding new API, no deprecation period is necessary. Leading underscores will be removed from the names of the moved APIs. The old underscored name of a renamed API will be available (as an alias using ``#define``) at least until that API changes. The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be documented, and documentation of each semi-stable API will be updated. Adjustments during Beta periods ------------------------------- New APIs can be added to the semi-stable tier, and private APIs can be moved to it, up to the first release candidate of a new minor version. Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta period. In the Beta period, no API may be moved to more private tier, e.g. what is public in Beta 1 must stay public until the final release. Initial semi-stable API ----------------------- The following API will initially be semi-stable. The set may be adjusted for 3.11. Code object constructors: - ``PyCode_New()`` - ``PyCode_NewWithPosOnlyArgs()`` Frame evaluation API (PEP 523): - ``_PyFrameEvalFunction`` - ``_PyInterpreterState_GetEvalFrameFunc()`` - ``_PyInterpreterState_SetEvalFrameFunc()`` - ``_PyEval_RequestCodeExtraIndex()`` - ``_PyCode_GetExtra()`` - ``_PyCode_SetExtra()`` - ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct) - ``_PyFrame_GetFrameObject`` - ``PyEval_EvalFrameDefault`` (new function that calls ``_PyEval_EvalFrameDefault``, but takes ``PyFrameObject`` rather than ``_PyInterpreterFrame``) (Leading underscores will be removed as mentioned above.) Backwards Compatibility ======================= The C API backwards compatibility story will be made clearer. How to Teach This ================= The changes affect advanced C programmers, who should consult the updated reference documentation, devguide and/or What's New document·. Reference Implementation ======================== https://github.com/python/cpython/issues/91744 Rejected Ideas ============== It might be good to add a similar tier in the Python (not C) API, e.g. for ``types.CodeType``. However, the opt-in mechanism would need to be different (if any). This is outside the scope of the PEP. Open Issues =========== - “Semi-stable” is not a perfect name. - The exact set of exposed API may change. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Ok, let me start with the serious business: API name. I'm not comfortable with "semi-stable". Python already has a "limited API" and a "stable ABI". Just by its name, it's unclear what "semi-stable" means. Honestly, I would be more comfortable with the name: "unstable API". It would be clear that the API *can* change often. People who want to know exactly the backward compatibility warranties can dig into the API documentation to learn more about it. "Unstable API" is also the name the Guido proposed for PyCode_New() last year: * Proposal: declare "unstable APIs" https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM... * Making code object APIs unstable https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY... Victor
On 29. 04. 22 16:32, Victor Stinner wrote:
Ok, let me start with the serious business: API name.
I'm not comfortable with "semi-stable". Python already has a "limited API" and a "stable ABI". Just by its name, it's unclear what "semi-stable" means.
Honestly, I would be more comfortable with the name: "unstable API". It would be clear that the API *can* change often. People who want to know exactly the backward compatibility warranties can dig into the API documentation to learn more about it.
"Unstable API" is also the name the Guido proposed for PyCode_New() last year:
* Proposal: declare "unstable APIs" https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM... * Making code object APIs unstable https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY...
Victor
Nick Coghlan argued against that term:
"unstable" is the wrong term. We already have an unstable API tier: the internal API, which can change even in maintenance releases. The value of the new tier is that it is "semi stable": stable in maintenance releases, unstable in feature releases.
— https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV... But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote:
Ok, let me start with the serious business: API name.
I'm not comfortable with "semi-stable". Python already has a "limited API" and a "stable ABI". Just by its name, it's unclear what "semi-stable" means.
Honestly, I would be more comfortable with the name: "unstable API". It would be clear that the API *can* change often. People who want to know exactly the backward compatibility warranties can dig into the API documentation to learn more about it.
"Unstable API" is also the name the Guido proposed for PyCode_New() last year:
* Proposal: declare "unstable APIs"
https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM...
* Making code object APIs unstable
https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY...
Victor
Nick Coghlan argued against that term:
"unstable" is the wrong term. We already have an unstable API tier: the internal API, which can change even in maintenance releases. The value of the new tier is that it is "semi stable": stable in maintenance releases, unstable in feature releases.
—
https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV...
But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core. So let's please go with "unstable". -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 2022-04-29 18:02, Guido van Rossum wrote:
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote: > Ok, let me start with the serious business: API name. > > I'm not comfortable with "semi-stable". Python already has a "limited > API" and a "stable ABI". Just by its name, it's unclear what > "semi-stable" means. > > Honestly, I would be more comfortable with the name: "unstable API". > It would be clear that the API *can* change often. People who want to > know exactly the backward compatibility warranties can dig into the > API documentation to learn more about it. > > "Unstable API" is also the name the Guido proposed for PyCode_New() last year: > > * Proposal: declare "unstable APIs" > https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM... <https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM...> > * Making code object APIs unstable > https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY... <https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY...> > > Victor
Nick Coghlan argued against that term:
> "unstable" is the wrong term. We already have an unstable API tier: the > internal API, which can change even in maintenance releases. The value of > the new tier is that it is "semi stable": stable in maintenance releases, > unstable in feature releases.
— https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV... <https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV...>
But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
I was going to suggest "metastable". Too late? :-)
MRAB writes:
On 2022-04-29 18:02, Guido van Rossum wrote:
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote: > Ok, let me start with the serious business: API name. > > I'm not comfortable with "semi-stable". Python already has a "limited > API" and a "stable ABI". Just by its name, it's unclear what > "semi-stable" means. > > Honestly, I would be more comfortable with the name: "unstable API". > It would be clear that the API *can* change often. People who want to > know exactly the backward compatibility warranties can dig into the > API documentation to learn more about it. > > "Unstable API" is also the name the Guido proposed for PyCode_New() last year: > > * Proposal: declare "unstable APIs" > https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM... <https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM...> > * Making code object APIs unstable > https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY... <https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY...> > > Victor
Nick Coghlan argued against that term:
> "unstable" is the wrong term. We already have an unstable API tier: the > internal API, which can change even in maintenance releases. The value of > the new tier is that it is "semi stable": stable in maintenance releases, > unstable in feature releases.
— https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV... <https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV...>
But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
I was going to suggest "metastable". Too late? :-) A Bikeshedding to a new level! --+
On 4/29/2022 11:42 AM, Stephen J. Turnbull wrote:
MRAB writes:
On 2022-04-29 18:02, Guido van Rossum wrote:
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote: > Ok, let me start with the serious business: API name. > > I'm not comfortable with "semi-stable". Python already has a "limited > API" and a "stable ABI". Just by its name, it's unclear what > "semi-stable" means. > > Honestly, I would be more comfortable with the name: "unstable API". > It would be clear that the API *can* change often. People who want to > know exactly the backward compatibility warranties can dig into the > API documentation to learn more about it. > > "Unstable API" is also the name the Guido proposed for PyCode_New() last year: > > * Proposal: declare "unstable APIs" > https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM... <https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM...> > * Making code object APIs unstable > https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY... <https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY...> > > Victor
Nick Coghlan argued against that term:
> "unstable" is the wrong term. We already have an unstable API tier: the > internal API, which can change even in maintenance releases. The value of > the new tier is that it is "semi stable": stable in maintenance releases, > unstable in feature releases.
— https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV... <https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV...>
But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
I was going to suggest "metastable". Too late? :-) A Bikeshedding to a new level! --+
Metabikeshedding...
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CGNNVU7C... Code of Conduct: http://python.org/psf/codeofconduct/
On 2022-04-30 03:17, Greg Ewing wrote:
On 30/04/22 5:25 am, MRAB wrote:
I was going to suggest "metastable". Too late? :-)
What, the API is balanced on a knife edge and likely to collapse into something else if you sneeze too hard?
There's a possibility that the universe might be metastable, so a metastable API might not be that big a deal.
On Sat, 30 Apr 2022, 3:02 am Guido van Rossum, <guido@python.org> wrote:
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
While I've advocated for semi-stable in previous threads, I now agree the pragmatic arguments for "unstable" hold up well enough to make the simpler term the better choice: * no question around using a hyphen or not * "unstable public C API" is sufficient to distinguish the new tier from Py_BUILD_CORE's completely unstable internal API The risks of misinterpretation are also low: * external users that need one of these APIs will presumably be invested enough to actually check the stability expectations in the docs * core devs will have regression tests to remind us that the published unstable APIs aren't allowed to change after beta 1 Cheers, Nick.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 29. 04. 22 19:02, Guido van Rossum wrote:
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote: > Ok, let me start with the serious business: API name. > > I'm not comfortable with "semi-stable". Python already has a "limited > API" and a "stable ABI". Just by its name, it's unclear what > "semi-stable" means. [...] Nick Coghlan argued against that term: [...] But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
Thanks, you worded that perfectly! Alright, the PEP now uses “unstable” rather than “semi-stable”. And I don't see any issues with the technical details, so I'll see if it can still get into Python 3.11. Hopefully Pablo agrees as the Release Manager. Thanks for the discussion, everyone!
On Wed, May 4, 2022, 04:11 Petr Viktorin <encukou@gmail.com> wrote:
On 29. 04. 22 19:02, Guido van Rossum wrote:
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote: > Ok, let me start with the serious business: API name. > > I'm not comfortable with "semi-stable". Python already has a "limited > API" and a "stable ABI". Just by its name, it's unclear what > "semi-stable" means. [...] Nick Coghlan argued against that term: [...] But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
Thanks, you worded that perfectly!
Alright, the PEP now uses “unstable” rather than “semi-stable”. And I don't see any issues with the technical details, so I'll see if it can still get into Python 3.11. Hopefully Pablo agrees as the Release Manager. Thanks for the discussion, everyone!
I've already brought this up to Petr directly, but I would greatly prefer new unstable API functions have leading underscores, and that existing functions being moved to the unstable API are _not_ renamed. Renaming existing functions means a lot of unnecessary code churn. It looks like we have more _-prefixed unstable functions than not, but I don't think the churn is worth renaming the currently public ones. Leading underscores for unstable API functions (that aren't currently public) means we keep the widely assumed guarantee that Py*/PY* are part of the public API. The Py_USING_UNSTABLE_API define is per-file, not per symbol/use, so I would rather not open the door to unintended or unaware use of unstable APIs. By giving the functions the leading underscore, we're forcing people to think about -- or check the documentation -- whether the specific function is okay to use. The unstable API is intended for specific use-cases, and I think it's preferable to put the burden of figuring out if a _Py/_PY* symbol is acceptable for them to use, rather than putting the burden of figuring out if a Py/PY* symbol is acceptable up use on _everyone else_. _______________________________________________
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L6IGXJ5O... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, May 5, 2022 at 5:35 AM Thomas Wouters <thomaswout@gmail.com> wrote: [...]
I've already brought this up to Petr directly, but I would greatly prefer new unstable API functions have leading underscores, and that existing functions being moved to the unstable API are _not_ renamed.
Renaming existing functions means a lot of unnecessary code churn. It looks like we have more _-prefixed unstable functions than not, but I don't think the churn is worth renaming the currently public ones.
Leading underscores for unstable API functions (that aren't currently public) means we keep the widely assumed guarantee that Py*/PY* are part of the public API. The Py_USING_UNSTABLE_API define is per-file, not per symbol/use, so I would rather not open the door to unintended or unaware use of unstable APIs. By giving the functions the leading underscore, we're forcing people to think about -- or check the documentation -- whether the specific function is okay to use.
The unstable API is intended for specific use-cases, and I think it's preferable to put the burden of figuring out if a _Py/_PY* symbol is acceptable for them to use, rather than putting the burden of figuring out if a Py/PY* symbol is acceptable up use on _everyone else_.
I don't think that's necessary. I still see the unstable API as public: it needs more maintenance, but it's not particularly dangerous to use it. I think defining Py_USING_UNSTABLE_API once is quite enough.
Hi, Unfortunately my newborn child is taking up even more time than I expected, so I won't be able to get the unstable API tier into Python 3.11 even if it's accepted today. (If anyone else would like to try, feel free to add yourself as Author, make any changes necessary and talk to Pablo (the RM) and the SC.) IMO it's fine to delay formalizing this until 3.12, so we can have a proper discussion. However, if someone has the time, it would be nice to ensure the use cases from PEP 523 are possible with the 3.11 API. Specifically, AFAIK, struct _PyInterpreterFrame needs to be exposed (as an incomplete, opaque struct) to make _PyFrameEvalFunction usable, _PyFrame_GetFrameObject needs to be exposed, and a PyEval_EvalFrameDefault that takes PyFrameObject should be added. Also, the What's New currently says “See PEP 523 for more details of how to use [_PyFrameEvalFunction]”, which isn't very helpful. On Thu, May 5, 2022 at 1:55 PM Petr Viktorin <encukou@gmail.com> wrote:
On Thu, May 5, 2022 at 5:35 AM Thomas Wouters <thomaswout@gmail.com> wrote: [...]
I've already brought this up to Petr directly, but I would greatly prefer new unstable API functions have leading underscores, and that existing functions being moved to the unstable API are _not_ renamed.
Renaming existing functions means a lot of unnecessary code churn. It looks like we have more _-prefixed unstable functions than not, but I don't think the churn is worth renaming the currently public ones.
Leading underscores for unstable API functions (that aren't currently public) means we keep the widely assumed guarantee that Py*/PY* are part of the public API. The Py_USING_UNSTABLE_API define is per-file, not per symbol/use, so I would rather not open the door to unintended or unaware use of unstable APIs. By giving the functions the leading underscore, we're forcing people to think about -- or check the documentation -- whether the specific function is okay to use.
The unstable API is intended for specific use-cases, and I think it's preferable to put the burden of figuring out if a _Py/_PY* symbol is acceptable for them to use, rather than putting the burden of figuring out if a Py/PY* symbol is acceptable up use on _everyone else_.
I don't think that's necessary. I still see the unstable API as public: it needs more maintenance, but it's not particularly dangerous to use it. I think defining Py_USING_UNSTABLE_API once is quite enough.
On Fri, 6 May 2022 at 22:50, Petr Viktorin <encukou@gmail.com> wrote:
IMO it's fine to delay formalizing this until 3.12, so we can have a proper discussion. However, if someone has the time, it would be nice to ensure the use cases from PEP 523 are possible with the 3.11 API. Specifically, AFAIK, struct _PyInterpreterFrame needs to be exposed (as an incomplete, opaque struct) to make _PyFrameEvalFunction usable, _PyFrame_GetFrameObject needs to be exposed, and a PyEval_EvalFrameDefault that takes PyFrameObject should be added. Also, the What's New currently says “See PEP 523 for more details of how to use [_PyFrameEvalFunction]”, which isn't very helpful.
Given the proximity of beta1, I think affected projects are going to need to define Py_BUILD_CORE for the Python 3.11 series - at least some of them have already been updated to declare that during the alpha cycle to handle the frame API changes. PEP 689 can then be informed by feedback from those projects regarding what breaks for them if they *don't* declare Py_BUILD_CORE. That said, one slightly awkward consequence of this approach is affected projects would end up needing to include CPython's "patchlevel.h" [1] directly, so they can query PY_VERSION_HEX *before* including "Python.h". Something like: #include "patchlevel.h" # CPython's version declaration header #if Py_VERSION_HEX+0 >= 0x030c0000 # CPython 3.12+ #define Py_USING_UNSTABLE_API #elif Py_VERSION_HEX+0 >= 0x030b0000 # CPython 3.11.x #define Py_BUILD_CORE #endif #include "Python.h" So maybe it would be worth asking the SC for an exception to the beta freeze, and allow the unstable API tier to land during the beta period? Cheers, Nick. [1] https://github.com/python/cpython/blob/main/Include/patchlevel.h -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
We discussed having leading underscores for this API tier, and it was decided that a leading underscore was preferred. This did start a discussion, though, about whether we should control API access/opt-in via `#include` by having `.h` files that convey what API the user is opting into, or use `#define` to control what gets exposed via `Python.h`. The general feeling was that the header file idea is ideal, but it is a little extra work to transition to if you want to be compatible with older versions of Python that wouldn't have the header files (Victor's compatibility project could help here). The question for the team is whether separate header files makes sense to others, or would people prefer using `#define` and `Python.h` to control API access/opt-in?
I would love to see header files used for this -- while I know there is a long tradition of feature-flags that must be #defined by the user before #including a header in order to affect what the header exports (or not!), 30 years later I still find that approach pretty unintuitive. But yes, it's going to be a complex transition. On Mon, May 30, 2022 at 12:30 PM Brett Cannon <brett@python.org> wrote:
We discussed having leading underscores for this API tier, and it was decided that a leading underscore was preferred.
This did start a discussion, though, about whether we should control API access/opt-in via `#include` by having `.h` files that convey what API the user is opting into, or use `#define` to control what gets exposed via `Python.h`. The general feeling was that the header file idea is ideal, but it is a little extra work to transition to if you want to be compatible with older versions of Python that wouldn't have the header files (Victor's compatibility project could help here). The question for the team is whether separate header files makes sense to others, or would people prefer using `#define` and `Python.h` to control API access/opt-in? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q5JU5YKG... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Mon, 30 May 2022 12:53:57 -0700 Guido van Rossum <guido@python.org> wrote:
I would love to see header files used for this -- while I know there is a long tradition of feature-flags that must be #defined by the user before #including a header in order to affect what the header exports (or not!), 30 years later I still find that approach pretty unintuitive.
Agreed that #defining a flag before #including a header is a brittle approach. If something else included the header before you set your #define, then include guards can prevent you from seeing any effects. This is a common issue in Windows land with the godawful Windows.h header file. Regards Antoine.
But yes, it's going to be a complex transition.
On Mon, May 30, 2022 at 12:30 PM Brett Cannon <brett@python.org> wrote:
We discussed having leading underscores for this API tier, and it was decided that a leading underscore was preferred.
This did start a discussion, though, about whether we should control API access/opt-in via `#include` by having `.h` files that convey what API the user is opting into, or use `#define` to control what gets exposed via `Python.h`. The general feeling was that the header file idea is ideal, but it is a little extra work to transition to if you want to be compatible with older versions of Python that wouldn't have the header files (Victor's compatibility project could help here). The question for the team is whether separate header files makes sense to others, or would people prefer using `#define` and `Python.h` to control API access/opt-in? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q5JU5YKG... Code of Conduct: http://python.org/psf/codeofconduct/
I prefer separate header files, provided people outside of core always have one (presumably "Python.h") that should be included first and includes enough info to check which headers will be available (i.e. the version defs). Modifying preprocessor definitions for different Python versions, or having to set them before knowing what version is being used, seems more complicated. Cheers, Steve On 5/30/2022 8:26 PM, Brett Cannon wrote:
We discussed having leading underscores for this API tier, and it was decided that a leading underscore was preferred.
This did start a discussion, though, about whether we should control API access/opt-in via `#include` by having `.h` files that convey what API the user is opting into, or use `#define` to control what gets exposed via `Python.h`. The general feeling was that the header file idea is ideal, but it is a little extra work to transition to if you want to be compatible with older versions of Python that wouldn't have the header files (Victor's compatibility project could help here). The question for the team is whether separate header files makes sense to others, or would people prefer using `#define` and `Python.h` to control API access/opt-in?
On Mon, May 30, 2022 at 12:54 PM Steve Dower <steve.dower@python.org> wrote:
I prefer separate header files, provided people outside of core always have one (presumably "Python.h") that should be included first and includes enough info to check which headers will be available (i.e. the version defs).
The idea we were kicking around was e.g. `Python-unstable.h` would be all of the limited API plus the unstable parts, `Python-unlimited.h` would be **everything**, etc. I would expect `Python.h` would continue to be what it is today for compatibility purposes. There wouldn't necessarily be an "always have one" header since these header files would cascade into each other as you opted into more and more unstable APIs (think about this as layers of APIs 😉 and representing each encompassing layer with a header file). This would also let teams set policies of how much instability risk they were willing to take by having CI have an allowlist/blocklist of Python header files. -Brett
Modifying preprocessor definitions for different Python versions, or having to set them before knowing what version is being used, seems more complicated.
Cheers, Steve
On 5/30/2022 8:26 PM, Brett Cannon wrote:
We discussed having leading underscores for this API tier, and it was decided that a leading underscore was preferred.
This did start a discussion, though, about whether we should control API access/opt-in via `#include` by having `.h` files that convey what API the user is opting into, or use `#define` to control what gets exposed via `Python.h`. The general feeling was that the header file idea is ideal, but it is a little extra work to transition to if you want to be compatible with older versions of Python that wouldn't have the header files (Victor's compatibility project could help here). The question for the team is whether separate header files makes sense to others, or would people prefer using `#define` and `Python.h` to control API access/opt-in?
Why you don't simplify with api A,B,C and forth and then follows explanation ofr what is stable, unstable, semi... So forth.... On Wed, May 4, 2022, 6:10 AM Petr Viktorin <encukou@gmail.com> wrote:
On 29. 04. 22 19:02, Guido van Rossum wrote:
On Fri, Apr 29, 2022 at 10:15 AM Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 29. 04. 22 16:32, Victor Stinner wrote: > Ok, let me start with the serious business: API name. > > I'm not comfortable with "semi-stable". Python already has a "limited > API" and a "stable ABI". Just by its name, it's unclear what > "semi-stable" means. [...] Nick Coghlan argued against that term: [...] But I also like “unstable” better than “semi-stable”. Splitting the internals into “private”/“internal” and “unstable” seems reasonable.
I think picking "semi-stable" would be giving in to the OCD nerd in all of us. :-) While perhaps technically less precise, "unstable" is the catchy name with the right association. (And yes, we should keep it stable within bugfix releases, but the name doesn't need to reflect that detail.) The "internal API" isn't an API at all (except for CPython core developers and contributors). The "unstable API" would definitely be an *API* for users outside the core.
So let's please go with "unstable".
Thanks, you worded that perfectly!
Alright, the PEP now uses “unstable” rather than “semi-stable”. And I don't see any issues with the technical details, so I'll see if it can still get into Python 3.11. Hopefully Pablo agrees as the Release Manager. Thanks for the discussion, everyone! _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L6IGXJ5O... Code of Conduct: http://python.org/psf/codeofconduct/
Sasha Kacanski writes:
Why you don't simplify with api A,B,C and forth and then follows explanation ofr what is stable, unstable, semi... So forth....
This is exactly what they're hammering out. It's not easy for several reasons, chief of which is that in each case the boundary is a matter of opinion as to the balance among what is most convenient for the developers of Python itself, the developers of separately distributed C/C++ modules, and for existing modules that were developed before the divisions were set and would need to either be changed or to risk API incompatibility with future versions of Python. The nomenclature also matters, as individual programmers have various ideas about the meaning of terms like "stable", and we want as much agreement as possible that the "stable API" is "stable enough", and so on. If you have specific ideas about which APIs belong where, feel free to bring them forward. But this is not a process that should be rushed nor would anyone benefit from pushing it forward more quickly.
I understand issues and welcome any discussions. For that matter I do not rush to conclusions. I am not expert in C and Python as the rest of the folks on this list but I am pretty good with Python itself. I just suggested naming to be as simple as possible for all relevant API's including full descriptions in the code base regarding stable, semi-stable, unstable and so forth. I do that in my projects with Python libraries I write ... Sorry for intruding and possibly clouding the email thread... Regards, On Wed, Jun 1, 2022, 4:39 AM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
Sasha Kacanski writes:
Why you don't simplify with api A,B,C and forth and then follows explanation ofr what is stable, unstable, semi... So forth....
This is exactly what they're hammering out. It's not easy for several reasons, chief of which is that in each case the boundary is a matter of opinion as to the balance among what is most convenient for the developers of Python itself, the developers of separately distributed C/C++ modules, and for existing modules that were developed before the divisions were set and would need to either be changed or to risk API incompatibility with future versions of Python. The nomenclature also matters, as individual programmers have various ideas about the meaning of terms like "stable", and we want as much agreement as possible that the "stable API" is "stable enough", and so on.
If you have specific ideas about which APIs belong where, feel free to bring them forward. But this is not a process that should be rushed nor would anyone benefit from pushing it forward more quickly.
I think that the main advantage of "unstable" over "semi-stable" is that it's a single word :-D It avoids the really hard question (!) about the separator between "semi" and "stable" ;-) (semistable? semi-stable? semi_stable?). Victor
Rejected Ideas ==============
It might be good to add a similar tier in the Python (not C) API, e.g. for ``types.CodeType``. However, the opt-in mechanism would need to be different (if any). This is outside the scope of the PEP.
For types.CodeType constructor, would it be possible to just a mention in the *documentation* that this API is "unstable"? It would come with a link to definition of the "unstable" C API: explain that it can change in 3.x.y bugfix releases, not not in 3.x.0 releases (major? minor? I never recall how they should be called). For now, I don't think that there is a need to actively remove this API from the "default" Python API and add an opt-in option to get access to these functions. But having a mention just in the documentation would be better than nothing. It seems to be popular complain and request. For example, most of the ast module would fall into this "unstable API". Previous discussions: * Proposal: declare "unstable APIs" https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNM... * Making code object APIs unstable https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESY... On one side, it's important to communicate that the API *can* change in 3.x.0 releases, but also provide some warranties that the API *must not change* in 3.x.y bugfix releases. Victor
participants (13)
-
Antoine Pitrou -
Brett Cannon -
Glenn Linderman -
Greg Ewing -
Guido van Rossum -
MRAB -
Nick Coghlan -
Petr Viktorin -
Sasha Kacanski -
Stephen J. Turnbull -
Steve Dower -
Thomas Wouters -
Victor Stinner