Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo We will likely move this into the core of PyTorch closer to release. If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?
On 3/28/2022 5:44 PM, Jason Ansel via Python-Dev wrote:
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo
We will likely move this into the core of PyTorch closer to release.
If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?
I believe that you will just have to add a new or different #include statement. I recommend that you add a comment to the issue, https://bugs.python.org/issue46850, and even more, that you create a branch with the diffs in https://github.com/python/cpython/pull/32052 https://github.com/python/cpython/pull/32054 applied so that you can experiment and verify what I believe. -- Terry Jan Reedy
On 3/28/2022 10:44 PM, Jason Ansel via Python-Dev wrote:
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo
We will likely move this into the core of PyTorch closer to release.
If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?
You'd be able to use it, but if we don't nail down the compatibility guarantees then you might find PyTorch doesn't work properly against 3.11.(n+1) while 3.11.n was fine. Right now, the API is allowed/expected to change between 3.x releases (which normally we don't allow without a deprecation period) but it still has to remain compatible within a single 3.x release. Making it fully internal *without adding a stability guarantee* means it could change more frequently, which you wouldn't be able to handle as an installable package. It's *unlikely* that it'll change that often, because there are still other public interfaces that cannot. But, the plan behind this is to make more stuff internal so that it can be modified more freely, so we may see that rate of change increase. In this world, your best bet is for TorchDynamo to become a full CPython fork. And when that's your best bet, it should set off alarms everywhere ;) But that is what relying on internal APIs implies, and is certainly what people would have to do if we were to remove the existing PEP's interface. (It's what Pyjion was doing before the PEP was written, and it's why Pyjion doesn't have to be a full fork these days.) So you probably want to state explicit support for either keeping the APIs public and slightly-flexible, or making them internal but stable. (Public and stable won't work at all for us, and normal internal won't work at all for you.) Cheers, Steve
On Wed, Mar 30, 2022 at 2:26 AM Steve Dower <steve.dower@python.org> wrote:
Right now, the API is allowed/expected to change between 3.x releases (which normally we don't allow without a deprecation period) but it still has to remain compatible within a single 3.x release. Making it fully internal *without adding a stability guarantee* means it could change more frequently, which you wouldn't be able to handle as an installable package.
It's *unlikely* that it'll change that often, because there are still other public interfaces that cannot. But, the plan behind this is to make more stuff internal so that it can be modified more freely, so we may see that rate of change increase.
Well, my plan is not break these internal C API just for fun. It's only to better "advertise" the separation between the "stable" public C API (backward compatibility warranty) and the "unstable" private/internal C API (can change any time, changes are not documented). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Got it, thanks for the clarifications! Tracking 3.x Python versions is fine. We already need to do that to support things like new bytecodes, and PyTorch supports an explicit list of 3.x Python versions with separate builds for each. Tracking 3.x.y Python versions would be much more painful, and make us need to rethink our approach. So what Steve Downer described as "remain compatible within a single 3.x release", seems like exactly what we want. I support that level of compatibility guarantee. Could we keep that guarantee with this change? Thanks, Jason ________________________________________ From: Victor Stinner <vstinner@python.org> Sent: Wednesday, March 30, 2022 7:56 AM To: Steve Dower Cc: Jason Ansel; python-dev@python.org Subject: Re: [Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API On Wed, Mar 30, 2022 at 2:26 AM Steve Dower <steve.dower@python.org> wrote:
Right now, the API is allowed/expected to change between 3.x releases (which normally we don't allow without a deprecation period) but it still has to remain compatible within a single 3.x release. Making it fully internal *without adding a stability guarantee* means it could change more frequently, which you wouldn't be able to handle as an installable package.
It's *unlikely* that it'll change that often, because there are still other public interfaces that cannot. But, the plan behind this is to make more stuff internal so that it can be modified more freely, so we may see that rate of change increase.
Well, my plan is not break these internal C API just for fun. It's only to better "advertise" the separation between the "stable" public C API (backward compatibility warranty) and the "unstable" private/internal C API (can change any time, changes are not documented). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
I don't think such a guarantee (to not vary internal APIs from 3.x.y to 3.x.(y+1)) has ever before been made in writing, although in practice we've been doing this (more so now than we were in the 2.x timeframe). I think we should not lightly vary internal APIs between bugfix releases, but at the same time I am not sure I want to absolutely guarantee that no internal API ever changes in a bugfix release (because that might prevent fixing certain bugs). So I think we need to officially embrace a category of "unstable public APIs" and set a policy specifically for those, before we can make progress. I'd like the SC to take some initiative here. On Wed, Mar 30, 2022 at 9:34 AM Jason Ansel via Python-Dev < python-dev@python.org> wrote:
Got it, thanks for the clarifications!
Tracking 3.x Python versions is fine. We already need to do that to support things like new bytecodes, and PyTorch supports an explicit list of 3.x Python versions with separate builds for each.
Tracking 3.x.y Python versions would be much more painful, and make us need to rethink our approach.
So what Steve Downer described as "remain compatible within a single 3.x release", seems like exactly what we want. I support that level of compatibility guarantee. Could we keep that guarantee with this change?
Thanks, Jason
________________________________________ From: Victor Stinner <vstinner@python.org> Sent: Wednesday, March 30, 2022 7:56 AM To: Steve Dower Cc: Jason Ansel; python-dev@python.org Subject: Re: [Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Wed, Mar 30, 2022 at 2:26 AM Steve Dower <steve.dower@python.org> wrote:
Right now, the API is allowed/expected to change between 3.x releases (which normally we don't allow without a deprecation period) but it still has to remain compatible within a single 3.x release. Making it fully internal *without adding a stability guarantee* means it could change more frequently, which you wouldn't be able to handle as an installable package.
It's *unlikely* that it'll change that often, because there are still other public interfaces that cannot. But, the plan behind this is to make more stuff internal so that it can be modified more freely, so we may see that rate of change increase.
Well, my plan is not break these internal C API just for fun. It's only to better "advertise" the separation between the "stable" public C API (backward compatibility warranty) and the "unstable" private/internal C API (can change any time, changes are not documented).
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U7M65SSD... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
The internal C API can be used on purpose. But there is no backward compatibility warranty and it can change anytime. In practice, usually it only changes in 3.x.0 releases. For example, these private C API changed in Python 3.9 and Python 3.11 (see my first email in the other PEP 523 thread). To use the internal C API, you have to declare the Py_BUILD_CORE macro and include an internal C API header file. For _PyInterpreterState_SetEvalFrameFunc(), it should be: #ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault Victor On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev <python-dev@python.org> wrote:
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo
We will likely move this into the core of PyTorch closer to release.
If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
In the not so distant past I have proposed to introduce a new category, "Unstable APIs". These are public but are not guaranteed to be backwards compatible in feature releases (though I feel they should remain so in bugfix releases). I'm not sure whether those should have a leading underscore or not. Perhaps (like some other languages do and like maybe we've used in a few places) the name could just include the word "Unstable"? On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner <vstinner@python.org> wrote:
The internal C API can be used on purpose. But there is no backward compatibility warranty and it can change anytime. In practice, usually it only changes in 3.x.0 releases. For example, these private C API changed in Python 3.9 and Python 3.11 (see my first email in the other PEP 523 thread).
To use the internal C API, you have to declare the Py_BUILD_CORE macro and include an internal C API header file. For _PyInterpreterState_SetEvalFrameFunc(), it should be:
#ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault
Victor
On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev <python-dev@python.org> wrote:
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this
proposal may break the next major release of PyTorch.
The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo
We will likely move this into the core of PyTorch closer to release.
If the changed happens, would PyTorch still be able to use the eval
frame API? Or would it prevent from being used entirely?
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQ... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 30. 03. 22 17:42, Guido van Rossum wrote:
In the not so distant past I have proposed to introduce a new category, "Unstable APIs". These are public but are not guaranteed to be backwards compatible in feature releases (though I feel they should remain so in bugfix releases).
I'm not sure whether those should have a leading underscore or not. Perhaps (like some other languages do and like maybe we've used in a few places) the name could just include the word "Unstable"?
IMO, the underscore should mark an API as internal: it can change at any time (though in practice it often doesn't, e.g. to accommodate projects that used it before a policy is written down). This is useful e.g. for macros/static functions that wrap access to something private, where the definition needs to be available but marked "keep off".
On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
The internal C API can be used on purpose. But there is no backward compatibility warranty and it can change anytime. In practice, usually it only changes in 3.x.0 releases. For example, these private C API changed in Python 3.9 and Python 3.11 (see my first email in the other PEP 523 thread).
To use the internal C API, you have to declare the Py_BUILD_CORE macro and include an internal C API header file. For _PyInterpreterState_SetEvalFrameFunc(), it should be:
#ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault
Victor
On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev <python-dev@python.org <mailto:python-dev@python.org>> wrote: > > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. > > The related project is TorchDynamo, which can be found here: > https://github.com/facebookresearch/torchdynamo <https://github.com/facebookresearch/torchdynamo> > > We will likely move this into the core of PyTorch closer to release. > > If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely? > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> > To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> > https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ... <https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ...> > Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQ... <https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQ...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EIK3XLPV... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, 2022-03-30 at 17:51 +0200, Petr Viktorin wrote:
On 30. 03. 22 17:42, Guido van Rossum wrote:
In the not so distant past I have proposed to introduce a new category, "Unstable APIs". These are public but are not guaranteed to be backwards compatible in feature releases (though I feel they should remain so in bugfix releases).
I'm not sure whether those should have a leading underscore or not. Perhaps (like some other languages do and like maybe we've used in a few places) the name could just include the word "Unstable"?
IMO, the underscore should mark an API as internal: it can change at any time (though in practice it often doesn't, e.g. to accommodate projects that used it before a policy is written down).
That is fair, although there are documented underscored ones: https://docs.python.org/3/search.html?q=_Py I suppose that means all bets are off _unless_ it is documented or later adopted as stable API (e.g. `_PyObject_Vectorcall`). With that, the only "not obviously OK" use in NumPy that I am aware of is `_Py_HashDouble` (it seems undocumented). Maybe "unless documented" is just a clear enough distinction in practice. Although, to some degree, I think it would be clearer if symbols that have a realistic chance of changing in bug-fix releases had an additional safe-guard. - Sebastian
This is useful e.g. for macros/static functions that wrap access to something private, where the definition needs to be available but marked "keep off".
On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
The internal C API can be used on purpose. But there is no backward compatibility warranty and it can change anytime. In practice, usually it only changes in 3.x.0 releases. For example, these private C API changed in Python 3.9 and Python 3.11 (see my first email in the other PEP 523 thread).
To use the internal C API, you have to declare the Py_BUILD_CORE macro and include an internal C API header file. For _PyInterpreterState_SetEvalFrameFunc(), it should be:
#ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault
Victor
On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev <python-dev@python.org <mailto:python-dev@python.org>> wrote: > > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch. > > The related project is TorchDynamo, which can be found here: > https://github.com/facebookresearch/torchdynamo <https://github.com/facebookresearch/torchdynamo> > > We will likely move this into the core of PyTorch closer to release. > > If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely? > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> > To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> > https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ... <https://mail.python.org/archives/list/python-dev@python.org/messag e/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/> > Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQ... <https://mail.python.org/archives/list/python-dev@python.org/messag e/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ < http://feministing.com/2015/02/03/how-using-they-as-a-singular-prono un-can-change-the-world/>
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EIK3XLPV... Code of Conduct: http://python.org/psf/codeofconduct/
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MBHN5AR5... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Mar 30, 2022 at 9:26 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
That is fair, although there are documented underscored ones: https://docs.python.org/3/search.html?q=_Py
I suppose that means all bets are off _unless_ it is documented or later adopted as stable API (e.g. `_PyObject_Vectorcall`).
With that, the only "not obviously OK" use in NumPy that I am aware of is `_Py_HashDouble` (it seems undocumented).
Maybe "unless documented" is just a clear enough distinction in practice. Although, to some degree, I think it would be clearer if symbols that have a realistic chance of changing in bug-fix releases had an additional safe-guard.
Since Python 3.7, there is a work-in-progress to (1) better hide internal C API functions and to (2) promote *private* C API functions being used by 3rd party projects as documented and well tested *public* functions. An example of (2) is the addition of float pack/unpack public functions, like PyFloat_Pack8() and PyFloat_Unpack8(): https://docs.python.org/dev/c-api/float.html#pack-and-unpack-functions There were previously known as private _PyFloat_Pack8() and _PyFloat_Unpack8() functions. They are used by a few serialization projects like msgpack. When they were made public, tests were added and the existing comments were converted to documentation and enhanced. I discovered that these private functions were used when I moved them to the internal C API in bpo-46906 (1) and it broke a few projects. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 3/30/2022 4:42 PM, Guido van Rossum wrote:
In the not so distant past I have proposed to introduce a new category, "Unstable APIs". These are public but are not guaranteed to be backwards compatible in feature releases (though I feel they should remain so in bugfix releases).
Agreed. This is definitely a new category, and it seems the only thing we're debating now is whether or not to add/omit the underscore.
I'm not sure whether those should have a leading underscore or not. Perhaps (like some other languages do and like maybe we've used in a few places) the name could just include the word "Unstable"?
I don't think we have "Unstable" anywhere, though we do have "Unsafe" (which is more about threading than stability, so not right for this). But I'd be okay with that as a compromise. I'd prefer to not have public-unstable APIs hidden behind the same preprocessor directive as internal APIs. That's a big switch to throw that may also activate other settings - for example, on Windows we will set the minimum Windows version in our headers if you enable internal APIs, and disable automatic linking of the Python DLL. Easy enough things to work around, but they probably need to be explicitly documented as well if we're going to document public APIs as requiring Py_BUILD_CORE (and I don't want to have to document that kind of stuff). Cheers, Steve
On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum <guido@python.org> wrote:
In the not so distant past I have proposed to introduce a new category, "Unstable APIs". These are public but are not guaranteed to be backwards compatible in feature releases (though I feel they should remain so in bugfix releases).
I'm not sure whether those should have a leading underscore or not. Perhaps (like some other languages do and like maybe we've used in a few places) the name could just include the word "Unstable"?
I recall discussions about PyCode_New(). IMO this API should not be public at all. It leaks way too many implementation details: cell variables, optimization for bytecode offset to line and column numbers, exception table, etc. This API changed often and will continue to change. It's not the right abstraction level. We just exposed the function because it was technically possible and it was convenient since Python consumes its own C API. The internal C API was created to draw a line between what API can be consumed outside Python (public) and what API must not be used outside Python (internal) unless you're writing a debugger or other uncommon very specific use case. The main difference is the warranties provided (public) or not (internal) by Python: tests, documentation, backward compatibility. In Python, sadly the types.CodeType type also has a public constructor and many projects break at each Python release because the API changes. Hopefully, it seems like the new CodeType.replace() method added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a better abstraction and closer to what developers need in practice. I'm not convinced that advertising an API as being Unstable (in the documentation?) is going to solve any kind of problem. People love to use private APIs, and they do it simply because it's technically possible :-) At the end of the day, we have to help them updating their code, otherwise we (at least my Red Hat team) cannot update Python. I designed the internal C API to be more annoying to be used (need to define a macro, need more specific #include) in the hope that people will think twice before using it :-) Victor
On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner@python.org> wrote:
In Python, sadly the types.CodeType type also has a public constructor and many projects break at each Python release because the API changes. Hopefully, it seems like the new CodeType.replace() method added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a better abstraction and closer to what developers need in practice.
It certainly has been for me. When I want to do bytecode hackery, I usually start by creating a function with def/lambda, then construct a modified function using f.__code__.replace(). It's the easiest way to ensure that all the little details are correct. ChrisA
On Fri, 1 Apr 2022, 6:47 pm Victor Stinner, <vstinner@python.org> wrote:
On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum <guido@python.org> wrote:
I'm not convinced that advertising an API as being Unstable (in the documentation?) is going to solve any kind of problem. People love to use private APIs, and they do it simply because it's technically possible :-) At the end of the day, we have to help them updating their code, otherwise we (at least my Red Hat team) cannot update Python.
I designed the internal C API to be more annoying to be used (need to define a macro, need more specific #include) in the hope that people will think twice before using it :-)
The changes you've made have been excellent, and the existing 3 categories (stable public ABI, stable public API, unstable internal API) cover the vast majority of cases. The final case that isn't quite covered yet is to offer a "semi-stable" API category for use cases that are intrinsically coupled to implementation details that may change between feature releases, but should remain stable within a release series. The concrete motivating example for the new category is the extra APIs you need in order to provide an alternative eval loop implementation. The internal API category doesn't properly cover that case, as the APIs there are free to change even in maintenance releases, and setting Py_BUILD_CORE exposes a lot more than what an alternative eval loop would need. Regular public functions may work in some cases, but aren't necessarily practical in others (such as exposing the internal frame details for use in alternative eval loops). From an implementation PoV, my own suggestion would be to define a new API tier with an opt-in macro rather than relying solely on documentation or naming conventions. For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under "Include/cpython/semi_stable/" (I don't like "unstable" as potential terminology here, since the internal API is already unstable - we're splitting the difference between that and the long term stability of the full public API) Cheers, Nick.
Victor _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5K... Code of Conduct: http://python.org/psf/codeofconduct/
+1 to Nick's analysis and proposal. I had been mulling over my own reply but this just about covers it. On Sun, Apr 3, 2022 at 6:29 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Fri, 1 Apr 2022, 6:47 pm Victor Stinner, <vstinner@python.org> wrote:
On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum <guido@python.org> wrote:
I'm not convinced that advertising an API as being Unstable (in the documentation?) is going to solve any kind of problem. People love to use private APIs, and they do it simply because it's technically possible :-) At the end of the day, we have to help them updating their code, otherwise we (at least my Red Hat team) cannot update Python.
I designed the internal C API to be more annoying to be used (need to define a macro, need more specific #include) in the hope that people will think twice before using it :-)
The changes you've made have been excellent, and the existing 3 categories (stable public ABI, stable public API, unstable internal API) cover the vast majority of cases.
The final case that isn't quite covered yet is to offer a "semi-stable" API category for use cases that are intrinsically coupled to implementation details that may change between feature releases, but should remain stable within a release series.
The concrete motivating example for the new category is the extra APIs you need in order to provide an alternative eval loop implementation.
The internal API category doesn't properly cover that case, as the APIs there are free to change even in maintenance releases, and setting Py_BUILD_CORE exposes a lot more than what an alternative eval loop would need.
Regular public functions may work in some cases, but aren't necessarily practical in others (such as exposing the internal frame details for use in alternative eval loops).
From an implementation PoV, my own suggestion would be to define a new API tier with an opt-in macro rather than relying solely on documentation or naming conventions.
For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under "Include/cpython/semi_stable/" (I don't like "unstable" as potential terminology here, since the internal API is already unstable - we're splitting the difference between that and the long term stability of the full public API)
Cheers, Nick.
Victor _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5K... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sun, Apr 3, 2022 at 6:32 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Fri, 1 Apr 2022, 6:47 pm Victor Stinner, <vstinner@python.org> wrote:
On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum <guido@python.org> wrote:
I'm not convinced that advertising an API as being Unstable (in the documentation?) is going to solve any kind of problem. People love to use private APIs, and they do it simply because it's technically possible :-) At the end of the day, we have to help them updating their code, otherwise we (at least my Red Hat team) cannot update Python.
I designed the internal C API to be more annoying to be used (need to define a macro, need more specific #include) in the hope that people will think twice before using it :-)
The changes you've made have been excellent, and the existing 3 categories (stable public ABI, stable public API, unstable internal API) cover the vast majority of cases.
The final case that isn't quite covered yet is to offer a "semi-stable" API category for use cases that are intrinsically coupled to implementation details that may change between feature releases, but should remain stable within a release series.
The concrete motivating example for the new category is the extra APIs you need in order to provide an alternative eval loop implementation.
The internal API category doesn't properly cover that case, as the APIs there are free to change even in maintenance releases, and setting Py_BUILD_CORE exposes a lot more than what an alternative eval loop would need.
Regular public functions may work in some cases, but aren't necessarily practical in others (such as exposing the internal frame details for use in alternative eval loops).
From an implementation PoV, my own suggestion would be to define a new API tier with an opt-in macro rather than relying solely on documentation or naming conventions.
For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under "Include/cpython/semi_stable/" (I don't like "unstable" as potential terminology here, since the internal API is already unstable - we're splitting the difference between that and the long term stability of the full public API)
Cheers, Nick.
+1 for an official "semi stable API tier". It's already the case that essentially anything in Python can change, it's just a question of how quickly and with how much friction. Public APIs need to go through a multi-version deprecation cycle (https://peps.python.org/pep-0387/#making-incompatible-changes). Private internal APIs can (theoretically) change without notice between patch releases. There's a missing tier for APIs that can change without notice between feature releases, but are guaranteed(*) to be backwards compatible within a feature release, and the PEP 523 frame evaluation API is an excellent example for this need (maybe any newly added API should always go through this stage for a few releases?). Even though the docs ( https://docs.python.org/3.10/c-api/intro.html#include-files) explicitly call out that _Py-prefixed APIs are internal and should not be used by extensions, this isn't the case in practice, so introducing the 3-tier concept could be an opportunity to clean up this situation a bit. What exactly should be the naming conventions per tier, and the names of the tiers, is bikeshedding that should happen after there's agreement about the tiers concept :-) (*) "guaranteed" with exceptions of course (e.g. security or other critical issue)
Victor _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5K... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZBNJTAXS... Code of Conduct: http://python.org/psf/codeofconduct/
On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
The changes you've made have been excellent, and the existing 3 categories (stable public ABI, stable public API, unstable internal API) cover the vast majority of cases.
The final case that isn't quite covered yet is to offer a "semi-stable" API category for use cases that are intrinsically coupled to implementation details that may change between feature releases, but should remain stable within a release series.
The concrete motivating example for the new category is the extra APIs you need in order to provide an alternative eval loop implementation.
The internal API category doesn't properly cover that case, as the APIs there are free to change even in maintenance releases, and setting Py_BUILD_CORE exposes a lot more than what an alternative eval loop would need.
Regular public functions may work in some cases, but aren't necessarily practical in others (such as exposing the internal frame details for use in alternative eval loops).
From an implementation PoV, my own suggestion would be to define a new API tier with an opt-in macro rather than relying solely on documentation or naming conventions.
For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under "Include/cpython/semi_stable/" (I don't like "unstable" as potential terminology here, since the internal API is already unstable - we're splitting the difference between that and the long term stability of the full public API)
For me an API is either stable (remains the same forever) or unstable (change time to time). Public API means: stable, documented, tested. Internal API means: unstable, not documented, not tested. I'm not convinced that it's worth it to create something in the middle. If you want to add doc and tests, it should become a public stable API. For example, IMO PyCode_New() (C API) and types.CodeType constructor (Python API) should be moved to the internal C API, likely with a deprecation period. Cython should not use it but a new stable API. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Wed, 6 Apr 2022, 7:05 am Victor Stinner, <vstinner@python.org> wrote:
The changes you've made have been excellent, and the existing 3 categories (stable public ABI, stable public API, unstable internal API) cover the vast majority of cases.
The final case that isn't quite covered yet is to offer a "semi-stable" API category for use cases that are intrinsically coupled to implementation
On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan <ncoghlan@gmail.com> wrote: details that may change between feature releases, but should remain stable within a release series.
The concrete motivating example for the new category is the extra APIs
you need in order to provide an alternative eval loop implementation.
The internal API category doesn't properly cover that case, as the APIs
there are free to change even in maintenance releases, and setting Py_BUILD_CORE exposes a lot more than what an alternative eval loop would need.
Regular public functions may work in some cases, but aren't necessarily
practical in others (such as exposing the internal frame details for use in alternative eval loops).
From an implementation PoV, my own suggestion would be to define a new
API tier with an opt-in macro rather than relying solely on documentation or naming conventions.
For example, define "Py_SEMI_STABLE_API" to opt in, with the headers
under "Include/cpython/semi_stable/" (I don't like "unstable" as potential terminology here, since the internal API is already unstable - we're splitting the difference between that and the long term stability of the full public API)
For me an API is either stable (remains the same forever) or unstable (change time to time).
Public API means: stable, documented, tested.
Internal API means: unstable, not documented, not tested.
I'm not convinced that it's worth it to create something in the middle. If you want to add doc and tests, it should become a public stable API.
The middle semi-stable tier formalises a concept that we already have: no guarantees across feature releases, but both API and ABI stable within a release series. It's useful for tightly coupled projects like Cython, as it means they can get core level performance without the risk of API compatibility breaks in maintenance releases. Without defining this tier, effectively the *entire* internal API becomes semi-stable, as any changes made will risk breaking the third party projects that we've told to define Py_BUILD_CORE when compiling. Cheers, Nick.
So here's my proposal: - This API stays with the regular public API (Include/cpython/), but to use it you'll need to #define Py_USING_UNSTABLE_API (name up for bikeshedding). - Since we're nearing Beta and there's no rush to break things, in 3.11 you only get a warning if you try to use it without the opt-in #define. In 3.12 it'll fail. - The functions will be renamed to drop the leading underscore. The old names will be available as aliases (using #define) and may be removed whenever the API changes. (Ideally, the underscore should always mark API that's fully private with no guarantees at all.) - The API will be stable during a minor release. (As usual, for extreme cases, exceptions are possible with SC approval.) - Docs will be updated: - https://devguide.python.org/c-api/ - Individual reference entries for the API and the new opt-in macro This applies to: - Functions added in PEP 523 - PyCode_New, PyCode_NewWithPosOnlyArgs - Ideally anything documented as subject to change between minor releases. (To be kind to users, if something is added later we should again have one release of compiler warnings before requiring the opt-in. Unless that API just changed and users would get errors anyway.) (Technically, this proposal needs SC approval -- PEP 387 exception for PyCode_New*. I'll play by the rules, of course.) On 06. 04. 22 17:21, Nick Coghlan wrote:
On Wed, 6 Apr 2022, 7:05 am Victor Stinner, <vstinner@python.org <mailto:vstinner@python.org>> wrote:
On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote: > The changes you've made have been excellent, and the existing 3 categories (stable public ABI, stable public API, unstable internal API) cover the vast majority of cases. > > The final case that isn't quite covered yet is to offer a "semi-stable" API category for use cases that are intrinsically coupled to implementation details that may change between feature releases, but should remain stable within a release series. > > The concrete motivating example for the new category is the extra APIs you need in order to provide an alternative eval loop implementation. > > The internal API category doesn't properly cover that case, as the APIs there are free to change even in maintenance releases, and setting Py_BUILD_CORE exposes a lot more than what an alternative eval loop would need. > > Regular public functions may work in some cases, but aren't necessarily practical in others (such as exposing the internal frame details for use in alternative eval loops). > > From an implementation PoV, my own suggestion would be to define a new API tier with an opt-in macro rather than relying solely on documentation or naming conventions. > > For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under "Include/cpython/semi_stable/" (I don't like "unstable" as potential terminology here, since the internal API is already unstable - we're splitting the difference between that and the long term stability of the full public API)
For me an API is either stable (remains the same forever) or unstable (change time to time).
Public API means: stable, documented, tested.
Internal API means: unstable, not documented, not tested.
I'm not convinced that it's worth it to create something in the middle. If you want to add doc and tests, it should become a public stable API.
The middle semi-stable tier formalises a concept that we already have: no guarantees across feature releases, but both API and ABI stable within a release series.
It's useful for tightly coupled projects like Cython, as it means they can get core level performance without the risk of API compatibility breaks in maintenance releases.
Without defining this tier, effectively the *entire* internal API becomes semi-stable, as any changes made will risk breaking the third party projects that we've told to define Py_BUILD_CORE when compiling.
Cheers, Nick.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7P3ENW56... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Apr 7, 2022 at 12:02 PM Petr Viktorin <encukou@gmail.com> wrote:
- This API stays with the regular public API (Include/cpython/), but to use it you'll need to #define Py_USING_UNSTABLE_API (name up for bikeshedding).
Since there is already something similar called "Py_LIMITED", I suggest dropping "USING_" for just: "Py_UNSTABLE_API".
- The functions will be renamed to drop the leading underscore. The old names will be available as aliases (using #define) and may be removed whenever the API changes. (Ideally, the underscore should always mark API that's fully private with no guarantees at all.)
Should functions entering the "unstable API" be documented and tested? For example, _PyEval_EvalFrameDefault() and _PyInterpreterState_SetEvalFrameFunc() have no test nor doc.
This applies to:
- PyCode_New, PyCode_NewWithPosOnlyArgs
It would be nice to update Cython to define the Py_UNSTABLE_API macro before the macro is required to get the function, since Cython still uses PyCode_New(). Should we deprecate types.CodeType constructor in the Python API, since types.CodeType.replace() exists and seems to be a better API ("more stable")? Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 07. 04. 22 17:10, Victor Stinner wrote:
On Thu, Apr 7, 2022 at 12:02 PM Petr Viktorin <encukou@gmail.com> wrote:
- This API stays with the regular public API (Include/cpython/), but to use it you'll need to #define Py_USING_UNSTABLE_API (name up for bikeshedding).
Since there is already something similar called "Py_LIMITED", I suggest dropping "USING_" for just: "Py_UNSTABLE_API".
But I really like the explicit “USING” :) I don't think there's too much value in these two being consistent. (Yay, bikeshedding!)
- The functions will be renamed to drop the leading underscore. The old names will be available as aliases (using #define) and may be removed whenever the API changes. (Ideally, the underscore should always mark API that's fully private with no guarantees at all.)
Should functions entering the "unstable API" be documented and tested?
For example, _PyEval_EvalFrameDefault() and _PyInterpreterState_SetEvalFrameFunc() have no test nor doc.
Yes. I'll add docs, for tests I'll at least open an issue.
This applies to:
- PyCode_New, PyCode_NewWithPosOnlyArgs
It would be nice to update Cython to define the Py_UNSTABLE_API macro before the macro is required to get the function, since Cython still uses PyCode_New().
That (for any project, not just Cython) is why I propose warnings in 3.11, before requiring the opt-in in 3.12.
Should we deprecate types.CodeType constructor in the Python API, since types.CodeType.replace() exists and seems to be a better API ("more stable")?
I don't know. But it's for a different discussion.
On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin, <encukou@gmail.com> wrote:
So here's my proposal:
- This API stays with the regular public API (Include/cpython/), but to use it you'll need to #define Py_USING_UNSTABLE_API (name up for bikeshedding).
I'm fine with the rest of what you suggest, but I don't think this is the right mechanical approach: * "unstable" is the wrong term. We already have an unstable API tier: the internal API, which can change even in maintenance releases. The value of the new tier is that it is "semi stable": stable in maintenance releases, unstable in feature releases. * the lesson I take from our stable ABI experience is that mixing two tiers of the API in a single header file is hard to maintain, as it's too easy to add a new API to the wrong section. A separate file that gets included automatically from the relevant header file(s) when the new definition is used makes the split much clearer. Cheers, Nick.
Here's the issue with the plan (including Nick's suggestions): https://github.com/python/cpython/issues/91744 On Sun, Apr 10, 2022 at 5:43 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin, <encukou@gmail.com> wrote:
So here's my proposal:
- This API stays with the regular public API (Include/cpython/), but to use it you'll need to #define Py_USING_UNSTABLE_API (name up for bikeshedding).
I'm fine with the rest of what you suggest, but I don't think this is the right mechanical approach:
* "unstable" is the wrong term. We already have an unstable API tier: the internal API, which can change even in maintenance releases. The value of the new tier is that it is "semi stable": stable in maintenance releases, unstable in feature releases. * the lesson I take from our stable ABI experience is that mixing two tiers of the API in a single header file is hard to maintain, as it's too easy to add a new API to the wrong section. A separate file that gets included automatically from the relevant header file(s) when the new definition is used makes the split much clearer.
Cheers, Nick.
On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin, <encukou@gmail.com> wrote:
This applies to:
- Functions added in PEP 523 - PyCode_New, PyCode_NewWithPosOnlyArgs - Ideally anything documented as subject to change between minor releases. (To be kind to users, if something is added later we should again have one release of compiler warnings before requiring the opt-in. Unless that API just changed and users would get errors anyway.)
* C APIs that provide access to compiled code whether in AST or opcode
Other candidate items for this tier: * non-opaque access to frame structs and any other key APIs needed to implement alternate eval loops with comparable performance to the default eval loop (unless & until we can figure out stable public APIs that can deliver equivalent performance) form (the API itself may be stable, but the compiled code isn't, so this is kinda covered by your last point) Cheers, Nick.
As someone who maintains a debugger that uses private api’s, I’d like to see some commitment to seeing them not change in micro releases such as 3.11.1 -> 3.11.2. Micro releases should be compatible with other micro releases of the same major.minor release such as 3.11 so that an extension compiled against 3.11.1 will work with 3.11.2. Compatibility was broken (at least) once in a 2.x series (maybe 2.4?) and we started getting reports of people using 2.x.0 were having problems. Fortunately, in that case compiling against 2.x.0 produced an extension that worked against all micro releases. Note that I’m not asking for private api’s not change in major.micro releases such as 3.10 -> 3.11. I know that there can be very good reasons to change private api's because that I probably will have work to do in order to support a new major.micro release. I also don’t think that exposing everything that every extension needs with a non-private api is a good idea because then the internals will be more difficult to change — you’d be signing up to support that api for all future major.micro versions until there’s a compatibility break. John
On Mar 30, 2022, at 11:01 AM, Victor Stinner <vstinner@python.org> wrote:
The internal C API can be used on purpose. But there is no backward compatibility warranty and it can change anytime. In practice, usually it only changes in 3.x.0 releases. For example, these private C API changed in Python 3.9 and Python 3.11 (see my first email in the other PEP 523 thread).
To use the internal C API, you have to declare the Py_BUILD_CORE macro and include an internal C API header file. For _PyInterpreterState_SetEvalFrameFunc(), it should be:
#ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault
Victor
On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev <python-dev@python.org> wrote:
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
The related project is TorchDynamo, which can be found here: https://github.com/facebookresearch/torchdynamo
We will likely move this into the core of PyTorch closer to release.
If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQ... Code of Conduct: http://python.org/psf/codeofconduct/
participants (11)
-
Chris Angelico -
Guido van Rossum -
Itamar O -
Jason Ansel -
John Ehresman -
Nick Coghlan -
Petr Viktorin -
Sebastian Berg -
Steve Dower -
Terry Reedy -
Victor Stinner