C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

Hi, I proposed two PRs to move the private C API (Include/cpython/) of PEP 523 "Adding a frame evaluation API to CPython" to the internal C API (Include/internals/): * https://github.com/python/cpython/pull/32052 * https://github.com/python/cpython/pull/32054 API: * _PyFrameEvalFunction type * _PyInterpreterState_GetEvalFrameFunc() * _PyInterpreterState_SetEvalFrameFunc() * (undocumented) _PyEval_EvalFrameDefault() The private API to get/set the eval function *is* documented at: https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFra... I added the Get/Set functions so debuggers don't have to access directly to the PyInterpreterState structure which has been moved to the internal C API in Python 3.8. This API causes me multiple issues: * It's a private API and I'm trying to remove the private API from the public C API header files. * The _PyFrameEvalFunction type is not stable: it got a new "tstate" parameter in Python 3.9 and the type of the second parameter changed from PyFrameObject* to _PyInterpreterFrame* in Python 3.11. * These functions use the _PyInterpreterFrame type which is part of the internal C API. While Pyston didn't bring a JIT compiler to Python with PEP 523, debuggers were made faster by using this API. Debuggers like pydevd, debugpy and ptvsd use it. I propose to move theses API to the internal header files (Include/internals/) to clarify that it's not part of the public C API and that there is no backward compatibility warranty. The change is being discussed at: https://bugs.python.org/issue46850 -- PEP 523 API added more private functions for code objects: * _PyEval_RequestCodeExtraIndex() * _PyCode_GetExtra() * _PyCode_SetExtra() The _PyEval_RequestCodeExtraIndex() function seems to be used by the pydevd debugger. The two others seem to be unused in the wild. I'm not sure if these ones should be moved to the internal C API. They can be left unchanged, since they don't use a type only defined by the internal C API. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 3/22/2022 6:07 PM, Victor Stinner wrote:
After a normal deprecation period, yes?
Pyjion (https://github.com/tonybaloney/Pyjion) uses it, and ptvsd never did. I also understand that pydevd - and implicitly debugpy - disabled its use of the API by default because of reliability, but that was a few years back so maybe they fixed it. So it's likely only being used by the project that requested it, though I don't think that's enough to justify skipping normal deprecation. Cheers, Steve

On 3/22/2022 11:28 PM, Victor Stinner wrote:
And yet you're asking the question, which means you know these are special ;) The PEP says: "The API is purposefully listed as private to communicate the fact that there are no semantic guarantees of the API between Python releases." Absence/presence isn't a semantic guarantee, it's an availability guarantee. Code using them should be able to rely on their presence, and ideally their prototype (though it seems we've messed that up in the past), but shouldn't expect code that worked against 3.8 to also work against 3.9 or 3.10. Perhaps in hindsight, we could have not used the underscore and just explicitly described them as being behaviorally unstable between major versions. I guess that would have raised exactly the same question though. The point is, it's a documented API that we've told people they can use. We can't simply revoke that without telling people that it's going to happen, even if we covered ourselves for there being version changes that affect how they need to be used. Cheers, Steve

On Wed, Mar 23, 2022 at 2:48 AM Petr Viktorin <encukou@gmail.com> wrote:
From the PEP <https://peps.python.org/pep-0523/#expanding-pycodeobject>: "The API is purposefully listed as private to communicate the fact that there are no semantic guarantees of the API between Python releases." -Brett
IMO, if external code should use these, they should lose the leading underscore.

Just to note, the pydevd/debugpy debuggers actually uses all of those APIs. i.e.: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... The debugger already has workarounds because of changes to evaluation api over time (see: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...) and I know 3.11 won't be different. I'm ok with changes as I understand that this is a special API -- as long as there's still a way to use it and get the information needed (the debugger already goes through many hops because it needs to use many internals of CPython -- in every new release it's a **really** big task to update to the latest version as almost everything that the debugger relies to make debugging fast changes across versions and I never really know if it'll be possible to support it until I really try to do the port -- I appreciate having less things in a public API so it's easier to have extensions work in other interpreters/not recompiling on newer versions, but please keep it possible to use private APIs which provides the same access that CPython has to access things internally for special cases such as the debugger). Maybe later on that PEP from mark which allows a better debugger API could alleviate that (but until then, if possible I appreciate it if there's some effort not to break things unless really needed -- ideally with instructions on how to port). Anyways, to wrap up, the debugger already needs to be built with `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as long as it's still accessible in that case) is probably not a big issue for the debugger and having setters/getters to set it instead of relying on `state.interp.eval_frame` seems good to me. Cheers, Fabio

Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny <fabiofz@gmail.com> escreveu:
I think the main issue here is the compatibility across the same version though... is it possible to have some kind of guarantee on private APIs that something won't change across micro-releases? I.e.: having the frame evaluation function change across major releases and having them be reworked seems reasonable, but then having the frame evaluation be changed across micro-releases wouldn't be. So, I'm ok in pushing things to the internal API, but then I still would like guarantees about the compatibility of that API in the same major release (otherwise those setters/getters/frame evaluation should probably remain on the public API if the related structure was moved to the internal API). Cheers, Fabio

On 24. 03. 22 20:06, Fabio Zadrozny wrote:
Currently we don't really have that (except that in practice, bugfix releases tend to not need internal API changes.)
Perhaps we need a new "tier" of C API for debuggers -- API that's guaranteed stable for a major release, and if it's changed it should always break with compile errors (e.g. the function gets a new argument), rather than silently change semantics. The internal API Cython & greenlet need might go it this category too.

On Mon, Mar 28, 2022 at 1:44 PM Petr Viktorin <encukou@gmail.com> wrote:
We should provide public C API functions for Cython and greenlet needs: well tested and documented functions. There is an on-going work in Python 3.11: * https://docs.python.org/dev/c-api/frame.html * https://github.com/faster-cpython/ideas/issues/309 * https://bugs.python.org/issue40421 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Hello Fabio, Let's talk a bit about which API should, exactly, be guaranteed to not change across minor releases. So far it looks like: - PyEval_RequestCodeExtraIndex - PyCode_GetExtra - PyCode_SetExtra - PyFrameEvalFunction - PyInterpreterState_GetEvalFrameFunc - PyInterpreterState_SetEvalFrameFunc Do any more come to mind? The issue with this set is that in 3.11, _PyFrameEvalFunction changes its signature to take _PyInterpreterFrame rather than PyFrameObject. Exposing _PyInterpreterFrame would be quite problematic. For example, since it's not a PyObject, it has its own lifetime management that's controlled by the interpreter itself,. And it includes several pointers whose lifetime and semantics also isn't guaranteed (they might be borrowed, cached or filled on demand). I don't think we can make any guarantees on these, so the info needs to be accessed using getter functions. There is the function _PyFrame_GetFrameObject, which returns a PyFrameObject. I think it would be best to only expose _PyInterpreterFrame as an opaque structure, and expose PyFrame_GetFrameObject so debuggers can get a PyFrameObject from it. Does that sound reasonable? On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny <fabiofz@gmail.com> wrote:

Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin <encukou@gmail.com> escreveu:
Humm, now I'm a bit worried... the approach the debugger is using gets the PyFrameObject that's about to be executed and changes the PyFrameObject.f_code just before the execution so that the new code is executed instead. From what you're saying the PyFrameObject isn't really used anymore (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will this approach still let the debugger patch the code object in the frame before it's actually executed? -- i.e.: the debugger changes the state.interp.eval_frame to its own custom evaluation function, but _PyEval_EvalFrameDefault is still what ends up being called afterwards (it works more as a hook to change the PyFrameObject.f_code prior to execution than as an alternate interpreter). On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny <fabiofz@gmail.com> wrote:

On 22. 04. 22 14:47, Fabio Zadrozny wrote:
PyFrameObject is a fairly thin wrapper around _PyInterpreterFrame -- it adds PyObject metadata (type & refcount), and not much else. It's allocated at most once for each _PyInterpreterFrame -- once it's created it stays attached to the frame. So, for the most heavily optimized code paths a PyFrameObject is not allocated, but it's trivial to get it whenever it's needed.
Ah, you also need PyEval_EvalFrameDefault exposed. The public version would take PyFrameObject and pass its _PyInterpreterFrame to the internal _PyEval_EvalFrameDefault.

On Fri, Apr 22, 2022 at 2:52 PM Fabio Zadrozny <fabiofz@gmail.com> wrote:
Humm, now I'm a bit worried... the approach the debugger is using gets the PyFrameObject that's about to be executed and changes the PyFrameObject.f_code just before the execution so that the new code is executed instead.
You can already modify _PyInterpreterFrame.f_code using the internal C API.
From what you're saying the PyFrameObject isn't really used anymore (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will this approach still let the debugger patch the code object in the frame before it's actually executed?
There is no public C API to modify the "f_code" attribute of a PyFrameObject. There is only PyFrame_GetCode() *getter*. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Hi, Update on this issue: I merged my 2 PRs. https://bugs.python.org/issue46850 The following APIs have been moved to the internal C API: - _PyFrameEvalFunction type - _PyInterpreterState_GetEvalFrameFunc() - _PyInterpreterState_SetEvalFrameFunc() - _PyEval_EvalFrameDefault() If you use any of these API in your debugger/profiler project, you have do add something like the code below to your project: --- #ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #if PY_VERSION_HEX >= 0x030B00A7 # include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() # include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault() #endif --- Contact me if you need help to update your affected projects. IMO PEP 523 doesn't have to be updated since it already says that the APIs are private. Since these APIs were added by PEP 523, I documented these changes in What's New in Python 3.11 > C API > Porting to Python 3.11,even if these APIs are private. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 4/1/2022 10:01 AM, Victor Stinner wrote:
Update on this issue: I merged my 2 PRs. https://bugs.python.org/issue46850
So what was the point of this discussion then? I don't see any additional discussion on the bug, and the prevailing opinion from actual users of this API is that it probably shouldn't change, and certainly shouldn't become internal without additional guarantees about stability. Did we all just waste a whole lot of electrons discussing a foregone conclusion? (My apologies to the people I invited into this thread. I genuinely thought it would help to have outside perspective on this potential change.) Cheers, Steve

On 01. 04. 22 11:01, Victor Stinner wrote:
Really? I haven't seen any support for that in this thread or on the issue, except from you. Now, the people who'd like a non-breaking solution will now need rush to develop and merge one until the next release, or the API breaks for users and it'll be too late to do anything about it. Meanwhile, you're off to make more changes. That is, frankly, very frustrating. Could you please stop unilaterally breaking documented API?
IMO, this is a terrible suggestion that undermines the whole point of having a private API.

On Fri, Apr 1, 2022 at 2:06 AM Victor Stinner <vstinner@python.org> wrote:
Thanks for bringing this up on python-dev, Victor. That was good. But the point of the discussion should've been to continue working with people based on the replies rather than proceeding to merge removals of the APIs after people said they used them. (echoing Steve and Petr here...) We discussed this on the steering council today. These APIs were in a weird state and despite past decisions at the time of PEP-523 <https://peps.python.org/pep-0523/> in 2016 they should be treated as public-ish rather than entirely private. Because we published a document saying "here they are, use them!" and multiple projects have done so to good effect. For 3.11 we'd like those PRs reverted. We see the following as the better way forward for these APIs: Add a new #define that can be set before the #include <Python.h> that exposes non-limited but stable within a bugfix/patch releases APIs (ie: Petr's earlier suggestion). These would be the first to fall within. To do so we should give these, behind that #define, non _-prefixed "public style" names as these are quasi-public and cannot be changed as readily as other true internals. We still, per the PEP, reserve the right to turn these into no-op potentially warning setting APIs in a future release (likely 3.12?) as they are at least documented as being unstable/private in the PEP. So in 3.11 these should continue to exist as in 3.6-3.10: - _PyFrameEvalFunction type - _PyInterpreterState_GetEvalFrameFunc() - _PyInterpreterState_SetEvalFrameFunc() - _PyEval_EvalFrameDefault() AND in 3.11: - #define protected versions of those without the leading _ become available. - (i'm intentionally not suggesting a #define name, y'all can pick something) In 3.12: - the _ prefixed versions can go away. People using the APIs should've updated their code to use the new #define and new names when building against >=3.11 by then. - Whether the APIs continue to be as useful and act as originally claimed in PEP 523 is up to the 3.12 implementors (out of scope for this thread). They occupy a newly defined middle ground between the "forever" style limited API and the "can break even on bugfix/patch release" internal API that wasn't a concept for us in 2016 when PEP 523 was written. Why? Being conservative with things in active use that weren't *really* private, and similar to what Mark Shannon and Petr said, we *do not* want people to #define Py_BUILD_CORE_MODULE and start poking at internals in arbitrary ways. That exposes a whole pile of other things for (ab)use that are even more unstable. Avoiding that helps avoid temptation to go wild and helps us identify users. -gps (steering council hat on)

Hi, Steve, Petr: sorry if you feel that I didn't take your feedback in account, it's not case. Thanks for your valuable feedback. It seems like there was some misunderstanding. On Tue, Apr 5, 2022 at 2:49 AM Gregory P. Smith <greg@krypto.org> wrote:
Ok, I created https://github.com/python/cpython/pull/32343 to revert these two PRs. I will merge it as soon as the Python 3.11 release manager (Pablo) unblocks the main branch (he is currently preparing the alpha7 release). On Fri, Apr 1, 2022 at 12:36 PM Steve Dower <steve.dower@python.org> wrote:
From what I understood my change basically only impacts "pydevd" users. Fabio who works on that project said that he was fine with that change ("I'm ok with changes"): https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55AN... debugpy, ptvsd, PyDev, PyCharm and VSCode Python use the same code base: https://github.com/fabioz/PyDev.Debugger Jason Ansel of TorchDynamo was worried that the API couldn't be used anymore. I replied that the API remains available. It's just about adding a few lines of code (that I provided). For me, all impacted users were made aware and joined the discussion. It's very rare to be able to reach *all* users impacted by an incompatible C API change! If I understood correctly, all questions were replied. For example, yes, the API remains accessible (with additional code), but no, sadly the API is not stable (can change at each 3.x.0 major release, but should not change in 3.x.y bugfix release). On Fri, Apr 1, 2022 at 12:36 PM Steve Dower <steve.dower@python.org> wrote:
and certainly shouldn't become internal without additional guarantees about stability.
The discussed API is not stable, Brett repeated that. The internal API is not stable on purpose. I missed the discussion proposing to design a stable API for PEP 523. PEP 523 users are that the API is unstable and seem to be used to update their code at each major (3.x) Python releases. I'm not surprised that a debugger which requires a fast low-level access to Python internals require that. Or are you talking about not breaking the API in 3.x.y bugfix releases? Currently, it's an unwritten rule. IMO it's well respected by all core devs who understand that it's important to not break any API, including private and internal APIs, in minor bugfix releases. My changes are unrelated to that. On Fri, Apr 1, 2022 at 1:24 PM Petr Viktorin <encukou@gmail.com> wrote:
We are talking about two projects (pydevd and TorchDynamo) and there are 6 months until Python 3.11 final release (PEP 664). We are still at the alpha phase, no? I don't see why a change only affecting two projects is a big deal, knowing that these two projects have been made aware, and I offered a patch that they can apply right now. They don't need to wait for next October to apply my patch. Anyway, I will revert my changes (once the main branch is unblocked) to apply the SC's decision. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 4/5/2022 11:52 PM, Victor Stinner wrote:
"I'm okay with" isn't really a sign of support, particularly in this kind of relationship where *we* have all the power. Fabio doesn't have a choice but to be okay with whatever we decide, and he obviously accepts that. I'm still going to strongly advocate for not going out of our way to break users like Fabio. Just because we have a reputation for doing it doesn't mean we should keep doing it.
There's no way we reached all the users :) But we did reach more than usual.
I can't tell whether you missed the discussion or if you're deliberately misrepresenting it as a persuasive technique. The API may not be "stable" according to our typical rules around public C APIs, but it has explicit rules about its own stability (well, had, though once the revert goes in it will have them again). Nobody suggested making it *more* stable than it already was. We just want to treat its existing stability rules with respect and either transition out of them slowly, or leave them as they are. Cheers, Steve

Hi, On 22/03/2022 6:07 pm, Victor Stinner wrote:
I just want to say that it is important the API is easy to use and access. Otherwise there is a temptation to directly set the underlying function pointer, and that means we don't get to see when PEP 523 is used. We might then mis-optimize on the assumption that PEP 523 isn't used. Cheers, Mark.

On 3/22/2022 6:07 PM, Victor Stinner wrote:
After a normal deprecation period, yes?
Pyjion (https://github.com/tonybaloney/Pyjion) uses it, and ptvsd never did. I also understand that pydevd - and implicitly debugpy - disabled its use of the API by default because of reliability, but that was a few years back so maybe they fixed it. So it's likely only being used by the project that requested it, though I don't think that's enough to justify skipping normal deprecation. Cheers, Steve

On 3/22/2022 11:28 PM, Victor Stinner wrote:
And yet you're asking the question, which means you know these are special ;) The PEP says: "The API is purposefully listed as private to communicate the fact that there are no semantic guarantees of the API between Python releases." Absence/presence isn't a semantic guarantee, it's an availability guarantee. Code using them should be able to rely on their presence, and ideally their prototype (though it seems we've messed that up in the past), but shouldn't expect code that worked against 3.8 to also work against 3.9 or 3.10. Perhaps in hindsight, we could have not used the underscore and just explicitly described them as being behaviorally unstable between major versions. I guess that would have raised exactly the same question though. The point is, it's a documented API that we've told people they can use. We can't simply revoke that without telling people that it's going to happen, even if we covered ourselves for there being version changes that affect how they need to be used. Cheers, Steve

On Wed, Mar 23, 2022 at 2:48 AM Petr Viktorin <encukou@gmail.com> wrote:
From the PEP <https://peps.python.org/pep-0523/#expanding-pycodeobject>: "The API is purposefully listed as private to communicate the fact that there are no semantic guarantees of the API between Python releases." -Brett
IMO, if external code should use these, they should lose the leading underscore.

Just to note, the pydevd/debugpy debuggers actually uses all of those APIs. i.e.: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... The debugger already has workarounds because of changes to evaluation api over time (see: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...) and I know 3.11 won't be different. I'm ok with changes as I understand that this is a special API -- as long as there's still a way to use it and get the information needed (the debugger already goes through many hops because it needs to use many internals of CPython -- in every new release it's a **really** big task to update to the latest version as almost everything that the debugger relies to make debugging fast changes across versions and I never really know if it'll be possible to support it until I really try to do the port -- I appreciate having less things in a public API so it's easier to have extensions work in other interpreters/not recompiling on newer versions, but please keep it possible to use private APIs which provides the same access that CPython has to access things internally for special cases such as the debugger). Maybe later on that PEP from mark which allows a better debugger API could alleviate that (but until then, if possible I appreciate it if there's some effort not to break things unless really needed -- ideally with instructions on how to port). Anyways, to wrap up, the debugger already needs to be built with `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as long as it's still accessible in that case) is probably not a big issue for the debugger and having setters/getters to set it instead of relying on `state.interp.eval_frame` seems good to me. Cheers, Fabio

Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny <fabiofz@gmail.com> escreveu:
I think the main issue here is the compatibility across the same version though... is it possible to have some kind of guarantee on private APIs that something won't change across micro-releases? I.e.: having the frame evaluation function change across major releases and having them be reworked seems reasonable, but then having the frame evaluation be changed across micro-releases wouldn't be. So, I'm ok in pushing things to the internal API, but then I still would like guarantees about the compatibility of that API in the same major release (otherwise those setters/getters/frame evaluation should probably remain on the public API if the related structure was moved to the internal API). Cheers, Fabio

On 24. 03. 22 20:06, Fabio Zadrozny wrote:
Currently we don't really have that (except that in practice, bugfix releases tend to not need internal API changes.)
Perhaps we need a new "tier" of C API for debuggers -- API that's guaranteed stable for a major release, and if it's changed it should always break with compile errors (e.g. the function gets a new argument), rather than silently change semantics. The internal API Cython & greenlet need might go it this category too.

On Mon, Mar 28, 2022 at 1:44 PM Petr Viktorin <encukou@gmail.com> wrote:
We should provide public C API functions for Cython and greenlet needs: well tested and documented functions. There is an on-going work in Python 3.11: * https://docs.python.org/dev/c-api/frame.html * https://github.com/faster-cpython/ideas/issues/309 * https://bugs.python.org/issue40421 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Hello Fabio, Let's talk a bit about which API should, exactly, be guaranteed to not change across minor releases. So far it looks like: - PyEval_RequestCodeExtraIndex - PyCode_GetExtra - PyCode_SetExtra - PyFrameEvalFunction - PyInterpreterState_GetEvalFrameFunc - PyInterpreterState_SetEvalFrameFunc Do any more come to mind? The issue with this set is that in 3.11, _PyFrameEvalFunction changes its signature to take _PyInterpreterFrame rather than PyFrameObject. Exposing _PyInterpreterFrame would be quite problematic. For example, since it's not a PyObject, it has its own lifetime management that's controlled by the interpreter itself,. And it includes several pointers whose lifetime and semantics also isn't guaranteed (they might be borrowed, cached or filled on demand). I don't think we can make any guarantees on these, so the info needs to be accessed using getter functions. There is the function _PyFrame_GetFrameObject, which returns a PyFrameObject. I think it would be best to only expose _PyInterpreterFrame as an opaque structure, and expose PyFrame_GetFrameObject so debuggers can get a PyFrameObject from it. Does that sound reasonable? On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny <fabiofz@gmail.com> wrote:

Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin <encukou@gmail.com> escreveu:
Humm, now I'm a bit worried... the approach the debugger is using gets the PyFrameObject that's about to be executed and changes the PyFrameObject.f_code just before the execution so that the new code is executed instead. From what you're saying the PyFrameObject isn't really used anymore (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will this approach still let the debugger patch the code object in the frame before it's actually executed? -- i.e.: the debugger changes the state.interp.eval_frame to its own custom evaluation function, but _PyEval_EvalFrameDefault is still what ends up being called afterwards (it works more as a hook to change the PyFrameObject.f_code prior to execution than as an alternate interpreter). On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny <fabiofz@gmail.com> wrote:

On 22. 04. 22 14:47, Fabio Zadrozny wrote:
PyFrameObject is a fairly thin wrapper around _PyInterpreterFrame -- it adds PyObject metadata (type & refcount), and not much else. It's allocated at most once for each _PyInterpreterFrame -- once it's created it stays attached to the frame. So, for the most heavily optimized code paths a PyFrameObject is not allocated, but it's trivial to get it whenever it's needed.
Ah, you also need PyEval_EvalFrameDefault exposed. The public version would take PyFrameObject and pass its _PyInterpreterFrame to the internal _PyEval_EvalFrameDefault.

On Fri, Apr 22, 2022 at 2:52 PM Fabio Zadrozny <fabiofz@gmail.com> wrote:
Humm, now I'm a bit worried... the approach the debugger is using gets the PyFrameObject that's about to be executed and changes the PyFrameObject.f_code just before the execution so that the new code is executed instead.
You can already modify _PyInterpreterFrame.f_code using the internal C API.
From what you're saying the PyFrameObject isn't really used anymore (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will this approach still let the debugger patch the code object in the frame before it's actually executed?
There is no public C API to modify the "f_code" attribute of a PyFrameObject. There is only PyFrame_GetCode() *getter*. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Hi, Update on this issue: I merged my 2 PRs. https://bugs.python.org/issue46850 The following APIs have been moved to the internal C API: - _PyFrameEvalFunction type - _PyInterpreterState_GetEvalFrameFunc() - _PyInterpreterState_SetEvalFrameFunc() - _PyEval_EvalFrameDefault() If you use any of these API in your debugger/profiler project, you have do add something like the code below to your project: --- #ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include <Python.h> #if PY_VERSION_HEX >= 0x030B00A7 # include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc() # include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault() #endif --- Contact me if you need help to update your affected projects. IMO PEP 523 doesn't have to be updated since it already says that the APIs are private. Since these APIs were added by PEP 523, I documented these changes in What's New in Python 3.11 > C API > Porting to Python 3.11,even if these APIs are private. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 4/1/2022 10:01 AM, Victor Stinner wrote:
Update on this issue: I merged my 2 PRs. https://bugs.python.org/issue46850
So what was the point of this discussion then? I don't see any additional discussion on the bug, and the prevailing opinion from actual users of this API is that it probably shouldn't change, and certainly shouldn't become internal without additional guarantees about stability. Did we all just waste a whole lot of electrons discussing a foregone conclusion? (My apologies to the people I invited into this thread. I genuinely thought it would help to have outside perspective on this potential change.) Cheers, Steve

On 01. 04. 22 11:01, Victor Stinner wrote:
Really? I haven't seen any support for that in this thread or on the issue, except from you. Now, the people who'd like a non-breaking solution will now need rush to develop and merge one until the next release, or the API breaks for users and it'll be too late to do anything about it. Meanwhile, you're off to make more changes. That is, frankly, very frustrating. Could you please stop unilaterally breaking documented API?
IMO, this is a terrible suggestion that undermines the whole point of having a private API.

On Fri, Apr 1, 2022 at 2:06 AM Victor Stinner <vstinner@python.org> wrote:
Thanks for bringing this up on python-dev, Victor. That was good. But the point of the discussion should've been to continue working with people based on the replies rather than proceeding to merge removals of the APIs after people said they used them. (echoing Steve and Petr here...) We discussed this on the steering council today. These APIs were in a weird state and despite past decisions at the time of PEP-523 <https://peps.python.org/pep-0523/> in 2016 they should be treated as public-ish rather than entirely private. Because we published a document saying "here they are, use them!" and multiple projects have done so to good effect. For 3.11 we'd like those PRs reverted. We see the following as the better way forward for these APIs: Add a new #define that can be set before the #include <Python.h> that exposes non-limited but stable within a bugfix/patch releases APIs (ie: Petr's earlier suggestion). These would be the first to fall within. To do so we should give these, behind that #define, non _-prefixed "public style" names as these are quasi-public and cannot be changed as readily as other true internals. We still, per the PEP, reserve the right to turn these into no-op potentially warning setting APIs in a future release (likely 3.12?) as they are at least documented as being unstable/private in the PEP. So in 3.11 these should continue to exist as in 3.6-3.10: - _PyFrameEvalFunction type - _PyInterpreterState_GetEvalFrameFunc() - _PyInterpreterState_SetEvalFrameFunc() - _PyEval_EvalFrameDefault() AND in 3.11: - #define protected versions of those without the leading _ become available. - (i'm intentionally not suggesting a #define name, y'all can pick something) In 3.12: - the _ prefixed versions can go away. People using the APIs should've updated their code to use the new #define and new names when building against >=3.11 by then. - Whether the APIs continue to be as useful and act as originally claimed in PEP 523 is up to the 3.12 implementors (out of scope for this thread). They occupy a newly defined middle ground between the "forever" style limited API and the "can break even on bugfix/patch release" internal API that wasn't a concept for us in 2016 when PEP 523 was written. Why? Being conservative with things in active use that weren't *really* private, and similar to what Mark Shannon and Petr said, we *do not* want people to #define Py_BUILD_CORE_MODULE and start poking at internals in arbitrary ways. That exposes a whole pile of other things for (ab)use that are even more unstable. Avoiding that helps avoid temptation to go wild and helps us identify users. -gps (steering council hat on)

Hi, Steve, Petr: sorry if you feel that I didn't take your feedback in account, it's not case. Thanks for your valuable feedback. It seems like there was some misunderstanding. On Tue, Apr 5, 2022 at 2:49 AM Gregory P. Smith <greg@krypto.org> wrote:
Ok, I created https://github.com/python/cpython/pull/32343 to revert these two PRs. I will merge it as soon as the Python 3.11 release manager (Pablo) unblocks the main branch (he is currently preparing the alpha7 release). On Fri, Apr 1, 2022 at 12:36 PM Steve Dower <steve.dower@python.org> wrote:
From what I understood my change basically only impacts "pydevd" users. Fabio who works on that project said that he was fine with that change ("I'm ok with changes"): https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55AN... debugpy, ptvsd, PyDev, PyCharm and VSCode Python use the same code base: https://github.com/fabioz/PyDev.Debugger Jason Ansel of TorchDynamo was worried that the API couldn't be used anymore. I replied that the API remains available. It's just about adding a few lines of code (that I provided). For me, all impacted users were made aware and joined the discussion. It's very rare to be able to reach *all* users impacted by an incompatible C API change! If I understood correctly, all questions were replied. For example, yes, the API remains accessible (with additional code), but no, sadly the API is not stable (can change at each 3.x.0 major release, but should not change in 3.x.y bugfix release). On Fri, Apr 1, 2022 at 12:36 PM Steve Dower <steve.dower@python.org> wrote:
and certainly shouldn't become internal without additional guarantees about stability.
The discussed API is not stable, Brett repeated that. The internal API is not stable on purpose. I missed the discussion proposing to design a stable API for PEP 523. PEP 523 users are that the API is unstable and seem to be used to update their code at each major (3.x) Python releases. I'm not surprised that a debugger which requires a fast low-level access to Python internals require that. Or are you talking about not breaking the API in 3.x.y bugfix releases? Currently, it's an unwritten rule. IMO it's well respected by all core devs who understand that it's important to not break any API, including private and internal APIs, in minor bugfix releases. My changes are unrelated to that. On Fri, Apr 1, 2022 at 1:24 PM Petr Viktorin <encukou@gmail.com> wrote:
We are talking about two projects (pydevd and TorchDynamo) and there are 6 months until Python 3.11 final release (PEP 664). We are still at the alpha phase, no? I don't see why a change only affecting two projects is a big deal, knowing that these two projects have been made aware, and I offered a patch that they can apply right now. They don't need to wait for next October to apply my patch. Anyway, I will revert my changes (once the main branch is unblocked) to apply the SC's decision. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 4/5/2022 11:52 PM, Victor Stinner wrote:
"I'm okay with" isn't really a sign of support, particularly in this kind of relationship where *we* have all the power. Fabio doesn't have a choice but to be okay with whatever we decide, and he obviously accepts that. I'm still going to strongly advocate for not going out of our way to break users like Fabio. Just because we have a reputation for doing it doesn't mean we should keep doing it.
There's no way we reached all the users :) But we did reach more than usual.
I can't tell whether you missed the discussion or if you're deliberately misrepresenting it as a persuasive technique. The API may not be "stable" according to our typical rules around public C APIs, but it has explicit rules about its own stability (well, had, though once the revert goes in it will have them again). Nobody suggested making it *more* stable than it already was. We just want to treat its existing stability rules with respect and either transition out of them slowly, or leave them as they are. Cheers, Steve

Hi, On 22/03/2022 6:07 pm, Victor Stinner wrote:
I just want to say that it is important the API is easy to use and access. Otherwise there is a temptation to directly set the underlying function pointer, and that means we don't get to see when PEP 523 is used. We might then mis-optimize on the assumption that PEP 523 isn't used. Cheers, Mark.
participants (7)
-
Brett Cannon
-
Fabio Zadrozny
-
Gregory P. Smith
-
Mark Shannon
-
Petr Viktorin
-
Steve Dower
-
Victor Stinner