On Tue, Mar 22, 2022 at 7:33 PM Steve Dower wrote:

After a normal deprecation period, yes?

There is no backward compatibility warranty and no deprecation process for private APIs. Victor

On 24. 03. 22 20:06, Fabio Zadrozny wrote:

Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny mailto:fabiofz@gmail.com> escreveu:

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex() * _PyCode_GetExtra() * _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the pydevd debugger. The two others seem to be unused in the wild. I'm not sure if these ones should be moved to the internal C API. They can be left unchanged, since they don't use a type only defined by the internal C API.

Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.

i.e.:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...

The debugger already has workarounds because of changes to evaluation api over time (see: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...) and I know 3.11 won't be different.

I'm ok with changes as I understand that this is a special API -- as long as there's still a way to use it and get the information needed (the debugger already goes through many hops because it needs to use many internals of CPython -- in every new release it's a **really** big task to update to the latest version as almost everything that the debugger relies to make debugging fast changes across versions and I never really know if it'll be possible to support it until I really try to do the port -- I appreciate having less things in a public API so it's easier to have extensions work in other interpreters/not recompiling on newer versions, but please keep it possible to use private APIs which provides the same access that CPython has to access things internally for special cases such as the debugger).

Maybe later on that PEP from mark which allows a better debugger API could alleviate that (but until then, if possible I appreciate it if there's some effort not to break things unless really needed -- ideally with instructions on how to port).

Anyways, to wrap up, the debugger already needs to be built with `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as long as it's still accessible in that case) is probably not a big issue for the debugger and having setters/getters to set it instead of relying on `state.interp.eval_frame` seems good to me.

Cheers,

Fabio

I think the main issue here is the compatibility across the same version though... is it possible to have some kind of guarantee on private APIs that something won't change across micro-releases?

Currently we don't really have that (except that in practice, bugfix releases tend to not need internal API changes.)

I.e.: having the frame evaluation function change across major releases and having them be reworked seems reasonable, but then having the frame evaluation be changed across micro-releases wouldn't be.

So, I'm ok in pushing things to the internal API, but then I still would like guarantees about the compatibility of that API in the same major release (otherwise those setters/getters/frame evaluation should probably remain on the public API if the related structure was moved to the internal API).

Perhaps we need a new "tier" of C API for debuggers -- API that's guaranteed stable for a major release, and if it's changed it should always break with compile errors (e.g. the function gets a new argument), rather than silently change semantics. The internal API Cython & greenlet need might go it this category too.

Hello Fabio, Let's talk a bit about which API should, exactly, be guaranteed to not change across minor releases. So far it looks like: - PyEval_RequestCodeExtraIndex - PyCode_GetExtra - PyCode_SetExtra - PyFrameEvalFunction - PyInterpreterState_GetEvalFrameFunc - PyInterpreterState_SetEvalFrameFunc Do any more come to mind? The issue with this set is that in 3.11, _PyFrameEvalFunction changes its signature to take _PyInterpreterFrame rather than PyFrameObject. Exposing _PyInterpreterFrame would be quite problematic. For example, since it's not a PyObject, it has its own lifetime management that's controlled by the interpreter itself,. And it includes several pointers whose lifetime and semantics also isn't guaranteed (they might be borrowed, cached or filled on demand). I don't think we can make any guarantees on these, so the info needs to be accessed using getter functions. There is the function _PyFrame_GetFrameObject, which returns a PyFrameObject. I think it would be best to only expose _PyInterpreterFrame as an opaque structure, and expose PyFrame_GetFrameObject so debuggers can get a PyFrameObject from it. Does that sound reasonable? On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny wrote:

Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny escreveu:

...

...

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex() * _PyCode_GetExtra() * _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the pydevd debugger. The two others seem to be unused in the wild. I'm not sure if these ones should be moved to the internal C API. They can be left unchanged, since they don't use a type only defined by the internal C API.

Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.

i.e.:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...

The debugger already has workarounds because of changes to evaluation api over time (see: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...) and I know 3.11 won't be different.

I'm ok with changes as I understand that this is a special API -- as long as there's still a way to use it and get the information needed (the debugger already goes through many hops because it needs to use many internals of CPython -- in every new release it's a **really** big task to update to the latest version as almost everything that the debugger relies to make debugging fast changes across versions and I never really know if it'll be possible to support it until I really try to do the port -- I appreciate having less things in a public API so it's easier to have extensions work in other interpreters/not recompiling on newer versions, but please keep it possible to use private APIs which provides the same access that CPython has to access things internally for special cases such as the debugger).

Maybe later on that PEP from mark which allows a better debugger API could alleviate that (but until then, if possible I appreciate it if there's some effort not to break things unless really needed -- ideally with instructions on how to port).

Anyways, to wrap up, the debugger already needs to be built with `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as long as it's still accessible in that case) is probably not a big issue for the debugger and having setters/getters to set it instead of relying on `state.interp.eval_frame` seems good to me.

Cheers,

Fabio

I think the main issue here is the compatibility across the same version though... is it possible to have some kind of guarantee on private APIs that something won't change across micro-releases?

I.e.: having the frame evaluation function change across major releases and having them be reworked seems reasonable, but then having the frame evaluation be changed across micro-releases wouldn't be.

So, I'm ok in pushing things to the internal API, but then I still would like guarantees about the compatibility of that API in the same major release (otherwise those setters/getters/frame evaluation should probably remain on the public API if the related structure was moved to the internal API).

Cheers,

Fabio _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DHKE7LVN... Code of Conduct: http://python.org/psf/codeofconduct/

On 22. 04. 22 14:47, Fabio Zadrozny wrote:

Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin mailto:encukou@gmail.com> escreveu:

Hello Fabio, Let's talk a bit about which API should, exactly, be guaranteed to not change across minor releases. So far it looks like: - PyEval_RequestCodeExtraIndex - PyCode_GetExtra - PyCode_SetExtra - PyFrameEvalFunction - PyInterpreterState_GetEvalFrameFunc - PyInterpreterState_SetEvalFrameFunc

Do any more come to mind?

The issue with this set is that in 3.11, _PyFrameEvalFunction changes its signature to take _PyInterpreterFrame rather than PyFrameObject. Exposing _PyInterpreterFrame would be quite problematic. For example, since it's not a PyObject, it has its own lifetime management that's controlled by the interpreter itself,. And it includes several pointers whose lifetime and semantics also isn't guaranteed (they might be borrowed, cached or filled on demand). I don't think we can make any guarantees on these, so the info needs to be accessed using getter functions.

There is the function _PyFrame_GetFrameObject, which returns a PyFrameObject. I think it would be best to only expose _PyInterpreterFrame as an opaque structure, and expose PyFrame_GetFrameObject so debuggers can get a PyFrameObject from it. Does that sound reasonable?

Humm, now I'm a bit worried... the approach the debugger is using gets the PyFrameObject that's about to be executed and changes the PyFrameObject.f_code just before the execution so that the new code is executed instead.

From what you're saying the PyFrameObject isn't really used anymore (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will this approach still let the debugger patch the code object in the frame before it's actually executed?

PyFrameObject is a fairly thin wrapper around _PyInterpreterFrame -- it adds PyObject metadata (type & refcount), and not much else. It's allocated at most once for each _PyInterpreterFrame -- once it's created it stays attached to the frame. So, for the most heavily optimized code paths a PyFrameObject is not allocated, but it's trivial to get it whenever it's needed.

-- i.e.: the debugger changes the state.interp.eval_frame to its own custom evaluation function, but _PyEval_EvalFrameDefault is still what ends up being called afterwards (it works more as a hook to change the PyFrameObject.f_code prior to execution than as an alternate interpreter).

Ah, you also need PyEval_EvalFrameDefault exposed. The public version would take PyFrameObject and pass its _PyInterpreterFrame to the internal _PyEval_EvalFrameDefault.

On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny mailto:fabiofz@gmail.com> wrote: > > > Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny mailto:fabiofz@gmail.com> escreveu: >>> >>> PEP 523 API added more private functions for code objects: >>> >>> * _PyEval_RequestCodeExtraIndex() >>> * _PyCode_GetExtra() >>> * _PyCode_SetExtra() >>> >>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the >>> pydevd debugger. The two others seem to be unused in the wild. I'm not >>> sure if these ones should be moved to the internal C API. They can be >>> left unchanged, since they don't use a type only defined by the >>> internal C API. >> >> Just to note, the pydevd/debugpy debuggers actually uses all of those APIs. >> >> i.e.: >> >> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... >> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... >> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... >> >> The debugger already has workarounds because of changes to evaluation api over time (see: https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd... https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd...) and I know 3.11 won't be different. >> >> I'm ok with changes as I understand that this is a special API -- as long as there's still a way to use it and get the information needed (the debugger already goes through many hops because it needs to use many internals of CPython -- in every new release it's a **really** big task to update to the latest version as almost everything that the debugger relies to make debugging fast changes across versions and I never really know if it'll be possible to support it until I really try to do the port -- I appreciate having less things in a public API so it's easier to have extensions work in other interpreters/not recompiling on newer versions, but please keep it possible to use private APIs which provides the same access that CPython has to access things internally for special cases such as the debugger). >> >> Maybe later on that PEP from mark which allows a better debugger API could alleviate that (but until then, if possible I appreciate it if there's some effort not to break things unless really needed -- ideally with instructions on how to port). >> >> Anyways, to wrap up, the debugger already needs to be built with `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as long as it's still accessible in that case) is probably not a big issue for the debugger and having setters/getters to set it instead of relying on `state.interp.eval_frame` seems good to me. >> >> Cheers, >> >> Fabio >> > > > I think the main issue here is the compatibility across the same version though... is it possible to have some kind of guarantee on private APIs that something won't change across micro-releases? > > I.e.: having the frame evaluation function change across major releases and having them be reworked seems reasonable, but then having the frame evaluation be changed across micro-releases wouldn't be. > > So, I'm ok in pushing things to the internal API, but then I still would like guarantees about the compatibility of that API in the same major release (otherwise those setters/getters/frame evaluation should probably remain on the public API if the related structure was moved to the internal API). > > Cheers, > > Fabio > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org mailto:python-dev@python.org > To unsubscribe send an email to python-dev-leave@python.org mailto:python-dev-leave@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DHKE7LVN... https://mail.python.org/archives/list/python-dev@python.org/message/DHKE7LVN... > Code of Conduct: http://python.org/psf/codeofconduct/ http://python.org/psf/codeofconduct/

On 01. 04. 22 11:01, Victor Stinner wrote:

Hi,

Update on this issue: I merged my 2 PRs. https://bugs.python.org/issue46850

The following APIs have been moved to the internal C API:

- _PyFrameEvalFunction type - _PyInterpreterState_GetEvalFrameFunc() - _PyInterpreterState_SetEvalFrameFunc() - _PyEval_EvalFrameDefault()

Really? I haven't seen any support for that in this thread or on the issue, except from you. Now, the people who'd like a non-breaking solution will now need rush to develop and merge one until the next release, or the API breaks for users and it'll be too late to do anything about it. Meanwhile, you're off to make more changes. That is, frankly, very frustrating. Could you please stop unilaterally breaking documented API?

If you use any of these API in your debugger/profiler project, you have do add something like the code below to your project: --- #ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include #if PY_VERSION_HEX >= 0x030B00A7 # include // _PyInterpreterState_SetEvalFrameFunc() # include // _PyEval_EvalFrameDefault() #endif ---

IMO, this is a terrible suggestion that undermines the whole point of having a private API.

Contact me if you need help to update your affected projects.

IMO PEP 523 doesn't have to be updated since it already says that the APIs are private.

Since these APIs were added by PEP 523, I documented these changes in What's New in Python 3.11 > C API > Porting to Python 3.11,even if these APIs are private.

Victor

Hi, Steve, Petr: sorry if you feel that I didn't take your feedback in account, it's not case. Thanks for your valuable feedback. It seems like there was some misunderstanding. On Tue, Apr 5, 2022 at 2:49 AM Gregory P. Smith wrote:

Thanks for bringing this up on python-dev, Victor. That was good. But the point of the discussion should've been to continue working with people based on the replies rather than proceeding to merge removals of the APIs after people said they used them. (echoing Steve and Petr here...)

We discussed this on the steering council today. These APIs were in a weird state and despite past decisions at the time of PEP-523 in 2016 they should be treated as public-ish rather than entirely private. Because we published a document saying "here they are, use them!" and multiple projects have done so to good effect.

For 3.11 we'd like those PRs reverted.

Ok, I created https://github.com/python/cpython/pull/32343 to revert these two PRs. I will merge it as soon as the Python 3.11 release manager (Pablo) unblocks the main branch (he is currently preparing the alpha7 release). On Fri, Apr 1, 2022 at 12:36 PM Steve Dower wrote:

I don't see any additional discussion on the bug, and the prevailing opinion from actual users of this API is that it probably shouldn't change,

From what I understood my change basically only impacts "pydevd" users. Fabio who works on that project said that he was fine with that change ("I'm ok with changes"): https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55AN... debugpy, ptvsd, PyDev, PyCharm and VSCode Python use the same code base: https://github.com/fabioz/PyDev.Debugger Jason Ansel of TorchDynamo was worried that the API couldn't be used anymore. I replied that the API remains available. It's just about adding a few lines of code (that I provided). For me, all impacted users were made aware and joined the discussion. It's very rare to be able to reach *all* users impacted by an incompatible C API change! If I understood correctly, all questions were replied. For example, yes, the API remains accessible (with additional code), but no, sadly the API is not stable (can change at each 3.x.0 major release, but should not change in 3.x.y bugfix release). On Fri, Apr 1, 2022 at 12:36 PM Steve Dower wrote:

and certainly shouldn't become internal without additional guarantees about stability.

The discussed API is not stable, Brett repeated that. The internal API is not stable on purpose. I missed the discussion proposing to design a stable API for PEP 523. PEP 523 users are that the API is unstable and seem to be used to update their code at each major (3.x) Python releases. I'm not surprised that a debugger which requires a fast low-level access to Python internals require that. Or are you talking about not breaking the API in 3.x.y bugfix releases? Currently, it's an unwritten rule. IMO it's well respected by all core devs who understand that it's important to not break any API, including private and internal APIs, in minor bugfix releases. My changes are unrelated to that. On Fri, Apr 1, 2022 at 1:24 PM Petr Viktorin wrote:

Now, the people who'd like a non-breaking solution will now need rush to develop and merge one until the next release, or the API breaks for users and it'll be too late to do anything about it.

We are talking about two projects (pydevd and TorchDynamo) and there are 6 months until Python 3.11 final release (PEP 664). We are still at the alpha phase, no? I don't see why a change only affecting two projects is a big deal, knowing that these two projects have been made aware, and I offered a patch that they can apply right now. They don't need to wait for next October to apply my patch. Anyway, I will revert my changes (once the main branch is unblocked) to apply the SC's decision. Victor -- Night gathers, and now my watch begins. It shall not end until my death.