Stalemate on bringing back PEP 523 support into Python 3.8
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue. A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely). And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any). If you have an opinion please weight in on the issue.
(Replying here since b.p.o doesn't seem to want to let me log in) I've used the PEP 523 API several times for multiple projects, from analyzing bytecode frequency (which probably could be done with other means) to inspecting type information from function calls. I could probably do such analysis a different way too, (some people do it via tracing). But from what I have experienced, the PEP 523 API is much faster than alternatives, and I would be sad to see it go. I also have written some of these experiments in Rust, and it would make it harder to use if I had to deal with enabling it. I would much prefer the proposed setters/getters. Also the discussion is here for those who are interested https://bugs.python.org/issue38500 Best, Ethan On Thu, Nov 21, 2019 at 12:07 PM Brett Cannon <brett@python.org> wrote:
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL... Code of Conduct: http://python.org/psf/codeofconduct/
I'm in favor of adding a public C API to get and set the frame evaluation function. API excluded from the limited API. I wrote https://github.com/python/cpython/pull/17340 to add such API to Python 3.8.1. For me, it's a Python 3.8.0 regression. Victor
Hi, On 21/11/2019 8:04 pm, Brett Cannon wrote:
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue.
I have opinions :) I am opposed to adding any new API to modify the frame evaluation function (the interpreter). Although I would like to remove PEP 523, for now I propose that we do absolutely nothing. The pydev/pycharm debugger can be easily modified to use the internal data structure, so there is no urgency to do anything. Longer term I would like to remove PEP 523. I will write a PEP for this, but in the meantime let me explain why. First of all, PEP 523 doesn't explain who gets priority if two modules want to change the interpreter. Who wins? Or does it just crash? In case that seems unlikely, consider that PyCharm and PyDev already (mis)use PEP 523 for debugging and that cProfile could be sped up by using PEP 523 (please don't). What happens if you want to debug the profiled code? PEP 523 is not a good fit for debugging. --------------------------------------- Because using sys.settrace is so slow and there is no API specifically for debugging, the PyDev debugger has little choice but to use PEP 523. Debuggers using this approach need to directly modify bytecode, which is fragile. We need a good API for debuggers and other tools. PEP 523 isn't great for "JIT"s either. ------------------------------------- (I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer" would be a better term, IMO) A frame is a poor region for optimization, any effective optimizer would only use PEP 523 as a starting hook to allow it to monitor execution and select better regions for optimization. It is however, fine for experimental purposes. I have no problem keeping until we have evidence of working "JIT". As mentioned above, any such optimizer would also interact very poorly with a PEP 523 based debugger. PEP 523 prevents some optimizations ----------------------------------- Making Python-to-Python calls without a C-level call is prohibited by PEP 523, but would allow faster to Python-to-Python calls and faster switching between stacks of coroutines. Alternative API for tools based loosely on the JVM -------------------------------------------------- The JVM does not offer any means to modify the evaluation engine, but it does provide a way for tools to interact with the JVM. JVM debuggers support watchpoints, but Python debuggers do not. JVM debuggers can interact with the JVM while the JVM runs at (near) full speed, whereas Pdb causes the CPython interpreter to slow down a lot. We can gain these advantages by adding a tooling API to (C)Python. A tooling API would support both debuggers and profilers by providing the following features: * Breakpoints * Watchpoints (both local and global variables) * Callbacks for events (call/return/exception/entry/exit/line). Cheers, Mark.
On Fri, Nov 22, 2019 at 10:10 AM Mark Shannon <mark@hotpy.org> wrote:
Hi,
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about
On 21/11/2019 8:04 pm, Brett Cannon wrote: providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue.
I have opinions :)
I am opposed to adding any new API to modify the frame evaluation function (the interpreter). Although I would like to remove PEP 523, for now I propose that we do absolutely nothing. The pydev/pycharm debugger can be easily modified to use the internal data structure, so there is no urgency to do anything.
Longer term I would like to remove PEP 523. I will write a PEP for this, but in the meantime let me explain why.
First of all, PEP 523 doesn't explain who gets priority if two modules want to change the interpreter. Who wins? Or does it just crash? In case that seems unlikely, consider that PyCharm and PyDev already (mis)use PEP 523 for debugging and that cProfile could be sped up by using PEP 523 (please don't). What happens if you want to debug the profiled code?
It is a process wide global for use in special circumstances. Those two examples are debuggers. Others are not. But as a global it should be expected that it isn't for general application level use. Only one user per process. Or else the old manual chaining rule applies. If you find it set when you are setting your own, you need to call the previously set one. and when unsetting, if it isn't set to yours, you should leave it set and turn your own into a no-op. Realistically these aren't problems and most people things aren't going to use the API. I don't think anyone has ever even encountered a combination of uses at once situation yet given the complexity and obscurity. At Google we use it to provide C/C++ level sampling profilers access to the Python level program state by plumbing all Python calls through a unique C function per Python callable such that a C level stack trace symbolizes to information directly related to which Python function was called. So that light weight sampling profilers can get an accurate picture at the function level of where time is being spent in a process executing Python code. Without heavy handed live running scary signal based interpreter state internal introspections (ie: vmprof). It's a cool hack to get Python functions to show up on the native C stack by proxy. No, I don't expect others to have implemented that (it requires a build step that creates C stubs for all python callables, links those in and loads it all up - not something most people have a way to do) [yes I call this a hack, so no I don't block anything on it - i'm just pointing out that there are many creative uses of being able to inject oneself as a C function call within a Python call stack whether we ultimately want that to be a thing or not] Another use was made that recorded live types of functions at runtime to be fed into type analyzers and ultimately feed back into adding practical annotations on existing code. your event callback approach would work for that case.
PEP 523 is not a good fit for debugging. --------------------------------------- Because using sys.settrace is so slow and there is no API specifically for debugging, the PyDev debugger has little choice but to use PEP 523. Debuggers using this approach need to directly modify bytecode, which is fragile. We need a good API for debuggers and other tools.
PEP 523 isn't great for "JIT"s either. ------------------------------------- (I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer" would be a better term, IMO) A frame is a poor region for optimization, any effective optimizer would only use PEP 523 as a starting hook to allow it to monitor execution and select better regions for optimization. It is however, fine for experimental purposes. I have no problem keeping until we have evidence of working "JIT".
Does evidence require an open source release? An existing JIT implementation *was* mentioned in the PEP... As mentioned above, any such optimizer would also interact very poorly
with a PEP 523 based debugger.
PEP 523 prevents some optimizations ----------------------------------- Making Python-to-Python calls without a C-level call is prohibited by PEP 523, but would allow faster to Python-to-Python calls and faster switching between stacks of coroutines.
True. This is an argument for a withdrawl PEP, not an argument for deleting the feature without notice though. Alternative API for tools based loosely on the JVM
--------------------------------------------------
The JVM does not offer any means to modify the evaluation engine, but it does provide a way for tools to interact with the JVM. JVM debuggers support watchpoints, but Python debuggers do not. JVM debuggers can interact with the JVM while the JVM runs at (near) full speed, whereas Pdb causes the CPython interpreter to slow down a lot. We can gain these advantages by adding a tooling API to (C)Python.
A tooling API would support both debuggers and profilers by providing the following features: * Breakpoints * Watchpoints (both local and global variables) * Callbacks for events (call/return/exception/entry/exit/line).
likely +1, though I don't think this scenario works for our C level profiling information exposure use case. neither would removing the C stack from Python function calls, and I'm supportive of getting off of the C stack so I'm not going to object on any grounds around my employer's existing hack using 523 functionality. Anyways given there are workarounds to be able to continue using it in 3.8.x, this doesn't feel 3.8.1 level *urgent*. It'd be good to document the workarounds and 3.8 know issue in the PEP though as a addendum. -gps
On Sat., 23 Nov. 2019, 9:46 am Gregory P. Smith, <greg@krypto.org> wrote:
Anyways given there are workarounds to be able to continue using it in 3.8.x, this doesn't feel 3.8.1 level *urgent*. It'd be good to document the workarounds and 3.8 know issue in the PEP though as a addendum.
Aye, adding a Python 3.8 porting guide entry noting that PEP 523 is now considered part of the internal API seems reasonable to me. You're so deep in interpreter internals when overriding the eval loop that needing to enable the internal API to do it at all makes sense. It *wouldn't* make sense to me for us to add a new public API for overriding the eval loop, only to turn around and deprecate it almost immediately in favour of a JVM style debugger/profiler integration API. Cheers, Nick.
-gps _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H56ZIYW7... Code of Conduct: http://python.org/psf/codeofconduct/
I agree with Mark on "for now I propose that we do absolutely nothing". (I'll wait on a PEP for the rest of his points.) The capability of PEP 523 (swapping in a different PyEval_EvalFrame() impl.) is deep in the CPython runtime functionality. It is low-level, highly impactful, and there-be-dragons. So in my mind it makes perfect sense to keep it "internal", which was an unintended (but not necessarily incorrect) side effect of making PyInterpreterState opaque. Nothing says "don't use this unless you know what you are doing" better than requiring that extensions define Py_BUILD_CORE_MODULE (or Py_BUILD_CORE or whatever) if they want to use it. Accessor functions seem like overkill in that case since they would require the same Py_BUILD_CORE_MODULE that the PyInterpreterState field now requires. -eric On Thu, Nov 21, 2019 at 1:06 PM Brett Cannon <brett@python.org> wrote:
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL... Code of Conduct: http://python.org/psf/codeofconduct/
I modified "make install" to install internal header files, so it's possible to use the internal C API in debuggers and profilers. For example, to be able to inspect Python internals without having to call functions (which might modify the Python internal state). I made this change to be able to move more APIs to the internal API, to reduce the size of the public C API. If it wouldn't be possible to use the internal C API, it would be more risky to move things into the internal API. We might break legit use cases with no solution for users. Here PyCharm (for example) could be modified to use the internal C API, no? Define Py_BUILD_CORE_MODULE macro and include "internal/pycore_pystate.h". Victor Le ven. 22 nov. 2019 à 20:36, Eric Snow <ericsnowcurrently@gmail.com> a écrit :
I agree with Mark on "for now I propose that we do absolutely nothing". (I'll wait on a PEP for the rest of his points.) The capability of PEP 523 (swapping in a different PyEval_EvalFrame() impl.) is deep in the CPython runtime functionality. It is low-level, highly impactful, and there-be-dragons.
So in my mind it makes perfect sense to keep it "internal", which was an unintended (but not necessarily incorrect) side effect of making PyInterpreterState opaque. Nothing says "don't use this unless you know what you are doing" better than requiring that extensions define Py_BUILD_CORE_MODULE (or Py_BUILD_CORE or whatever) if they want to use it.
Accessor functions seem like overkill in that case since they would require the same Py_BUILD_CORE_MODULE that the PyInterpreterState field now requires.
-eric
On Thu, Nov 21, 2019 at 1:06 PM Brett Cannon <brett@python.org> wrote:
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QCHNEWGB... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
Victor Stinner wrote:
I modified "make install" to install internal header files, so it's possible to use the internal C API in debuggers and profilers. For example, to be able to inspect Python internals without having to call functions (which might modify the Python internal state). I made this change to be able to move more APIs to the internal API, to reduce the size of the public C API. If it wouldn't be possible to use the internal C API, it would be more risky to move things into the internal API. We might break legit use cases with no solution for users. Here PyCharm (for example) could be modified to use the internal C API, no? Define Py_BUILD_CORE_MODULE macro and include "internal/pycore_pystate.h".
https://bugs.python.org/msg356983 suggests that Fabio is okay with it being internal only for pydevd.
Victor Le ven. 22 nov. 2019 à 20:36, Eric Snow ericsnowcurrently@gmail.com a écrit :
I agree with Mark on "for now I propose that we do absolutely nothing". (I'll wait on a PEP for the rest of his points.) The capability of PEP 523 (swapping in a different PyEval_EvalFrame() impl.) is deep in the CPython runtime functionality. It is low-level, highly impactful, and there-be-dragons. So in my mind it makes perfect sense to keep it "internal", which was an unintended (but not necessarily incorrect) side effect of making PyInterpreterState opaque. Nothing says "don't use this unless you know what you are doing" better than requiring that extensions define Py_BUILD_CORE_MODULE (or Py_BUILD_CORE or whatever) if they want to use it. Accessor functions seem like overkill in that case since they would require the same Py_BUILD_CORE_MODULE that the PyInterpreterState field now requires. -eric On Thu, Nov 21, 2019 at 1:06 PM Brett Cannon brett@python.org wrote:
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed PEP 523 support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue. A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely). And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any). If you have an opinion please weight in on the issue.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL... Code of Conduct: http://python.org/psf/codeofconduct/
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QCHNEWGB... Code of Conduct: http://python.org/psf/codeofconduct/ --
Night gathers, and now my watch begins. It shall not end until my death.
Hi, I plan to merge my PR 17340 at the end of week to not miss Python 3.9 feature freeze deadline, unless someone speaks up and find a good reason to not merge this PR. The PR adds a public C API PyInterpreterState_SetEvalFrameFunc() and now pass tstate to the frame evaluation function. Please discuss the issue at https://bugs.python.org/issue38500 Victor
participants (7)
-
Brett Cannon -
Eric Snow -
Ethan Smith -
Gregory P. Smith -
Mark Shannon -
Nick Coghlan -
Victor Stinner