Hi,
On 21/11/2019 8:04 pm, Brett Cannon wrote:
> An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
>
> A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
>
> And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
>
> If you have an opinion please weight in on the issue.
I have opinions :)
I am opposed to adding any new API to modify the frame evaluation
function (the interpreter). Although I would like to remove PEP 523, for
now I propose that we do absolutely nothing.
The pydev/pycharm debugger can be easily modified to use the internal
data structure, so there is no urgency to do anything.
Longer term I would like to remove PEP 523. I will write a PEP for this,
but in the meantime let me explain why.
First of all, PEP 523 doesn't explain who gets priority if two modules
want to change the interpreter. Who wins? Or does it just crash?
In case that seems unlikely, consider that PyCharm and PyDev already
(mis)use PEP 523 for debugging and that cProfile could be sped up by
using PEP 523 (please don't). What happens if you want to debug the
profiled code?
It is a process wide global for use in special circumstances. Those two examples are debuggers. Others are not. But as a global it should be expected that it isn't for general application level use. Only one user per process. Or else the old manual chaining rule applies. If you find it set when you are setting your own, you need to call the previously set one. and when unsetting, if it isn't set to yours, you should leave it set and turn your own into a no-op. Realistically these aren't problems and most people things aren't going to use the API. I don't think anyone has ever even encountered a combination of uses at once situation yet given the complexity and obscurity.
At Google we use it to provide C/C++ level sampling profilers access to the Python level program state by plumbing all Python calls through a unique C function per Python callable such that a C level stack trace symbolizes to information directly related to which Python function was called. So that light weight sampling profilers can get an accurate picture at the function level of where time is being spent in a process executing Python code. Without heavy handed live running scary signal based interpreter state internal introspections (ie: vmprof). It's a cool hack to get Python functions to show up on the native C stack by proxy. No, I don't expect others to have implemented that (it requires a build step that creates C stubs for all python callables, links those in and loads it all up - not something most people have a way to do) [yes I call this a hack, so no I don't block anything on it - i'm just pointing out that there are many creative uses of being able to inject oneself as a C function call within a Python call stack whether we ultimately want that to be a thing or not]
Another use was made that recorded live types of functions at runtime to be fed into type analyzers and ultimately feed back into adding practical annotations on existing code. your event callback approach would work for that case.
PEP 523 is not a good fit for debugging.
---------------------------------------
Because using sys.settrace is so slow and there is no API specifically
for debugging, the PyDev debugger has little choice but to use PEP 523.
Debuggers using this approach need to directly modify bytecode, which is
fragile.
We need a good API for debuggers and other tools.
PEP 523 isn't great for "JIT"s either.
-------------------------------------
(I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer"
would be a better term, IMO)
A frame is a poor region for optimization, any effective optimizer would
only use PEP 523 as a starting hook to allow it to monitor execution and
select better regions for optimization.
It is however, fine for experimental purposes. I have no problem keeping
until we have evidence of working "JIT".
Does evidence require an open source release? An existing JIT implementation was mentioned in the PEP...
As mentioned above, any such optimizer would also interact very poorly
with a PEP 523 based debugger.
PEP 523 prevents some optimizations
-----------------------------------
Making Python-to-Python calls without a C-level call is prohibited by
PEP 523, but would allow faster to Python-to-Python calls and faster
switching between stacks of coroutines.
True. This is an argument for a withdrawl PEP, not an argument for deleting the feature without notice though.
Alternative API for tools based loosely on the JVM
--------------------------------------------------
The JVM does not offer any means to modify the evaluation engine, but it
does provide a way for tools to interact with the JVM.
JVM debuggers support watchpoints, but Python debuggers do not.
JVM debuggers can interact with the JVM while the JVM runs at (near)
full speed, whereas Pdb causes the CPython interpreter to slow down a lot.
We can gain these advantages by adding a tooling API to (C)Python.
A tooling API would support both debuggers and profilers by providing
the following features:
* Breakpoints
* Watchpoints (both local and global variables)
* Callbacks for events (call/return/exception/entry/exit/line).
likely +1, though I don't think this scenario works for our C level profiling information exposure use case. neither would removing the C stack from Python function calls, and I'm supportive of getting off of the C stack so I'm not going to object on any grounds around my employer's existing hack using 523 functionality.
Anyways given there are workarounds to be able to continue using it in 3.8.x, this doesn't feel 3.8.1 level urgent. It'd be good to document the workarounds and 3.8 know issue in the PEP though as a addendum.
-gps