Hi, On 21/11/2019 8:04 pm, Brett Cannon wrote:
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue.
I have opinions :) I am opposed to adding any new API to modify the frame evaluation function (the interpreter). Although I would like to remove PEP 523, for now I propose that we do absolutely nothing. The pydev/pycharm debugger can be easily modified to use the internal data structure, so there is no urgency to do anything. Longer term I would like to remove PEP 523. I will write a PEP for this, but in the meantime let me explain why. First of all, PEP 523 doesn't explain who gets priority if two modules want to change the interpreter. Who wins? Or does it just crash? In case that seems unlikely, consider that PyCharm and PyDev already (mis)use PEP 523 for debugging and that cProfile could be sped up by using PEP 523 (please don't). What happens if you want to debug the profiled code? PEP 523 is not a good fit for debugging. --------------------------------------- Because using sys.settrace is so slow and there is no API specifically for debugging, the PyDev debugger has little choice but to use PEP 523. Debuggers using this approach need to directly modify bytecode, which is fragile. We need a good API for debuggers and other tools. PEP 523 isn't great for "JIT"s either. ------------------------------------- (I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer" would be a better term, IMO) A frame is a poor region for optimization, any effective optimizer would only use PEP 523 as a starting hook to allow it to monitor execution and select better regions for optimization. It is however, fine for experimental purposes. I have no problem keeping until we have evidence of working "JIT". As mentioned above, any such optimizer would also interact very poorly with a PEP 523 based debugger. PEP 523 prevents some optimizations ----------------------------------- Making Python-to-Python calls without a C-level call is prohibited by PEP 523, but would allow faster to Python-to-Python calls and faster switching between stacks of coroutines. Alternative API for tools based loosely on the JVM -------------------------------------------------- The JVM does not offer any means to modify the evaluation engine, but it does provide a way for tools to interact with the JVM. JVM debuggers support watchpoints, but Python debuggers do not. JVM debuggers can interact with the JVM while the JVM runs at (near) full speed, whereas Pdb causes the CPython interpreter to slow down a lot. We can gain these advantages by adding a tooling API to (C)Python. A tooling API would support both debuggers and profilers by providing the following features: * Breakpoints * Watchpoints (both local and global variables) * Callbacks for events (call/return/exception/entry/exit/line). Cheers, Mark.