[Python-Dev] Re: Improve CPython tracing performance

On Fri, Oct 30, 2020 at 7:02 AM Nick Coghlan wrote:

On Fri, 30 Oct 2020 at 00:19, Fabio Zadrozny wrote:

...

On Thu, Oct 29, 2020 at 9:45 AM Victor Stinner wrote:

...

...

If it's non controversial, is a PEP needed or just an issue to track

it would be enough to remove those 2 lines?

...

Incompatible changes should be well documented in What's New in Python 3.10. In this case, I don't think that a deprecation period is needed.

Just open an issue. Please post the URL to your issue in reply to your email. It's even better if you can write a PR to implement your idea ;-)

Removing those calls would require a PEP, as it would break all sorts of tools in cases that currently work correctly.

...

Ok, I've created https://bugs.python.org/issue42197 to track it.

Please also have a look at PEP 558 and its draft reference implementation at https://github.com/python/cpython/pull/3640

The way these trampoline calls currently work isn't just slow, it's actually broken in various ways, and changing them to use a write-through proxy instead of a dict-based snapshot means that the cost of producing those dict-based snapshots simply because tracing is turned on will go away.

The PEP itself didn't seem to be particularly controversial (at least in its current form - earlier versions drew more objections), but there's a bunch of preparatory work that needs to be done before it could seriously be submitted for final review (specifically: the write-through proxy isn't actually implementing the full mutable mapping API. In order for it to do that without excessive code duplication, the helper functions already written for ordered dictionaries needed to moved out to a separate linkable module so that the new write-through proxy can reuse them without taking a separate copy of them)

Hi Nick! As a note, the current implementation does allow debuggers to mutate frame locals -- as long as they understand that they need to call ` PyFrame_LocalsToFast ` when doing such a change -- potentially using ctypes (I'm just mentioning this because PEP 558 seems to imply this isn't possible). i.e.: Debuggers already *must* call ` PyFrame_LocalsToFast ` if locals from a frame which is not the current frame are being mutated, so, as far as I can see a debugger is already broken if it isn't doing that -- some years ago I even thought about exposing it in the frame API: https://bugs.python.org/issue1654367, but in the end accessing it through the C-API through ctypes does get the job done, debugger authors just need to be aware of it -- PyPy also has a counterpart mentioned in that issue. I agree that having f_locals be a proxy that does everything transparently would be better, but unfortunately I don't currently have the time available to help there... in your opinion, just removing those calls as I proposed (requiring that debuggers call `PyFrame_LocalsToFast`) would be acceptable? If you think it is, I'll proceed on creating the PEP, otherwise I'll probably just drop it until f_locals becomes a proxy (in which case I'd expect the `PyFrame_FastToLocalsWithError` and `PyFrame_LocalsToFast` to be removed in the same commit which converts f_locals to a proxy). Cheers, Fabio