API design question: how to extend sys.settrace()?
Hi, In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead? Would it be possible to make the API "future-proof", so we can extend it again later? I almost never used sys.settrace(), so I prefer to ask on this python-dev list. Copy of the George King's message: https://bugs.python.org/issue29400#msg298441 """ After reviewing the thread, I'm reminded that the main design problem concerns preserving behavior of this idiom: "old=sys.gettrace(); ...; sys.settrace(old)" If we add more state, i.e. the `trace_instructions` bool, then the above idiom no longer correctly stores/reloads the full state. Here are the options that I see: 1. New APIs: * `gettrace() -> Callable # no change.` * `settrace(tracefunc: Callable) -> None # no change.` * `gettraceinst() -> Callable # new.` * `settraceinst(tracefunc: Callable) -> None # new.` Behavior: * `settrace()` raises if `gettraceinst()` is not None. * `settraceinst()` raises if `gettrace()` is not None. 2. Add keyword arg to `settrace`. * `gettrace() -> Callable # no change.` * `settrace(tracefunc: Callable, trace_instructions:Optional[bool]=False) -> None # added keyword.` * `gettracestate() -> Dict[str, Any] # new.` Behavior: * `gettracestate()` returns the complete trace-related state: currently, `tracefunc` and `trace_instructions` fields. * `gettrace()` raises an exception if any of the trace state does not match the old behavior, i.e. if `trace_instructions` is set. 3. New API, but with extensible keyword args: * `settracestate(tracefunc: Callable, trace_instructions:Optional[bool]=False) -> None # new.` * `gettracestate() -> Dict[str, Any] # new.` Behavior: * `settrace()` raises if `gettracestate()` is not None. * `settracestate()` raises if `gettrace()` is not None. As I see it: * approach #1 is more conservative because it does not modify the old API. * approach #2 is more extensible because all future features can be represented by the state dictionary. * approach #3 has both of these strengths, but is also the most work. Examples of possible future features via keywords: * branch-level tracing, as briefly disscussed above. * instruction filtering set, which could be a more general version of the branch tracing. I intend to prototype one or both of these, but I'm also feeling a bit of time pressure to get the basic feature on track for 3.7. """ settraceinst() doesn't seem "future-proof" to me. Victor
On 09/27/2017 02:56 PM, Victor Stinner wrote:
Hi,
In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead?
One possibility would be for settrace to query the capability on the part of the provided callable. For example: def settrace(tracefn): if getattr(tracefn, '__settrace_trace_instructions__', False): ... we need to trace instructions ... In general, the "trace function" can be upgraded to the concept of a "trace object" with (optional) methods under than __call__. This is extensible, easy to document, and fully supports the original "old=sys.gettrace(); ...; sys.settrace(old)" idiom.
27.09.17 15:56, Victor Stinner пише:
In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead?
I afraid that this change breaks an assumption in frame_setlineno() about the state of the stack. This can corrupt the stack if you jump from the instruction which is a part of Python operation. For example FOR_ITER expects an iterator on the stack. If you jump to the end of the loop from the middle of an assignment operator and skip say STORE_FAST, you will left an arbitrary value on the stack. This can lead to unpredictable consequences.
On Wed, Sep 27, 2017 at 8:10 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
I afraid that this change breaks an assumption in frame_setlineno() about the state of the stack. This can corrupt the stack if you jump from the instruction which is a part of Python operation. For example FOR_ITER expects an iterator on the stack. If you jump to the end of the loop from the middle of an assignment operator and skip say STORE_FAST, you will left an arbitrary value on the stack. This can lead to unpredictable consequences.
Well, probably OT but the solution for that would be to stop using a local stack and instead use explicit addressing. -- --Guido van Rossum (python.org/~guido)
On 27 September 2017 at 22:56, Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead?
As part of investigating the current signal safety problems in with statements [1], I added the ability to trace lines, opcodes, both, or neither by setting a couple of flags on a per-frame basis: https://docs.python.org/dev/whatsnew/3.7.html#other-cpython-implementation-c... So the idea is that if you want per-opcode tracing, your trace function has to turn it on for each frame when handling the call event, and you accept the responsibility of not messing up the frame stack. Cheers, Nick. [1] See https://bugs.python.org/issue29988 if you're interested in the gory details -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (5)
-
Guido van Rossum
-
Hrvoje Niksic
-
Nick Coghlan
-
Serhiy Storchaka
-
Victor Stinner