Idea: Allow multiple levels of tracers
Hi, Here's something I want in Python: Multiple levels of tracers working on top of each other, instead of just one. I'm talking about the tracer that one can set by calling sys.settrace. I've recently released PySnooper: https://github.com/cool-RR/PySnooper/ One of the difficulties I have, is that I can't debug or run the `coverage` tool on the core of this module. That's because the core is a trace function, and debuggers and coverage tools work by setting a trace function. When PySnooper sets its trace function using `sys.settrace`, the code that runs in that trace function runs without getting traced by the coverage tracer. This means that people who develop debuggers and coverage tools can't use a debugger or a coverage tool on the core of their tool. It's quite an annoying problem. My proposed solution: Multiple levels of tracing, instead of just one. When you install a tracer, you're not replacing the existing one, you're appending a tracer to the existing list of tracers. If this was implemented, then when PySnooper would install its tracer, the coverage tracer would still be active and running, for every line of code including the ones in PySnooper's tracer. Obviously, we'll need to figure out the API and any other kind of problems with this proposal. What do you think? Thanks, Ram.
Hmm, looks like, for this to work, you'll need the existing tracer to be cooperative. Right now there are existing tracers, for example coverage's tracer and Wing IDE's tracer, and I would need to modify them for your idea to work, right? If I understand your idea correctly, the first tracer would monkeypatch `sys.settrace` so whenever someone else adds a tracer, it doesn't really do `sys.settrace` but just add a function that the real tracer would be calling after it's done tracing. But this can't really be done without the original tracer implementing it, right? On Thu, Apr 25, 2019 at 6:16 PM Ram Rachum <ram@rachum.com> wrote:
Perhaps I misunderstand what's implied by "simple(!) monkeypatch of sys.settrace", but the trickiest part of Ram's proposal is that the body of one trace function would still trigger the remaining trace functions. That to me sounds like it's going to require changes to ceval.c --Ned. On 4/25/19 12:26 PM, Ram Rachum wrote:
It wouldn't be difficult to have a list of trace functions, so that every line of "real" Python executed, would invoke all the trace functions. But Ram has asked for something more: when the first trace function is executing, its line should themselves be traced by the remaining trace functions in the list. Presumably the lines in the second trace function should also be traced by the function third in the list, and so on. This is the thing that will be difficult to accomplish. --Ned. On 4/25/19 2:02 PM, Anders Hovmöller wrote:
On 26 Apr 2019, at 05:47, Ram Rachum <ram@rachum.com> wrote:
Ah, I thought about it now and Ned is right. This would require modifications to ceval.c and others.
Pity!
The question is... Does anyone else think it's a good idea?
I do. It seems to me that coverage is a very useful tool that shouldn’t be unusable for certain programs if we can avoid it. If we should include it in CPython in the end probably depends on how much it complicates the implementation and/or how solid tests are written obviously. I would point out that we can still get another coverage metric for these scenarios though: mutation coverage. But that’s extremely slow to collect compared to traditional coverage. / Anders
Personally I would look to other means to get the coverage report for a tracing tool or debugger. For example why not use unittesting and mocking to allow the trace code to be run and measured? After all you only have to mock for one functions interface. As for debugging I would use print() or logging to find what I need. Barry
It's possible, but it would be very cumbersome, for a bunch of reasons. One of them is that the tracing code inspects the frame, the variables referenced in it, and it even opens the file of the code object of the frame. It will be difficult to mock all of that, and even if that's possible, we won't have high confidence that the mock is reliable. On Sun, Apr 28, 2019, 11:06 Barry Scott <barry@barrys-emacs.org> wrote:
On 28 Apr 2019, at 09:12, Ram Rachum <ram.rachum@gmail.com> wrote:
It's possible, but it would be very cumbersome, for a bunch of reasons. One of them is that the tracing code inspects the frame, the variables referenced in it, and it even opens the file of the code object of the frame. It will be difficult to mock all of that, and even if that's possible, we won't have high confidence that the mock is reliable.
It is not so scary that I would not take it on myself if I had the need. Maybe you use real frames and only fake up the call into the tracecode. What I'm thinking is that it is possible and can be done today without C level changes to Python. Maybe if there are changes to the C python it only needs a second tracehook that runs for the code in the sys.settrace only. Barry
Hmm, looks like, for this to work, you'll need the existing tracer to be cooperative. Right now there are existing tracers, for example coverage's tracer and Wing IDE's tracer, and I would need to modify them for your idea to work, right? If I understand your idea correctly, the first tracer would monkeypatch `sys.settrace` so whenever someone else adds a tracer, it doesn't really do `sys.settrace` but just add a function that the real tracer would be calling after it's done tracing. But this can't really be done without the original tracer implementing it, right? On Thu, Apr 25, 2019 at 6:16 PM Ram Rachum <ram@rachum.com> wrote:
Perhaps I misunderstand what's implied by "simple(!) monkeypatch of sys.settrace", but the trickiest part of Ram's proposal is that the body of one trace function would still trigger the remaining trace functions. That to me sounds like it's going to require changes to ceval.c --Ned. On 4/25/19 12:26 PM, Ram Rachum wrote:
It wouldn't be difficult to have a list of trace functions, so that every line of "real" Python executed, would invoke all the trace functions. But Ram has asked for something more: when the first trace function is executing, its line should themselves be traced by the remaining trace functions in the list. Presumably the lines in the second trace function should also be traced by the function third in the list, and so on. This is the thing that will be difficult to accomplish. --Ned. On 4/25/19 2:02 PM, Anders Hovmöller wrote:
On 26 Apr 2019, at 05:47, Ram Rachum <ram@rachum.com> wrote:
Ah, I thought about it now and Ned is right. This would require modifications to ceval.c and others.
Pity!
The question is... Does anyone else think it's a good idea?
I do. It seems to me that coverage is a very useful tool that shouldn’t be unusable for certain programs if we can avoid it. If we should include it in CPython in the end probably depends on how much it complicates the implementation and/or how solid tests are written obviously. I would point out that we can still get another coverage metric for these scenarios though: mutation coverage. But that’s extremely slow to collect compared to traditional coverage. / Anders
Personally I would look to other means to get the coverage report for a tracing tool or debugger. For example why not use unittesting and mocking to allow the trace code to be run and measured? After all you only have to mock for one functions interface. As for debugging I would use print() or logging to find what I need. Barry
It's possible, but it would be very cumbersome, for a bunch of reasons. One of them is that the tracing code inspects the frame, the variables referenced in it, and it even opens the file of the code object of the frame. It will be difficult to mock all of that, and even if that's possible, we won't have high confidence that the mock is reliable. On Sun, Apr 28, 2019, 11:06 Barry Scott <barry@barrys-emacs.org> wrote:
On 28 Apr 2019, at 09:12, Ram Rachum <ram.rachum@gmail.com> wrote:
It's possible, but it would be very cumbersome, for a bunch of reasons. One of them is that the tracing code inspects the frame, the variables referenced in it, and it even opens the file of the code object of the frame. It will be difficult to mock all of that, and even if that's possible, we won't have high confidence that the mock is reliable.
It is not so scary that I would not take it on myself if I had the need. Maybe you use real frames and only fake up the call into the tracecode. What I'm thinking is that it is possible and can be done today without C level changes to Python. Maybe if there are changes to the C python it only needs a second tracehook that runs for the code in the sys.settrace only. Barry
participants (5)
-
Anders Hovmöller -
Barry Scott -
Ned Batchelder -
Ram Rachum -
Ram Rachum