Oh wow, I didn't even consider that. I think you're right, I'll do more thinking about this. Thanks Anders! On Thu, Apr 25, 2019 at 6:10 PM Anders Hovmöller wrote:

Can't this be implemented today by a simple monkey patch of sys.settrace?

On 25 Apr 2019, at 16:51, Ram Rachum wrote:

Hi,

Here's something I want in Python: Multiple levels of tracers working on top of each other, instead of just one.

I'm talking about the tracer that one can set by calling sys.settrace.

I've recently released PySnooper: https://github.com/cool-RR/PySnooper/

One of the difficulties I have, is that I can't debug or run the `coverage` tool on the core of this module. That's because the core is a trace function, and debuggers and coverage tools work by setting a trace function. When PySnooper sets its trace function using `sys.settrace`, the code that runs in that trace function runs without getting traced by the coverage tracer.

This means that people who develop debuggers and coverage tools can't use a debugger or a coverage tool on the core of their tool. It's quite an annoying problem.

My proposed solution: Multiple levels of tracing, instead of just one. When you install a tracer, you're not replacing the existing one, you're appending a tracer to the existing list of tracers.

If this was implemented, then when PySnooper would install its tracer, the coverage tracer would still be active and running, for every line of code including the ones in PySnooper's tracer.

Obviously, we'll need to figure out the API and any other kind of problems with this proposal.

What do you think?

Thanks, Ram.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Perhaps I misunderstand what's implied by "simple(!) monkeypatch of sys.settrace", but the trickiest part of Ram's proposal is that the body of one trace function would still trigger the remaining trace functions.  That to me sounds like it's going to require changes to ceval.c --Ned. On 4/25/19 12:26 PM, Ram Rachum wrote:

Hmm, looks like, for this to work, you'll need the existing tracer to be cooperative. Right now there are existing tracers, for example coverage's tracer and Wing IDE's tracer, and I would need to modify them for your idea to work, right?

If I understand your idea correctly, the first tracer would monkeypatch `sys.settrace` so whenever someone else adds a tracer, it doesn't really do `sys.settrace` but just add a function that the real tracer would be calling after it's done tracing. But this can't really be done without the original tracer implementing it, right?

On Thu, Apr 25, 2019 at 6:16 PM Ram Rachum mailto:ram@rachum.com> wrote:

Oh wow, I didn't even consider that. I think you're right, I'll do more thinking about this. Thanks Anders!

On Thu, Apr 25, 2019 at 6:10 PM Anders Hovmöller mailto:boxed@killingar.net> wrote:

Can't this be implemented today by a simple monkey patch of sys.settrace?

On 25 Apr 2019, at 16:51, Ram Rachum mailto:ram@rachum.com> wrote:

...

Hi,

Here's something I want in Python: Multiple levels of tracers working on top of each other, instead of just one.

I'm talking about the tracer that one can set by calling sys.settrace.

I've recently released PySnooper: https://github.com/cool-RR/PySnooper/

One of the difficulties I have, is that I can't debug or run the `coverage` tool on the core of this module. That's because the core is a trace function, and debuggers and coverage tools work by setting a trace function. When PySnooper sets its trace function using `sys.settrace`, the code that runs in that trace function runs without getting traced by the coverage tracer.

This means that people who develop debuggers and coverage tools can't use a debugger or a coverage tool on the core of their tool. It's quite an annoying problem.

My proposed solution: Multiple levels of tracing, instead of just one. When you install a tracer, you're not replacing the existing one, you're appending a tracer to the existing list of tracers.

If this was implemented, then when PySnooper would install its tracer, the coverage tracer would still be active and running, for every line of code including the ones in PySnooper's tracer.

Obviously, we'll need to figure out the API and any other kind of problems with this proposal.

What do you think?

Thanks, Ram.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org mailto:Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Well, it would trigger the top level chaining trace function, but they should be able to decide when to call the sub-trace functions. Hmm... Maybe :)

On 25 Apr 2019, at 19:16, Ned Batchelder wrote:

Perhaps I misunderstand what's implied by "simple(!) monkeypatch of sys.settrace", but the trickiest part of Ram's proposal is that the body of one trace function would still trigger the remaining trace functions. That to me sounds like it's going to require changes to ceval.c

--Ned.

...

On 4/25/19 12:26 PM, Ram Rachum wrote: Hmm, looks like, for this to work, you'll need the existing tracer to be cooperative. Right now there are existing tracers, for example coverage's tracer and Wing IDE's tracer, and I would need to modify them for your idea to work, right?

If I understand your idea correctly, the first tracer would monkeypatch `sys.settrace` so whenever someone else adds a tracer, it doesn't really do `sys.settrace` but just add a function that the real tracer would be calling after it's done tracing. But this can't really be done without the original tracer implementing it, right?

...

On Thu, Apr 25, 2019 at 6:16 PM Ram Rachum wrote: Oh wow, I didn't even consider that. I think you're right, I'll do more thinking about this. Thanks Anders!

...

On Thu, Apr 25, 2019 at 6:10 PM Anders Hovmöller wrote: Can't this be implemented today by a simple monkey patch of sys.settrace?

On 25 Apr 2019, at 16:51, Ram Rachum wrote:

...

Hi,

Here's something I want in Python: Multiple levels of tracers working on top of each other, instead of just one.

I'm talking about the tracer that one can set by calling sys.settrace.

I've recently released PySnooper: https://github.com/cool-RR/PySnooper/

One of the difficulties I have, is that I can't debug or run the `coverage` tool on the core of this module. That's because the core is a trace function, and debuggers and coverage tools work by setting a trace function. When PySnooper sets its trace function using `sys.settrace`, the code that runs in that trace function runs without getting traced by the coverage tracer.

This means that people who develop debuggers and coverage tools can't use a debugger or a coverage tool on the core of their tool. It's quite an annoying problem.

My proposed solution: Multiple levels of tracing, instead of just one. When you install a tracer, you're not replacing the existing one, you're appending a tracer to the existing list of tracers.

If this was implemented, then when PySnooper would install its tracer, the coverage tracer would still be active and running, for every line of code including the ones in PySnooper's tracer.

Obviously, we'll need to figure out the API and any other kind of problems with this proposal.

What do you think?

Thanks, Ram.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Ah, I thought about it now and Ned is right. This would require modifications to ceval.c and others. The question is... Does anyone else think it's a good idea? On Fri, Apr 26, 2019 at 12:31 AM Ned Batchelder wrote:

It wouldn't be difficult to have a list of trace functions, so that every line of "real" Python executed, would invoke all the trace functions. But Ram has asked for something more: when the first trace function is executing, its line should themselves be traced by the remaining trace functions in the list. Presumably the lines in the second trace function should also be traced by the function third in the list, and so on. This is the thing that will be difficult to accomplish.

--Ned. On 4/25/19 2:02 PM, Anders Hovmöller wrote:

Well, it would trigger the top level chaining trace function, but they should be able to decide when to call the sub-trace functions. Hmm... Maybe :)

On 25 Apr 2019, at 19:16, Ned Batchelder wrote:

Perhaps I misunderstand what's implied by "simple(!) monkeypatch of sys.settrace", but the trickiest part of Ram's proposal is that the body of one trace function would still trigger the remaining trace functions. That to me sounds like it's going to require changes to ceval.c

--Ned. On 4/25/19 12:26 PM, Ram Rachum wrote:

Hmm, looks like, for this to work, you'll need the existing tracer to be cooperative. Right now there are existing tracers, for example coverage's tracer and Wing IDE's tracer, and I would need to modify them for your idea to work, right?

If I understand your idea correctly, the first tracer would monkeypatch `sys.settrace` so whenever someone else adds a tracer, it doesn't really do `sys.settrace` but just add a function that the real tracer would be calling after it's done tracing. But this can't really be done without the original tracer implementing it, right?

On Thu, Apr 25, 2019 at 6:16 PM Ram Rachum wrote:

...

Oh wow, I didn't even consider that. I think you're right, I'll do more thinking about this. Thanks Anders!

On Thu, Apr 25, 2019 at 6:10 PM Anders Hovmöller wrote:

...

Can't this be implemented today by a simple monkey patch of sys.settrace?

On 25 Apr 2019, at 16:51, Ram Rachum wrote:

Hi,

Here's something I want in Python: Multiple levels of tracers working on top of each other, instead of just one.

I'm talking about the tracer that one can set by calling sys.settrace.

I've recently released PySnooper: https://github.com/cool-RR/PySnooper/

One of the difficulties I have, is that I can't debug or run the `coverage` tool on the core of this module. That's because the core is a trace function, and debuggers and coverage tools work by setting a trace function. When PySnooper sets its trace function using `sys.settrace`, the code that runs in that trace function runs without getting traced by the coverage tracer.

This means that people who develop debuggers and coverage tools can't use a debugger or a coverage tool on the core of their tool. It's quite an annoying problem.

My proposed solution: Multiple levels of tracing, instead of just one. When you install a tracer, you're not replacing the existing one, you're appending a tracer to the existing list of tracers.

If this was implemented, then when PySnooper would install its tracer, the coverage tracer would still be active and running, for every line of code including the ones in PySnooper's tracer.

Obviously, we'll need to figure out the API and any other kind of problems with this proposal.

What do you think?

Thanks, Ram.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-ideas mailing listPython-ideas@python.orghttps://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 28 Apr 2019, at 09:12, Ram Rachum wrote:

It's possible, but it would be very cumbersome, for a bunch of reasons. One of them is that the tracing code inspects the frame, the variables referenced in it, and it even opens the file of the code object of the frame. It will be difficult to mock all of that, and even if that's possible, we won't have high confidence that the mock is reliable.

It is not so scary that I would not take it on myself if I had the need. Maybe you use real frames and only fake up the call into the tracecode. What I'm thinking is that it is possible and can be done today without C level changes to Python. Maybe if there are changes to the C python it only needs a second tracehook that runs for the code in the sys.settrace only. Barry