[Python-Dev] Unifying trace and profile

[Nicholas Bastin]

On 2/21/06, Robert Brewer <fumanchu at amor.org> wrote:
> 1. Allow trace hooks to receive c_call, c_return, and c_exception events
> (like profile does).

I can easily make this modification.  You can also register the same
bound method for trace and profile, which sortof eliminates this
problem.

> 2. Allow profile hooks to receive line events (like trace does).

You really don't want this in the general case.  Line events make
profiling *really* slow, and they're not that accurate (although many
thanks to Armin last year for helping me make them much more
accurate).  I guess what you require is to be able to selectively turn
on events, thus eliminating the notion of 'trace' or 'profile'
entirely, but I don't have a good idea of how to implement that at
least as efficiently as the current system at the moment - I'm sure it
could be done, I just haven't put any thought into it.

> 3. Expose new sys.gettrace() and getprofile() methods, so trace and
> profile functions that want to play nice can call
> sys.settrace/setprofile(None) only if they are the current hook.

Not a bad idea, although are you really running into this problem a lot?

> 4. Make "the same move" that sys.exitfunc -> atexit made (from a single
> function to multiple functions via registration), so multiple
> tracers/profilers can play nice together.

It seems very unlikely that you'll want to have a trace hook and
profile hook installed at the same time, given the extreme
unreliability this will introduce into the profiler.

> 5. Allow the core to filter on the "event" arg before hook(frame, event,
> arg) is called.

What do you mean by this, exactly?  How would you use this feature?

> 6. Unify tracing and profiling, which would remove a lot of redundant
> code in ceval and sysmodule and free up some space in the PyThreadState
> struct to boot.

The more events you throw in profiling makes it slow, however.  Line
events, while a nice thing to have, theoretically, would probably make
a profiler useless.  If you want to create line-by-line timing data,
we're going to have to look for a more efficient way (like sampling).

> 7. As if the above isn't enough of a dream, it would be nice to have a
> bytecode tracer, which didn't bother with the f_lineno logic in
> maybe_call_line_trace, but just called the hook on every instruction.

I'm working on one, but given how much time I've had to work on my
profiler in the last year, I'm not even going to guess when I'll get a
real shot at looking at that.

My long-term goal is to eliminate profiling and tracing from the core
interpreter entirely and implement the functionality in such a way
that they don't cost you when not in use (i.e., implement profilers
and debuggers which poke into the process from the outside, rather
than be supported natively through events).  This isn't impossible,
but it's difficult because of the large variety of platforms.  I have
access to most of them, but again, my time is hugely constrained right
now for python development work.

--
Nick