[Python-Dev] s/hotshot/lsprof

[Tim Peters]

[Armin Rigo]
...
> ...
> 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added
> overhead).  The log file is then loaded and turned into an instance of
> the same 'pstats.Stats'.  This loading takes ages.  The reason is that
> the log file only records events, and loading is done by instantiating a
> 'profile.Profile' and sending it all the events.  In other words, it
> takes exactly as long as the time it spared in the first place!

We should note that hotshot didn't intend to reduce total time
overhead.  What it's aiming at here is to be less disruptive (than
profile.py) to the code being profiled _while_ that code is running. 
On modern boxes, any kind of profiling gimmick has the unfortunate
side effect of _changing_ the runtime behavior of the code being
profiled, at least by polluting I and D caches with droppings from the
profiling code itself (or, in the case of profile.py, possibly
overwhelming I and top-level D caches -- and distorting non-profiling
runtime so badly that, e.g., networked apps may end up taking entirely
different code paths).

hotshot tries to stick with tiny little C functions that pack away a
tiny amount of data each time, and avoid memory alloc/dealloc, to try
to minimize this disruption.  It looked like it was making real
progress on this at one time ;-)

> Moreover, for some reasons, the results given by hotshot seem sometimes
> quite wrong.  (I don't understand why, but I've seen it myself, and it's
> been reported by various people, e.g. [2].)  'hotshot' doesn't know
> about C calls, but it can log line events, although this information is
> lost(!) in the final conversion to a 'pstats.Stats'.

Ya, hotshot isn't finished.  It had corporate support for its initial
development, but lost that, and became an orphan then.  That's the
eventual fate of most profilers, alas.  They're fiddly, difficult, and
always wrong in some respect.  Because of this, the existence of an
eager maintainer without a real life is more important than the code
;-).