[Python-Dev] s/hotshot/lsprof

[Nick Coghlan]

Jim Jewett wrote:
> Jeremy Hylton jeremy at alum.mit.edu
>>  Should lsprof be added to the standard distribution?
>>  Should hotshot be removed from the standard distribution?
> 
>>  These two aren't at all related, unless you believe that two is the
>>  maximum number of profiles allowed per Python distribution.
> 
> One is a better number.
> 
> ("There should be one-- and preferably only one --obvious way to do it.")
> 
> Adding a second (let alone third) module to the stdlib to do
> the same thing just makes the documentation bulkier,
> and makes the "where do I start" problem harder for beginners.
> 
> And yes, I think beginners are the most important audience
> here; anyone sufficiently comfortable with python to make an
> intelligent choice between different code profilers is probably
> also able to install 3rd-party modules anyway.

Chiming in as a user of 'profile', that has also attempted to use hotshot. . .

I used profile heavily when we working on the implementation of the decimal 
module, trying to figure out where the bottlenecks were (e.g., profile showed 
that converting to integers to do arithmetic and back to sequences to do 
rounding was a net win, despite the conversion costs in switching back and 
forth between the two formats).

I tried using hotshot to do the same thing (profiled runs of the arithmetic 
tests took a *long* time), and found the results to be well-nigh useless (I 
seem to recall it was related to the fact that profile separated out C calls, 
while hotshot didn't).

So my experience of hotshot has been "sure it's slightly less invasive, but it 
doesn't actually work". If hotshot can be replaced with something that 
actually works as intended, or if lsprof can be added in a way that is more 
closely coupled with profile (so that there is a clear choice between "less 
invasive but less detailed results" and "more detailed results but more 
invasive during execution"), I'd be quite happy.

If a statistical profiler was later added to round out the minimally invasive 
end, that actually makes for a decent profiling toolset:

1. Use the statistical profiler to identify potential problem areas
2. Use hotshot/lsprof to further analyse the potential problem areas
3. Use profile to get detailed results on the bottlenecks

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org