[Python-Dev] s/hotshot/lsprof

[Brett Cannon]

On 11/21/05, Armin Rigo <arigo at tunes.org> wrote:
> Hi Martin,
>
> On Mon, Nov 21, 2005 at 08:12:53AM +0100, "Martin v. L?wis" wrote:
> > If bugs are in the heavily-used parts of the library, like regular
> > expressions, it doesn't matter much if the original author goes
> > away for some period of time - other contributors will fix the bugs
> > that they care about, and not by rewriting the entire thing.
>
> I see no incremental way of fixing some of the downsides of hotshot,
> like its huge log file size and loading time.  I doubt people often find
> the motivation to dig into this large orphaned piece of software.
> Instead, they rewrite their own profilers, because writing a basic one
> is not difficult.  It is much less difficult than, say, writing a basic
> regular expression engine (but even the latter has gotten rewritten at
> times) -- unless you want to go into the advanced corners mentioned by
> Tim.
>
> Some guys posted their 'lsprof' on SF because it was well-polished and
> they found it useful, so here I am, arguing for a standard library
> containing preferably simple pieces of code that work and are practical
> for the common advertised use case.  I'm not even sure in this case why
> we are arguing: the new piece of code's interface can be made 100%
> compatible with the documented parts of the previous interface; the
> previous module has been around for longer but so far it produced
> half-meaningless numbers due to bugs.
>

Just because it is starting to feel like the objections are getting
spread out amongst various parts of this thread, I want to try to
summarize them as I remember them and give my input on them.

So one objection seems to be the question of maintenance.  Who is
going to keep this code updated and running?  As has been pointed out,
Hotshot is not perfect and its development basically stopped.  So
people being a little on edge about yet another profiler that might
not be maintained seems reasonable.

But this worry, in my mind, is alleviated since I believe both Michael
and Armin are willing to maintain the code.  With them both willing to
make sure it stays working (which is a pretty damn good commitment
since we have two core developers willing to keep this going and not
just one) I think this worry is dealt with.

The other issue seems to be some people wanting to keep Hotshot around
for a few releases until lsprof can prove its worth.  I believe this
is what Barry is asking for.  Now Armin has said that a wrapper around
lsprof can be written that will match Hotshot's public API so its need
is not there if lsprof works and the wrapper is good.

If it wasn't Armin or someone else whose opinion I trusted, I would
say go ahead and keep Hotshot around and then eventually do the
wrapper.  But since it is Armin making this claim and the PyPy team
uses this thing (who has several members who I think know what they
are doing  =)  I have faith in them coming up with a good wrapper. 
Thus I say removing Hotshot is fine.

Lastly, there is the argument of whether we should even include a
profiler.  Personally I say yes.  It is another battery that is rather
nice.  I think if the profiler finally had a good reputation of being
accurate and useful it would get more play in the real world.  Plus we
already include other development tools such as IDLE with Python so it
seems fitting to include other dev tools when we have the code and a
maintenance commitment.

In other words, I say let Armin and Michael add lsprof and the
wrappers for it (all while removing any redundant profilers that they
have wrappers for) with them knowing we will have a public stoning at
PyCon the instant they don't keep it all working.  =)

-Brett