bcannon at gmail.com
Sun Nov 20 01:12:28 CET 2005
Just for everyone's FYI while we are talking about profilers, Floris
Bruynooghe (who I am cc'ing on this so he can contribute to the
conversation), for Google's Summer of Code, wrote a replacement for
'profile' that uses Hotshot directly. Thanks to his direct use of
Hotshot and rewrite of pstats it loads Hotshot data 30% faster and
also alleviates keeping 'profile' around and its slightly questionable
You can find his project at
http://savannah.nongnu.org/projects/pyprof/ . I believe he also
tweaked Hotshot to accept custom timing functions. I have not had a
chance to go over his code to clean it up for putting it up on SF, but
I thought people should be aware of it.
On 11/19/05, Armin Rigo <arigo at tunes.org> wrote:
> The current Python profilers situation is a mess.
> 'profile.Profile' is the ages-old pure Python profiler. At the end of a
> run, it builds a dict that is inspected by 'pstats.Stats'. It has some
> recent support for profiling C calls, which however make it crash in
> some cases . And of course it's slow (makes a run take about 10x
> 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added
> overhead). The log file is then loaded and turned into an instance of
> the same 'pstats.Stats'. This loading takes ages. The reason is that
> the log file only records events, and loading is done by instantiating a
> 'profile.Profile' and sending it all the events. In other words, it
> takes exactly as long as the time it spared in the first place!
> Moreover, for some reasons, the results given by hotshot seem sometimes
> quite wrong. (I don't understand why, but I've seen it myself, and it's
> been reported by various people, e.g. .) 'hotshot' doesn't know
> about C calls, but it can log line events, although this information is
> lost(!) in the final conversion to a 'pstats.Stats'.
> 'lsprof' is a third profiler by Brett Rosen and Ted Czotter, posted on
> SF in June . Michael Hudson and me did some minor clean-ups and
> improvements on it, and it seems to be quite useful. It is, for
> example, the only of the three profilers that managed to give sensible
> information about the PyPy translation process without crashing,
> allowing us to accelerate it from over 30 to under 20 minutes. The SF
> patch contains a more detailed account on the reasons for writing
> 'lsprof'. The current version  does not support C calls nor line
> events. It has its own simple interface, which is not compatible with
> any of the other two profilers. However, unlike the other two
> profilers, it can record detailed stats about children, which I found
> quite useful (e.g. how much take is spent in a function when it is
> called by another specific function).
> Therefore, I think it would be a great idea to add 'lsprof' to the
> standard library. Unless there are objections, it seems that the best
> plan is to keep 'profile.py' as a pure Python implementation and replace
> 'hotshot' with 'lsprof'. Indeed, I don't see any obvious advantage that
> 'hotshot' has over 'lsprof', and I certainly see more than one downside.
> Maybe someone has a use for (and undocumented ways to fish for) line
> events generated by hotshot. Well, there is a script  to convert
> hotshot log files to some format that a KDE tool  can display. (It
> even looks like hotshot files were designed with this in mind.) Given
> that the people doing that can still compile 'hotshot' as a separate
> extension module, it doesn't strike me as a particularly good reason to
> keep Yet Another Profiler in the standard library.
> So here is my plan:
> Unify a bit more the interfaces of the pure Python and the C profilers.
> This also means that 'lsprof' should be made to use a pstats-compatible
> log format. The 'pstats' documentation specifically says that the file
> format can change: that would give 'lsprof' a place to store its
> detailed children stats.
> Then we can provide a dummy 'hotshot.py' for compatibility, remove its
> documentation, and provide documentation for 'lsprof'.
> If anyone feels like this is a bad idea, please speak up.
> A bientot,
>  https://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=1117670
>  http://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=1212837
>  http://codespeak.net/svn/user/arigo/hack/misc/lsprof (Subversion)
>  http://mail.python.org/pipermail/python-list/2003-September/183887.html
>  http://kcachegrind.sourceforge.net/cgi-bin/show.cgi
> Python-Dev mailing list
> Python-Dev at python.org
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
More information about the Python-Dev