Hi, Python ships with a profiler module which, unfortunately, is almost useless in a multi-threaded environment. * I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users). It implements two modes: a "sampling" (does CPU time accounting based on stack fraames 100 times per second, by default) and a "deterministic" profiler (does CPU time accounting on each function call/return, based on sys.profiler interface). The deterministic profiler is currently implemented in pure python (except for taskstats interface) and much slower than the sampling profiler. Usage (don't forget make to build the c module): python
from Profiler import * def f(): do_something() sampling_profiler(f) or deterministic_profiler(f)
Output is currently in the form of annotated source code (xyz.py.html, in the same directory where xyz.py resides). Before the *_profiler function returns, it iterates over all code objects it encountered and annotates the source files with 2 columns in front: - 1st column: real time - 2nd column: CPU time numbers are log2(time_in_ns), colors are green-to-yellow for below-average and yellow-to-red for above-average metrics (relative to the average metric for all lines of the code object with a metric > 0). Is there common need for such a module? Is it possible to have this included in the standard cPython distribution? Which functional changes (besides a modification of the annotation output which shouldn't spread its result all over the FS) would be required to get this included? Which non-functional changes would be required to get this included? Please direct traffic regarding this subject to pyprof-devel@lists.sourceforge.net (no I'm not subscribed to python-dev). SF project page: https://sourceforge.net/projects/pyprof/ git repository: git://pyprof.git.sourceforge.net/gitroot/pyprof Regards, Christian *) to be more exact there are at least three profiler modules: profile, cProfile, and hotshot, while I did only try (and failed) to use profile in a multi-threaded environment (by manually setting threading.profile to the profiling function), glancing at the source, I'm pretty sure that cProfile behaves similarly; I didn't test the hotshot module, but it does some other trade-offs (space-for-time), so I think that "pyprof" still adds some value
On Mon, May 04, 2009, Christian Schubert wrote:
Python ships with a profiler module which, unfortunately, is almost useless in a multi-threaded environment. *
I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users). It implements two modes: a "sampling" (does CPU time accounting based on stack fraames 100 times per second, by default) and a "deterministic" profiler (does CPU time accounting on each function call/return, based on sys.profiler interface). The deterministic profiler is currently implemented in pure python (except for taskstats interface) and much slower than the sampling profiler.
If you want to discuss this, please subscribe to python-ideas and repost your message. Generally speaking, in order to include modules like this, they need to prove themselves over time and may require PEP approval. If you choose to move the discussion to python-ideas, it would help if you mention known uses of your module. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan
Christian Schubert wrote:
I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users).
One of the uses for a profiling module is to compare runs on various platforms. And please, stop perpetuating the myth that only end-users use anything but Linux. Robert Brewer fumanchu@aminus.org
Hi, Christian. Christian Schubert <mail@apexo.de> wrote:
I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users).
A surprisingly large # of developers are running on OS X these days, though. I suggest make it work there, too. Bill
2009/5/4 Bill Janssen <janssen@parc.com>:
Hi, Christian.
Christian Schubert <mail@apexo.de> wrote:
I've created an alternative profiler module which queries per-thread CPU usage via netlink/taskstats, which limits the applicability to Linux (which shouldn't be much of an issue, profiling is usually not done by end users).
A surprisingly large # of developers are running on OS X these days, though. I suggest make it work there, too.
And Windows. I doubt that the various Windows-specific modules available were developed on Linux. And I wouldn't assume that all of the platform-neutral modules are developed on Linux, or even that the developers have access to Linux. (I know I don't, short of building a brand new virtual machine...) Paul.
participants (5)
-
Aahz -
Bill Janssen -
Christian Schubert -
Paul Moore -
Robert Brewer