[issue9609] make cProfile multi-stack aware
Kristján Valur Jónsson
report at bugs.python.org
Sun Aug 15 17:09:36 CEST 2010
New submission from Kristján Valur Jónsson <kristjan at ccpgames.com>:
One of the problems with the profiling modules provided with Python is that they are not useful in the presence of multiple threads. This is because time spent in a different thread may be falsely attributed to some random place in a thread being profiled.
This patch helps fix that, by making the _lsprof.c module (the engine behind cProfile) multi-stack aware. At every entry into the profiler, a check is made to see which stack is in operation (by looking at the thread state). The previous stack is then paused and profiling commences on the new stack.
Time spent on other stacks is then subtracted from the measured time on each stack.
A complication arises because it is no longer possible to determine the recursion level of each function (or subcall instance) on each stack by looking at the function's entry alone. For this reason, it becomes necessary to walk the stack in cases where there are multiple stacks and multiple total recursions seen for the entries.
This patch has been successfully used, with a modifiaction for stackless python, in production at CCP (the modification uses the Tasklet ID rather than the TLS pointer as a key to the stack map).
To be useful, it is important that all threads in the process are set to use the same cProfile.Profiler() instance. Currently there is no easy way to do that and this patch doesn't attempt to fix that. But is is possible that an application designed for profiling would attach the profiler at each thread start point. (In the version of Stackless Python that this is used on, it is possible to enable tracing/profiling of all tasklets simultaneously)
components: Extension Modules
keywords: patch, patch
title: make cProfile multi-stack aware
type: feature request
versions: Python 3.2
Added file: http://bugs.python.org/file18539/_lsprof.patch
Python tracker <report at bugs.python.org>
More information about the Python-bugs-list