Hi all, I've recently written a C version of the trace function used in figleaf (the coverage tool written by Titus). After a few rewrites to add in caching, etc, it gives users a significant speedup. One person stated that switching to the C version caused coverage to decrease from a 442% slowdown to only a 56% slowdown. You can see my C implementation at: http://github.com/ctb/figleaf/blob/e077155956c288b68704b09889ebcd675ba02240/... (Specific comments about the implementation welcome off-list). I'd like to attempt something similar for bdb.py (only for the trace function). A naive C trace function which duplicated the current python function should speed up bdb significantly. I would initially write the smallest part of the C implementation that I could. Basically the tracing function would call back out to python at any point where a line requires action. I'd be willing to maintain the C implementation. I would be willing to write those tests that are possible as well. Is this something that would be likely to be accepted? Thanks, David Christian Senior Software Engineer rPath, Inc.
2009/4/1 David Christian <david.christian@gmail.com>:
Hi all, I've recently written a C version of the trace function used in figleaf (the coverage tool written by Titus). After a few rewrites to add in caching, etc, it gives users a significant speedup. One person stated that switching to the C version caused coverage to decrease from a 442% slowdown to only a 56% slowdown.
You can see my C implementation at: http://github.com/ctb/figleaf/blob/e077155956c288b68704b09889ebcd675ba02240/...
(Specific comments about the implementation welcome off-list).
I'd like to attempt something similar for bdb.py (only for the trace function). A naive C trace function which duplicated the current python function should speed up bdb significantly. I would initially write the smallest part of the C implementation that I could. Basically the tracing function would call back out to python at any point where a line requires action.
I'd be willing to maintain the C implementation. I would be willing to write those tests that are possible as well.
Is this something that would be likely to be accepted?
Generally debugging doesn't require good performance, so this is definitely low priority. However, if you can contribute it, I don't have a problem with it. -- Regards, Benjamin
On Wed, Apr 1, 2009 at 3:25 PM, Benjamin Peterson <benjamin@python.org> wrote:
2009/4/1 David Christian <david.christian@gmail.com>:
Hi all, I've recently written a C version of the trace function used in figleaf (the coverage tool written by Titus). After a few rewrites to add in caching, etc, it gives users a significant speedup. One person stated that switching to the C version caused coverage to decrease from a 442% slowdown to only a 56% slowdown.
You can see my C implementation at: http://github.com/ctb/figleaf/blob/e077155956c288b68704b09889ebcd675ba02240/...
(Specific comments about the implementation welcome off-list).
I'd like to attempt something similar for bdb.py (only for the trace function). A naive C trace function which duplicated the current python function should speed up bdb significantly. I would initially write the smallest part of the C implementation that I could. Basically the tracing function would call back out to python at any point where a line requires action.
I'd be willing to maintain the C implementation. I would be willing to write those tests that are possible as well.
Is this something that would be likely to be accepted?
Generally debugging doesn't require good performance, so this is definitely low priority. However, if you can contribute it, I don't have a problem with it.
Tracing has other uses besides debugging though. In particular, coverage, which usually wants per-line data. Also, sometimes if you set a breakpoint in a function it turns on tracing for the entire function. This can sometimes be annoyingly slow. So, personally, I am more positive than that, and hope it will make it in. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 2009-04-01 17:53, Benjamin Peterson wrote:
2009/4/1 Guido van Rossum<guido@python.org>:
Tracing has other uses besides debugging though.
The OP said he wished to implement a C trace function for bdb. Wouldn't that make it only applicable to debugging?
Once you are at the breakpoint and stepping through the code manually, the performance is not all that important. However, up until that breakpoint, you are running a lot of code "in bulk". It would be useful to have a performant trace function that interferes with that code the least. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Wed, Apr 1, 2009 at 6:53 PM, Benjamin Peterson <benjamin@python.org> wrote:
2009/4/1 Guido van Rossum <guido@python.org>:
Tracing has other uses besides debugging though.
The OP said he wished to implement a C trace function for bdb. Wouldn't that make it only applicable to debugging?
Benjamin
I was suggesting a speedup for debugging. However, I could certainly also contribute my figleaf work that I referenced earlier, with a few tweaks, as a tracing replacement for the tracing function in trace.py. My concern with moving the coverage tracing code in particular to the standard library is that it tries to extract the maximum speed by being clever*, and certainly has not been out in the wild for long enough. I would write something much more conservative as a starting point for bdb.py. I expect that any C implementation that was thinking about performance at all would be much better than the status quo. * figleaf checks a regular expression to determine whether or not we wish to trace a particular file. If the file is not being traced, I switch to the profiler instead of the line tracer, which means that the trace function only gets called twice per function instead of once per line. This can give a large speedup when you are skipping the entire standard library, at some measurable cost per function call, and a cost in code complexity. --- David Christian Senior Software Engineer rPath, Inc
On Wed, Apr 1, 2009 at 3:53 PM, Benjamin Peterson <benjamin@python.org> wrote:
2009/4/1 Guido van Rossum <guido@python.org>:
Tracing has other uses besides debugging though.
The OP said he wished to implement a C trace function for bdb. Wouldn't that make it only applicable to debugging?
I honestly don't recall, but I believe pretty much everyone who uses tracing does so via bdb.py. And yes, when debugging sometimes you have to silently skip 1000 iterations until a condition becomes true, and the tracking speed matters. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (4)
-
Benjamin Peterson -
David Christian -
Guido van Rossum -
Robert Kern