Timing hefty or long-running blocks of code
The timeit module is great for timing small code snippets, but it is rather inconvenient for timing larger blocks of code. It's also over-kill: under normal circumstances, there is little need for the heroic measures timeit goes through to accurately time long-running code. When people want to time something that doesn't require timeit, they have to write their own timer boilerplate: from time import time t = time() do_this() do_that() print(time() - t) Although only a few lines of boilerplate, this has a few problems, the biggest two being: * the user has to decide which of the many clocks provided by Python is suitable (hint: time.time is not necessarily the best); * this is inconvenient at the interactive interpreter, since the timer is started before the code being typed has been entered. I propose a context manager in the timeit module: with Stopwatch(): do_this() do_that() => results are automatically printed, or available for programmatic access I have had a recipe for this on ActiveState for about 18 months, where it is relatively popular: http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-stat... and I have extensively used it interactively, in Python 2.6 through 3.3 inclusive. The latest version can be found here: https://code.google.com/p/my-startup-file/source/browse/timer.py This solves the two problems listed above: * the Stopwatch can use the most appropriate timer by default, and allow the user to override it if they so choose; * the Stopwatch will not start until the with-block is actually entered. Is there interest in seeing this in the standard library? -- Steven
On 2013-06-01, at 11:24 , Steven D'Aprano wrote:
The timeit module is great for timing small code snippets, but it is rather inconvenient for timing larger blocks of code. It's also over-kill: under normal circumstances, there is little need for the heroic measures timeit goes through to accurately time long-running code.
Does timeit really go through much heroics when using it in code? The CLI version does a bunch of testing and adaptation to try to get accurate measures but as far as I know the Python API has no such facilities, it runs what you gave it the number of time you tell it. The only "heroics" it goes through is using timeit.default_timer(), which is supposed to provide the most accurate clock available.
When people want to time something that doesn't require timeit, they have to write their own timer boilerplate:
from time import time t = time() do_this() do_that() print(time() - t)
Although only a few lines of boilerplate, this has a few problems, the biggest two being:
* the user has to decide which of the many clocks provided by Python is suitable (hint: time.time is not necessarily the best);
But that's what timeit.default_timer() is for right? This seems to be a problem of awareness more than availability.
* this is inconvenient at the interactive interpreter, since the timer is started before the code being typed has been entered.
I propose a context manager in the timeit module:
with Stopwatch(): do_this() do_that()
=> results are automatically printed, or available for programmatic access
I have had a recipe for this on ActiveState for about 18 months, where it is relatively popular:
http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-stat...
and I have extensively used it interactively, in Python 2.6 through 3.3 inclusive. The latest version can be found here:
https://code.google.com/p/my-startup-file/source/browse/timer.py
This solves the two problems listed above:
* the Stopwatch can use the most appropriate timer by default, and allow the user to override it if they so choose;
* the Stopwatch will not start until the with-block is actually entered.
Is there interest in seeing this in the standard library?
A more general version can be achieved through timeit.Timer, though it's got more boilerplate and as far as I know there's no example of it in the documentation: @timeit.Timer def foo(): do_this() do_that() print foo.timeit(1) On the other hand it allows repeated benching to try and refine the iterations count. Maybe context-management could simply be added to timeit.Timer, rather than be a separate object?
On 06/01/2013 05:50 AM, Masklinn wrote:
A more general version can be achieved through timeit.Timer, though it's got more boilerplate and as far as I know there's no example of it in the documentation:
@timeit.Timer def foo(): do_this() do_that() print foo.timeit(1)
On the other hand it allows repeated benching to try and refine the iterations count.
Maybe context-management could simply be added to timeit.Timer, rather than be a separate object?
I like the decorator variation. +1 for adding it to timeit. I would like to be able to specify the maximum count to time, and to be able to specify where to send the output. function_timer = timeit.FunctionTimer(count=3, file=sys.stderr) @function_timer def foo(): ... Which would print ... Foo timer 1/3: n seconds Foo timer 2/3: n seconds Foo timer 3/3: n seconds It would only time 'count' times through the function. A function timer like this requires minimal changes to the code and can be pasted in or commented out as needed. For me, timing a function is the most common type of timing I do in real programs. Cheers, Ron
On 2013-06-01, at 17:16 , Ron Adam wrote:
On 06/01/2013 05:50 AM, Masklinn wrote:
A more general version can be achieved through timeit.Timer, though it's got more boilerplate and as far as I know there's no example of it in the documentation:
@timeit.Timer def foo(): do_this() do_that() print foo.timeit(1)
On the other hand it allows repeated benching to try and refine the iterations count.
Maybe context-management could simply be added to timeit.Timer, rather than be a separate object?
I like the decorator variation.
+1 for adding it to timeit.
I would like to be able to specify the maximum count to time, and to be able to specify where to send the output.
function_timer = timeit.FunctionTimer(count=3, file=sys.stderr)
@function_timer def foo(): ...
Which would print ...
Foo timer 1/3: n seconds Foo timer 2/3: n seconds Foo timer 3/3: n seconds
It would only time 'count' times through the function.
A function timer like this requires minimal changes to the code and can be pasted in or commented out as needed.
FWIW that can also be achieved by using timeit.Timer as a decorator: simply call Timer.repeat afterwards instead of Timer.timeit, it defaults to repeat=3 (which corresponds to your count=3) and returns a list of results. So @timeit.Timer def foo(): do_this() do_that() print foo.repeat(repeat=3, number=1) will print a list of 3 elements, each of which corresponds to running the function once.
For me, timing a function is the most common type of timing I do in real programs.
All of timeit's stuff can take functions instead of statement strings, both are supported.
On 06/01/2013 01:15 PM, Masklinn wrote:
@function_timer def foo(): ...
Which would print ...
Foo timer 1/3: n seconds Foo timer 2/3: n seconds Foo timer 3/3: n seconds
It would only time 'count' times through the function.
A function timer like this requires minimal changes to the code and can be pasted in or commented out as needed. FWIW that can also be achieved by using timeit.Timer as a decorator: simply call Timer.repeat afterwards instead of Timer.timeit, it defaults to repeat=3 (which corresponds to your count=3) and returns a list of results.
So
@timeit.Timer def foo(): do_this() do_that() print foo.repeat(repeat=3, number=1)
will print a list of 3 elements, each of which corresponds to running the function once.
For me, timing a function is the most common type of timing I do in real programs. All of timeit's stuff can take functions instead of statement strings, both are supported.
Yes, but that is very different in context. A more realistic example would include arguments... @function_timer def foo(*args, **kwds): ... If the args and kwds are generated by other parts of the program, then you would need to create more boiler plate to generate the arguments for the out of context print call. This could get quite involved depending on how complex the arguments are. The advantage is that it gives a more accurate time for a very specific set of arguments. The decorator I suggest gives the times of the actual usage while the program (or module) is running with a minimal amount of changes to the program, and minimal amount of work to get meaningful output. The function timer counter should be called max_count rather than count. If the function is only called once, then you will only get one time for it. If the function is called many times, then you get a *max* of three times in the case max_count is 3. Then it would stop timing that function. And the function_timer decorator can be replaced with a null decorator at the top to disable all of them at once. function_timer = null_decorator # disable this decorator I think this is closer to what Steven was asking for. His with statement example is better suited for targeted timing of particular blocks of code in a program while it's executing. Ron
01.06.13 13:50, Masklinn написав(ла):
A more general version can be achieved through timeit.Timer, though it's got more boilerplate and as far as I know there's no example of it in the documentation:
@timeit.Timer def foo(): do_this() do_that() print foo.timeit(1)
Why not just timeit.Timer(foo).timeit(1)?
On 01/06/13 20:50, Masklinn wrote:
On 2013-06-01, at 11:24 , Steven D'Aprano wrote:
The timeit module is great for timing small code snippets, but it is rather inconvenient for timing larger blocks of code. It's also over-kill: under normal circumstances, there is little need for the heroic measures timeit goes through to accurately time long-running code.
Does timeit really go through much heroics when using it in code? The CLI version does a bunch of testing and adaptation to try to get accurate measures but as far as I know the Python API has no such facilities, it runs what you gave it the number of time you tell it.
Perhaps "heroic" is an exaggeration, but timeit.Timer does more than just run the code snippet. By default it repeats the code one million times, to dissipate any cache effects or other interference; takes efforts to minimize the overhead of the looping construct; it tries to pick the best timer function it can; and it disables garbage collection. Most of these things are unnecessary or even counter-productive when timing a single run of some long-running block of code. [...]
A more general version can be achieved through timeit.Timer, though it's got more boilerplate and as far as I know there's no example of it in the documentation:
@timeit.Timer def foo(): do_this() do_that() print foo.timeit(1)
I'm not suggesting that timeit.Timer go away :-) But I am suggesting that for timing code with a minimum of friction and a maximum of convenience, particularly in the interactive interpreter, a context manager beats Timer, even the decorator version. with Stopwatch(): do_this() do_that() No mess, no fuss, no need to wrap things in an extraneous function, or to use nonlocal (as per Andrew Barnet's email), it just times the code you run. This is not just for benchmarking, sometimes in the interactive interpreter you just want to know how long something will take, in a hand-wavy sense. "It takes about a minute to process this directory, so when I process this other one I'll allow about ten minutes. Time enough for a coffee break!"
On the other hand it allows repeated benching to try and refine the iterations count.
Maybe context-management could simply be added to timeit.Timer, rather than be a separate object?
I had thought about that, but the constructor signatures are just too different. -- Steven
On Jun 1, 2013, at 2:24, Steven D'Aprano <steve@pearwood.info> wrote:
The timeit module is great for timing small code snippets, but it is rather inconvenient for timing larger blocks of code.
If you factor the block out into a function, you can just timeit that function. And often (not always, I realize, but often) the refactoring will be worth doing anyway. After all, you have a block of code that's sufficiently separate to be worth timing separately.
from time import time t = time() do_this() do_that() print(time() - t)
def do_them(): do_this() do_that() print(timeit(do_them, number=1)) This eliminates both of your problems. You don't have to guess which timing function to use, and it doesn't start the timer until the function starts. In 2.x I've used your recipe, because there's no trivial way to handle this: x = do_this() y = do_that() But in 3.x you just just add "nonlocal x, y" inside the function. So, I've rarely used it. (I've also just called timeit(more_functools.sequence(do_this, do_that)), but that isn't very pythonic. If you want a new non-trivial function, it's usually better to just def it.) Again, I realize that sometimes (even in 3.x) it's not appropriate to convey the block into a function. But often enough to be worth adding to the stdlib? I don't know.
2013/6/1 Steven D'Aprano <steve@pearwood.info>:
The timeit module is great for timing small code snippets, but it is rather inconvenient for timing larger blocks of code. It's also over-kill: under normal circumstances, there is little need for the heroic measures timeit goes through to accurately time long-running code. [...] Is there interest in seeing this in the standard library?
A strong +1. This is one of those things I personally need quite often. FWIW In the 'utils' module I use across multiple projects I have a modified version of this: http://dabeaz.blogspot.it/2010/02/function-that-works-as-context-manager.htm... --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/
participants (6)
-
Andrew Barnert -
Giampaolo Rodola' -
Masklinn -
Ron Adam -
Serhiy Storchaka -
Steven D'Aprano