[Python-Dev] timeit module

[Connelly Barnes]

Connelly Barnes wrote:
>> Hi,
>> 
>> Perhaps I am the only one bothered by the timeit
>> module, but it seems poorly designed to me.
>> 
>> First of all, it should use a geometric series with a
>> timeout value to detect how many iterations it should
>> perform.  Currently, the user is required to manually
>> specify the number of iterations (the default is 1
>
>The provision of a default value generally seems to chop the legs from

>your argument that the number of iterations is required.

Suppose we wish to time two functions: f1 and f2.  Function f1 takes
1e-5
seconds to run, and function f2 takes 0.1 seconds to run.  If these
functions
are defined in interactive mode, we can time them with timeit via:

>>> timeit.Timer('f1()', 'from __main__ import
f1').timeit(1000000)/1000000
>>> timeit.Timer('f2()', 'from __main__ import f2').timeit(10)/10

The first line gives the approximate time to run f1 and the second line
gives
the approximate time to run f2.  The number parameter to timeit() must
be
supplied for the second line, since the default 1 million iterations
would
take too long to be practical.  The number parameter to timeit() is
also
supplied for the first line, since it makes the code more robust in
case
the default argument value in timeit() is changed.

My first observation is that this is a lot of contortion to get the
time of
functions f1() and f2().  Something simpler would be nice.  For
example,
using Recipe 440657 this can be simplified to:

>>> pytime.pytime(f1, ())
>>> pytime.pytime(f2, ())

Here the timing routine calculates the number of iterations to make,
and
the timing routine divides by the number of iterations of the supplied
callable.

This is also convenient if you move from a computer to one that is
faster or slower.  If you move to a computer that is 10 times slower,
then the timeit.Timer() code above will take 10 times as long to run.
The usual solution is for the programmer to adjust the number of
iterations as he changes computers (so he doesn't wait around all day
for his benchmarks to run).  Likewise, if one optimizes f2() so that
it runs 100 times faster, one would again have to adjust the number
of iterations in order to gain an accurate timing result (if the total
time spent in the timing routine is less than 1 msec, then the
resulting time for f2() is off by more than 10% on my Windows machine,
which is undesirable, so the number of iterations cannot be made too
small).

Summary:
Module timeit is complicated to use on an existing function, it does
not time the number of seconds to execute 1 iteration of the function
(this is the value one usually wants in benchmarking, or its
reciprocal), and it requires that the end user tweak the number of
iterations as he or she changes between computers with
different CPU speed, or as he or she optimizes code.  It is easy to
make a cleaner module that fixes these problems.  It might be good
to incorporate such a module into Python, due to its general
usefulness.

>> million).  If the user optimizes his or her code, then
>> the number of iterations must be changed.  If the user
>> moves to a slower or faster computer, then the number
>> of iterations must be changed again.  This is
>> annoying.
>> 
>What? The purpose of timeit is to give an approximation to the length
of 
>time to run a specific piece ofcode.
>
>Why must the number of iterations be changed when moving to a slower
or 
>faster computer?
>
>> Secondly, there should be a way to time a callable
>> directly.  That is, without finding the string name of
>> the callable and using a string "import X" statement. 
>> These contortions violate rules #1 and #3 of the Zen
>> of Python.
>> 
>Presumably for benchmarking purposes the function call overhead would
be 
>present for all compaered techniques. Do you mean rules #0 and #2?

I mean "Beautiful is better than ugly" and "Simple is better than
complex."


> > [...]
>regards
>  Steve
>-- 
>Steve Holden       +44 150 684 7255  +1 800 494 3119
>Holden Web LLC                     www.holdenweb.com
>PyCon TX 2006                  www.python.org/pycon/


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com