[Python-ideas] Timing hefty or long-running blocks of code

[Steven D'Aprano]

On 02/06/13 14:15, Clay Sweetser wrote:
>      I quite like this proposal. To me, the main advantage of such a context
> manager/decorator over the current timeit module is that it can be used for
> pieces of code that either can't or shouldn't be arbitrarily repeated.
>      For example, I am currently writing a plugin for the sublime text
> editor, and it would be very convenient for me to be able to get a quick
> summery of timing statistics for functions that make use of the editor's
> plugin api to modify documents.
>      My only suggestion for the proposed function is to allow the output of
> statistics similar to the ones the timeit module allows, such as the mean
> runtime of a block of code.

While I don't wish to discourage anyone with the good sense to agree with my proposal *wink*, in fairness I should say a couple of things:

* A context manager is not capable of running the body of the
   block multiple times automatically. If you want to do that,
   you need to wrap it in a for-loop, or just use timeit.Timer.

* Even when you run multiple trials, say using Timer.repeat(),
   taking the mean is statistically the wrong thing to do.

The docs already say this:

[quote]
It’s tempting to calculate mean and standard deviation from the result vector and report these. However, this is not very useful. In a typical case, the lowest value gives a lower bound for how fast your machine can run the given code snippet; higher values in the result vector are typically not caused by variability in Python’s speed, but by other processes interfering with your timing accuracy. So the min() of the result is probably the only number you should be interested in. After that, you should look at the entire vector and apply common sense rather than statistics.
[end quote]

http://docs.python.org/2/library/timeit.html#timeit.Timer.repeat


To put it another way, you can assume that each timeit result is made up of two components:

- the "actual" or "real" time that the code would take in a perfect world where no other processes are running and your code is run at the highest speed possible;

- plus some unknown error, due to external factors, overhead of the timing code, etc.

Note that the error is always strictly positive, never negative or zero. So the min() of the timing results is the best estimate of the "actual" time. Any other estimate, including mean or median, will be higher than the minimum, and therefore less accurate.



-- 
Steven