Testing for performance regressions

[Steven D'Aprano]

On Mon, 04 Apr 2011 20:59:52 -0700, geremy condra wrote:

> On Mon, Apr 4, 2011 at 7:45 PM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:

>> * The disclaimers about timing code snippets that can be found in the
>> timeit module apply. If possible, use timeit rather than roll-you-own
>> timers.
> 
> Huh. In looking into timing attacks actually one of the biggest lessons
> I learned was *not* to use timeit- that the overhead and variance
> involved in using it will wind up consuming small changes in behavior in
> ways that are fairly opaque until you really take it apart.

Do you have more details?

I follow the advice in the timeit module, and only ever look at the 
minimum value, and never try to calculate a mean or variance. Given the 
number of outside influences ("What do you mean starting up a browser 
with 200 tabs at the same time will affect the timing?"), I wouldn't 
trust a mean or variance to be meaningful.



-- 
Steven