Testing for performance regressions
steve+comp.lang.python at pearwood.info
Tue Apr 5 07:25:43 CEST 2011
On Mon, 04 Apr 2011 20:59:52 -0700, geremy condra wrote:
> On Mon, Apr 4, 2011 at 7:45 PM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>> * The disclaimers about timing code snippets that can be found in the
>> timeit module apply. If possible, use timeit rather than roll-you-own
> Huh. In looking into timing attacks actually one of the biggest lessons
> I learned was *not* to use timeit- that the overhead and variance
> involved in using it will wind up consuming small changes in behavior in
> ways that are fairly opaque until you really take it apart.
Do you have more details?
I follow the advice in the timeit module, and only ever look at the
minimum value, and never try to calculate a mean or variance. Given the
number of outside influences ("What do you mean starting up a browser
with 200 tabs at the same time will affect the timing?"), I wouldn't
trust a mean or variance to be meaningful.
More information about the Python-list