[Python-Dev] Stop using timeit, use perf.timeit!

[Terry Reedy]

On 6/10/2016 12:09 PM, Victor Stinner wrote:
> 2016-06-10 17:09 GMT+02:00 Paul Moore <p.f.moore at gmail.com>:
>> Also, the way people commonly use
>> micro-benchmarks ("hey, look, this way of writing the expression goes
>> faster than that way") doesn't really address questions like "is the
>> difference statistically significant".
>
> If you use the "python3 -m perf compare method1.json method2.json",
> perf will checks that the difference is significant using the
> is_significant() method:
> http://perf.readthedocs.io/en/latest/api.html#perf.is_significant
> "This uses a Student’s two-sample, two-tailed t-test with alpha=0.95."

Depending on the sampling design, a matched-pairs t-test may be more 
appropriate.

-- 
Terry Jan Reedy