2016-06-02 10:38 GMT+02:00 Armin Rigo arigo@tunes.org:

Looks good. IMHO the important bit is that timeit is simple to use, readily available, and gives just a number, which makes it very attractive for people.

By default min & max are hidden. You can show them using -v option.

To make the output even simpler, maybe the standard deviation can be displayed "in english". Something like:

• "Average: 250 ns +- 3 ns" => "Average: 250 ns (stable)", or just "Average: 250 ns"
• "Average: 250 ns +- 120 ns" => "Average: 250 ns (not reliable, try again on an idle system)"

Usually, timeit it used to compare two versions of Python. Maybe we should focus on this use case, and check if the difference is significant, as perf.py does? By default, perf.py does *not* display any number if the difference is not significant. I like this behaviour, even if it can be surprising for the first time.

For the CLI, we can extend timeit CLI to accept the path/name of two python binaries. Or we can use something like pybench to store result into files and then load & compare two files.

Victor