2017-03-16 2:04 GMT+01:00 Wang, Peter Xihong <peter.xihong.wang@intel.com>:

Understood on the obsolete benchmark part. This was the work done before the new benchmark was created on github.

I strongly advice you to move to performance. It also has a nice a API. It now produces a JSON file with *all* data, instead of just writing into summaries into stdout.

I thought this is related, and thus didn't open a new thread.

The other thread was a discussion about statistics, how to summarize all timing into two numbers :-)

Maybe you could point me to one single micro-benchmark for the time being, and then we could compare result across?

The "new" performance project is a fork of the old "benchmark" project. Benchmark names are very close or even the same for many benchmarks.

If you would like to validate that your benchmark runner is stable: run call_method and call_simple microbenchmarks on different revisions of CPython, reboot sometimes the computer used to run benchmarks, and make sure that results are stable.

Compare them with results of speed.python.org.

call_method: https://speed.python.org/timeline/#/?exe=5&ben=call_method&env=1&revs=50&equid=off&quarts=on&extr=on

call_simple: https://speed.python.org/timeline/#/?exe=5&ben=call_simple&env=1&revs=50&equid=off&quarts=on&extr=on

Around november and december 2016, you should notice a significant speedup on call_method.

The best is to be able to avoid "temporary spikes" like this one: https://haypo.github.io/analysis-python-performance-issue.html

The API of the perf project, PGO and LTO compilation, new performance using perf, "perf system tune" for system tuning, etc. helped to get more stable results.

Victor