[pypy-dev] Tune performance benchmarks for PyPy

[Victor Stinner]

Hi,

I ran the performance benchmark suite with PyPy 5.7 on the
speed-python server, the benchmark runner feeding speed.python.org. I
chose to spawn 10 processes, each process computed 250 values. The
number of outer loops was calibrated per benchmark. Here are raw data:

http://www.haypocalc.com/tmp/pypy_p10_w0_n250.json.gz

The plan is now to choose the number of warmup values from these data
and copy the number of loops to get fixed parameters.

You can directly analyze the (compressed) JSON file using plot.py to
plot values:

https://github.com/haypo/perf/raw/master/doc/examples/plot.py

I suggest to use the --split-runs value to have one line per run (per process):

python3 doc/examples/plot.py ~/pypy_p10_w0_n250.json.gz -b telco --split-runs

It's likely that the first values are much larger: use the UI to zoom,
or pass --skip=N to skip the first N values of each run.

python3 doc/examples/plot.py ~/pypy_p10_w0_n250.json.gz -b telco
--split-runs --skip=3

Or you can only render one specific run:

python3 doc/examples/plot.py ~/pypy_p10_w0_n250.json.gz -b telco
--split-runs --run=2

The JSON file contains results of 67 benchmarks: use "python3 -m perf
show ~/pypy_p10_w0_n250.json.gz -q" to list them. plot.py script
requires a benchmark name (-b/--benchmark option).

The game is now to choose the number of values to skip to reach the
"steady state". plot.py renders the mean to help you ;-)

You can also use perf commands to analyze data:
* perf stats
* perf dump
* perf hist
* etc.

These commands can display all benchmarks, or only one if you use the -b option.

Or you can use perf API to export data and use whatever you want to
analyze data.

See also the perf doc:
http://perf.readthedocs.io/en/latest/analyze.html

Next week, I will try to do the work for each benchmark, and then I
will come back to you with numbers, so we can double-check together
that everything looks fine.

Parameters are likely to change depending on the CPU, PyPy version,
etc. But my first goal is to get reproductible results on speed-python
with the latest version of PyPy, so I will start with that. Later, we
can adjust parameters depending on the CPU and/or PyPy version.

Enjoy!
Victor