[Speed] New instance of CodeSpeed at speed.python.org running performance on CPython and PyPy?

[Victor Stinner]

2016-09-01 19:53 GMT+02:00 Kevin Modzelewski <kmod at dropbox.com>:
> Just my two cents -- having a benchmark change underneath the benchmark
> runner is quite confusing to debug, because it looks indistinguishable from
> a non-reproducible regression that happens in the performance itself.

I agree. That's why I proposed to use semantic versionning. I'm not
sure that old results must be removed. We should just be explicit
about versions.

The main issue is when you *compare* two results produced by two
different performance versions. I have an item in my TODO list to emit
a warning if the exact version (minor version) is different, and
display an error if the major version is different.

About reproductability: I made another change in the development
version, indirect dependencies are now pinned as well:
https://github.com/python/performance/blob/master/performance/requirements.txt#L15

It should help to have a more reproductible benchmark ;-)

The last known issue about reproductability is that I dropped the code
to remove environment variables. I should fix this in the perf module
directly.

Interesting link: https://reproducible-builds.org/


> Another thing to consider is that there will be other people using this
> benchmark set than just the codespeed setup: there will be long-lived
> benchmark results in the form of blogs and academic papers.  I think it's
> important to get some good wording about having to include the version of
> the benchmarks when publishing results, and then it would be good to follow
> that advice internally as well.

The perf module has a feature for that: it supports storing metadata
in JSON files. I modified many benchmarks to store dependency version
(like Django verison), but now also store performance version (since
performance 0.2). The perf module stores its own version by default
;-)

I suggest to store JSON file rather than only the compact text output
(which contains much less information).

Victor