Just my two cents -- having a benchmark change underneath the benchmark runner is quite confusing to debug, because it looks indistinguishable from a non-reproducible regression that happens in the performance itself. My vote would be to wipe the benchmark results when this happens (and if that is too expensive, not upgrade that often).

Another thing to consider is that there will be other people using this benchmark set than just the codespeed setup: there will be long-lived benchmark results in the form of blogs and academic papers. I think it's important to get some good wording about having to include the version of the benchmarks when publishing results, and then it would be good to follow that advice internally as well.

kmod

On Thu, Sep 1, 2016 at 3:58 AM, Victor Stinner <victor.stinner@gmail.com> wrote:

Hi,

Would it be possible to run a new instance of CodeSpeed (the website behing speed.python.org) which would run the "performance" benchmark suite rather than the "benchmarks" benchmark suite? And would it be possible to run it on CPython (2.7 and 3.5 branches) and PyPy (master branch, maybe also the py3k branch)?

I found https://github.com/tobami/codespeed/ but I didn't look at it right now. I guess that some code should be written to convert perf JSON file to the format expected by CodeSpeed?

FYI I released performance 0.2 yesterday. JSON files now contain the version of the benchmark suite ("performance_version: 0.2"). I plan to use semantic version: increase the major version (ex: upgrade to 0.3, but later it will be 1.x, 2.x, etc.) when benchmark results are considered to not be compatible.

For example, I upgraded Django (from 1.9 to 1.10) and Chameleon (from 2.22 to 2.24) in performance 0.2.

The question is how to upgrade the performance to a new major version: should we drop previous benchmark results?

Maybe we should put the performance version in the URL, and use "/latest/" by default. Only /latest/ would get new results, and /latest/ would restart from an empty set of results when performance is upgraded?

Another option, less exciting, is to never upgrade benchmarks. The benchmarks project *added* new benchmarks when a dependency was "upgraded". In fact, the old dependency was kept and a new dependency (full copy of the code in fact ;-)) was added. So it has django, django_v2, django_v3, etc. The problem is that it still uses Mercurial 1.2 which was released 7 years ago (2009)... Since it's painful to upgrade, most dependencies were outdated.

Do you care of old benchmark results? It's quite easy to regenerate them (on demand?) if needed, no? Using Mercurial and Git, it's easy to update to any old revisions to run again a benchmark on an old version of CPython / PyPy / etc.

Victor

---

Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed