On Sun, 03 Jan 2016 19:57:50 +0000 Brett Cannon <brett@python.org> wrote:

With the planned move to GitHub, there is an opportunity to try and rework the set of benchmarks -- and anything else -- in 2016 by starting a new benchmark repo from scratch. E.g., modern numeric benchmarks, long-running benchmarks that warm up JITs, using pip with pegged bugfix versions so we stop shipping library code with the benchmarks, etc.

That feature list should be discussed :-) For example I don't think downloading dependencies with pip is desirable compared to the ease of cloning a single repo (also what happens if the files become unavailable?).

We could also standardize results output -- e.g. should we just make everything run under codespeed? -- so that the benchmarks are easy to run locally for one-off results as well as continuous benchmarking for trend details with a common benchmark driver?

I don't know who uses codespeed apart from PyPy, but asv is a possible benchmark runner: https://asv.readthedocs.org/en/latest/index.html

Would people be interested and motivated enough in getting representatives from the various Python implementations together at PyCon and have a BoF to discuss what we want from a proper, unified, baseline benchmark suite and see if we can pull one together -- or at least start one -- in 2016?

I probably won't be at PyCon.

Regards

Antoine.