On Sun, 03 Jan 2016 19:57:50 +0000
Brett Cannon <brett(a)python.org> wrote:
> With the planned move to GitHub, there is an opportunity to try and rework
> the set of benchmarks -- and anything else -- in 2016 by starting a new
> benchmark repo from scratch. E.g., modern numeric benchmarks, long-running
> benchmarks that warm up JITs, using pip with pegged bugfix versions so we
> stop shipping library code with the benchmarks, etc.
That feature list should be discussed :-) For example I don't think
downloading dependencies with pip is desirable compared to the ease of
cloning a single repo (also what happens if the files become
unavailable?).
> We could also
> standardize results output -- e.g. should we just make everything run under
> codespeed? -- so that the benchmarks are easy to run locally for one-off
> results as well as continuous benchmarking for trend details with a common
> benchmark driver?
I don't know who uses codespeed apart from PyPy, but asv is a possible
benchmark runner:
https://asv.readthedocs.org/en/latest/index.html
> Would people be interested and motivated enough in getting representatives
> from the various Python implementations together at PyCon and have a BoF to
> discuss what we want from a proper, unified, baseline benchmark suite and
> see if we can pull one together -- or at least start one -- in 2016?
I probably won't be at PyCon.
Regards
Antoine.
count me in (even if I don't make it to pycon)
On Sun, Jan 3, 2016 at 9:57 PM, Brett Cannon <brett(a)python.org> wrote:
> With the planned move to GitHub, there is an opportunity to try and rework
> the set of benchmarks -- and anything else -- in 2016 by starting a new
> benchmark repo from scratch. E.g., modern numeric benchmarks, long-running
> benchmarks that warm up JITs, using pip with pegged bugfix versions so we
> stop shipping library code with the benchmarks, etc. We could also
> standardize results output -- e.g. should we just make everything run under
> codespeed? -- so that the benchmarks are easy to run locally for one-off
> results as well as continuous benchmarking for trend details with a common
> benchmark driver?
>
> Would people be interested and motivated enough in getting representatives
> from the various Python implementations together at PyCon and have a BoF to
> discuss what we want from a proper, unified, baseline benchmark suite and
> see if we can pull one together -- or at least start one -- in 2016?
>
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> https://mail.python.org/mailman/listinfo/speed
>
With the planned move to GitHub, there is an opportunity to try and rework
the set of benchmarks -- and anything else -- in 2016 by starting a new
benchmark repo from scratch. E.g., modern numeric benchmarks, long-running
benchmarks that warm up JITs, using pip with pegged bugfix versions so we
stop shipping library code with the benchmarks, etc. We could also
standardize results output -- e.g. should we just make everything run under
codespeed? -- so that the benchmarks are easy to run locally for one-off
results as well as continuous benchmarking for trend details with a common
benchmark driver?
Would people be interested and motivated enough in getting representatives
from the various Python implementations together at PyCon and have a BoF to
discuss what we want from a proper, unified, baseline benchmark suite and
see if we can pull one together -- or at least start one -- in 2016?
