Mailman 3 January 2016 - Speed

Re: [Speed] People interested in reworking the benchmark suite in 2016?
by Antoine Pitrou Jan. 12, 2016

Jan. 12, 2016

On Sun, 03 Jan 2016 19:57:50 +0000 Brett Cannon <brett(a)python.org> wrote: > With the planned move to GitHub, there is an opportunity to try and rework > the set of benchmarks -- and anything else -- in 2016 by starting a new > benchmark repo from scratch. E.g., modern numeric benchmarks, long-running > benchmarks that warm up JITs, using pip with pegged bugfix versions so we > stop shipping library code with the benchmarks, etc. That feature list should be discussed :-) For example I don't think downloading dependencies with pip is desirable compared to the ease of cloning a single repo (also what happens if the files become unavailable?). > We could also > standardize results output -- e.g. should we just make everything run under > codespeed? -- so that the benchmarks are easy to run locally for one-off > results as well as continuous benchmarking for trend details with a common > benchmark driver? I don't know who uses codespeed apart from PyPy, but asv is a possible benchmark runner: https://asv.readthedocs.org/en/latest/index.html > Would people be interested and motivated enough in getting representatives > from the various Python implementations together at PyCon and have a BoF to > discuss what we want from a proper, unified, baseline benchmark suite and > see if we can pull one together -- or at least start one -- in 2016? I probably won't be at PyCon. Regards Antoine.

1 0

Re: [Speed] People interested in reworking the benchmark suite in 2016?
by Maciej Fijalkowski Jan. 3, 2016

Jan. 3, 2016

count me in (even if I don't make it to pycon) On Sun, Jan 3, 2016 at 9:57 PM, Brett Cannon <brett(a)python.org> wrote: > With the planned move to GitHub, there is an opportunity to try and rework > the set of benchmarks -- and anything else -- in 2016 by starting a new > benchmark repo from scratch. E.g., modern numeric benchmarks, long-running > benchmarks that warm up JITs, using pip with pegged bugfix versions so we > stop shipping library code with the benchmarks, etc. We could also > standardize results output -- e.g. should we just make everything run under > codespeed? -- so that the benchmarks are easy to run locally for one-off > results as well as continuous benchmarking for trend details with a common > benchmark driver? > > Would people be interested and motivated enough in getting representatives > from the various Python implementations together at PyCon and have a BoF to > discuss what we want from a proper, unified, baseline benchmark suite and > see if we can pull one together -- or at least start one -- in 2016? > > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

1 0

People interested in reworking the benchmark suite in 2016?
by Brett Cannon Jan. 3, 2016

Jan. 3, 2016

With the planned move to GitHub, there is an opportunity to try and rework the set of benchmarks -- and anything else -- in 2016 by starting a new benchmark repo from scratch. E.g., modern numeric benchmarks, long-running benchmarks that warm up JITs, using pip with pegged bugfix versions so we stop shipping library code with the benchmarks, etc. We could also standardize results output -- e.g. should we just make everything run under codespeed? -- so that the benchmarks are easy to run locally for one-off results as well as continuous benchmarking for trend details with a common benchmark driver? Would people be interested and motivated enough in getting representatives from the various Python implementations together at PyCon and have a BoF to discuss what we want from a proper, unified, baseline benchmark suite and see if we can pull one together -- or at least start one -- in 2016?

1 0