[Python-Dev] speed.python.org (was: 2.7 is here until 2020, please don't call it a waste.)

[Zachary Ware]

On Thu, Jun 4, 2015 at 10:51 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Thu, Jun 4, 2015 at 4:32 PM, R. David Murray <rdmurray at bitdance.com> wrote:
>> OK, so what you are saying is that speed.python.org will run a buildbot
>> slave so that when a change is committed to cPython, a speed run will be
>> triggered?  Is "the runner" a normal buildbot slave, or something
>> custom?  In the normal case the master controls what the slave
>> runs...but regardless, you'll need to let us know how the slave
>> invocation needs to be configured on the master.
>
> Ideally nightly (benchmarks take a while). The setup for pypy looks like this:
>
>
> https://bitbucket.org/pypy/buildbot/src/5fa1f1a4990f842dfbee416c4c2e2f6f75d451c4/bot2/pypybuildbot/builds.py?at=default#cl-734
>
> so fairly easy. This already generates a json file that you can plot.
> We can setup an upload automatically too.

I've been looking at what it will take to set up the buildmaster for
this, and it looks like it's just a matter of checking out the
benchmarks, building Python, testing it, and running the benchmarks.
There is the question of which benchmark repo to use:
https://bitbucket.org/pypy/benchmarks or
https://hg.python.org/benchmarks; ideally, we should use
hg.python.org/benchmarks for CPython benchmarks, but it looks like
pypy/benchmarks has the necessary runner, so I suppose we'll be using
it for now.  Is there interest from both sides to merge those
repositories?

The big question for the buildmaster is what options to pass to the
benchmark runner.  I suppose most of them should match the
CPythonBenchmark BuildFactory from the PyPy buildbot master
configuration, but otherwise I'm not sure.

The other big question is where the benchmarks will be run.  The
speed.python.org page makes it sound like there's a box intended for
that purpose (the one running the speed.python.org page?); can anyone
with access to it contact me to get the build slave set up?

-- 
Zach