[pypy-dev] Moving the project forward

[Antonio Cuni]

Hi Jesse, hi all,

On 31/08/11 02:37, Jesse Noller wrote:
> Yes, I should have looped Noah in sooner.
>
> I have all the keys / passwords. Right now we need:
>
> 1. codespeed
> 2. benchmark runners

The second item is divided into two sub-items: the runner itself, and 
something that triggers a run nightly/on request.

In PyPy, we currently use a modified version of the Unladen Swallow benchmark 
runner (look at runner.py):
https://bitbucket.org/pypy/benchmarks/src

in its current state, it is sub-optimal, because it was designed to compare 
two pythons side-by-side, and not to save the results and then upload them to 
a website.  As a consequence, it always runs the benchmarks twice.  With PyPy, 
it's lesser of a problem because for each binary we always run the benchmarks 
with the default options and with "--jit off" anyway, but for CPython running 
them twice would result in a waste of time.

I suppose that the current runner is good enough for now and to get started, 
however at some point it will need some care.


Then, we need something which triggers the actual benchmark run.  In PyPy, we 
use buildbot, and each benchmark run first do a full translation, then 
executes the benchmarks.

I can setup a buildbot instance on speed.pypy.org, if you give me access to 
the machine.  I propose that as a very first step, we just make speed.pypy.org 
a buildslave which depends on pypy's own buildmaster.   This makes it very 
easy and fast to setup it, so we can have something running soon.

ciao,
Anto