[Speed] steps to get pypy benchmarks running

[Matti Picus]

On 20/03/18 17:31, Matti Picus wrote:
> On 14/02/18 20:18, Nick Coghlan wrote:
>> On 14 February 2018 at 07:52, Mark Shannon <mark at hotpy.org> wrote:
>>> Hi,
>>>
>>> On 13/02/18 14:27, Matti Picus wrote:
>>>> I have begun to dive into the performance/perf code. My goal is to get
>>>> pypy benchmarks running on http://speed.python.org. Since PyPy has 
>>>> a JIT,
>>>> the benchmark runs must have a warmup stage.
>>> Why?
>>> The other interpreters don't get an arbitrary chunk of time for 
>>> free, so
>>> neither should PyPy. Warmup in an inherent cost of dynamic 
>>> optimisers. The
>>> benefits should outweigh the costs, but the costs shouldn't be ignored.
>> For speed.python.org purposes, that would likely be most usefully
>> reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
>> the former runs under the same conditions as CPython, while the latter
>> is given the benefit of warming up the JIT first).
>>
>> Only reporting the former would miss the point of PyPy's main use case
>> (i.e. long lived processes), while only reporting the latter would
>> miss one of the main answers to "Why hasn't everyone already switched
>> to PyPy for all their Python needs?" (i.e. when the app doesn't run
>> long enough to pay back the increased start-up overhead).
>>
>> Cheers,
>> Nick.
> So would it be reasonable as a first step to get the PyPy runner(s) 
> into operation by modifying the nightly runs to download from the 
> latest nightly builds [1], [2]?
> We can deal with reporting cold/warm statistics later. As people have 
> said, they are really two orthogonal issues.
>
> [1] 
> http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 
> for python 2.7
> [2] 
> http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 
> for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
>
> Matti

No responses, maybe I asked the wrong question.
I would be willing to issue a pull request to get PyPy runners into 
operation on "the beast" so it can report results to speed.python.org.
Which repo holds the code that stages `performance` runs and reports to 
speed.pypy.org?
Matti