Re: [Speed] steps to get pypy benchmarks running
On 14/02/18 20:18, Nick Coghlan wrote:
Hi,
On 13/02/18 14:27, Matti Picus wrote:
I have begun to dive into the performance/perf code. My goal is to get pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT, the benchmark runs must have a warmup stage. Why? The other interpreters don't get an arbitrary chunk of time for free, so neither should PyPy. Warmup in an inherent cost of dynamic optimisers. The benefits should outweigh the costs, but the costs shouldn't be ignored. For speed.python.org purposes, that would likely be most usefully reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
On 14 February 2018 at 07:52, Mark Shannon <mark@hotpy.org> wrote: the former runs under the same conditions as CPython, while the latter is given the benefit of warming up the JIT first).
Only reporting the former would miss the point of PyPy's main use case (i.e. long lived processes), while only reporting the latter would miss one of the main answers to "Why hasn't everyone already switched to PyPy for all their Python needs?" (i.e. when the app doesn't run long enough to pay back the increased start-up overhead).
Cheers, Nick. So would it be reasonable as a first step to get the PyPy runner(s) into operation by modifying the nightly runs to download from the latest nightly builds [1], [2]? We can deal with reporting cold/warm statistics later. As people have said, they are really two orthogonal issues.
[1] http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 for python 2.7 [2] http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
Matti
On 20/03/18 17:31, Matti Picus wrote:
On 14/02/18 20:18, Nick Coghlan wrote:
Hi,
On 13/02/18 14:27, Matti Picus wrote:
I have begun to dive into the performance/perf code. My goal is to get pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT, the benchmark runs must have a warmup stage. Why? The other interpreters don't get an arbitrary chunk of time for free, so neither should PyPy. Warmup in an inherent cost of dynamic optimisers. The benefits should outweigh the costs, but the costs shouldn't be ignored. For speed.python.org purposes, that would likely be most usefully reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
On 14 February 2018 at 07:52, Mark Shannon <mark@hotpy.org> wrote: the former runs under the same conditions as CPython, while the latter is given the benefit of warming up the JIT first).
Only reporting the former would miss the point of PyPy's main use case (i.e. long lived processes), while only reporting the latter would miss one of the main answers to "Why hasn't everyone already switched to PyPy for all their Python needs?" (i.e. when the app doesn't run long enough to pay back the increased start-up overhead).
Cheers, Nick. So would it be reasonable as a first step to get the PyPy runner(s) into operation by modifying the nightly runs to download from the latest nightly builds [1], [2]? We can deal with reporting cold/warm statistics later. As people have said, they are really two orthogonal issues.
[1] http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 for python 2.7 [2] http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
Matti
No responses, maybe I asked the wrong question.
I would be willing to issue a pull request to get PyPy runners into
operation on "the beast" so it can report results to speed.python.org.
Which repo holds the code that stages performance runs and reports to
speed.pypy.org?
Matti
On Sun, 25 Mar 2018 at 07:37 Matti Picus <matti.picus@gmail.com> wrote:
On 20/03/18 17:31, Matti Picus wrote:
On 14/02/18 20:18, Nick Coghlan wrote:
Hi,
On 13/02/18 14:27, Matti Picus wrote:
I have begun to dive into the performance/perf code. My goal is to get pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT, the benchmark runs must have a warmup stage. Why? The other interpreters don't get an arbitrary chunk of time for free, so neither should PyPy. Warmup in an inherent cost of dynamic optimisers. The benefits should outweigh the costs, but the costs shouldn't be ignored. For speed.python.org purposes, that would likely be most usefully reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
On 14 February 2018 at 07:52, Mark Shannon <mark@hotpy.org> wrote: the former runs under the same conditions as CPython, while the latter is given the benefit of warming up the JIT first).
Only reporting the former would miss the point of PyPy's main use case (i.e. long lived processes), while only reporting the latter would miss one of the main answers to "Why hasn't everyone already switched to PyPy for all their Python needs?" (i.e. when the app doesn't run long enough to pay back the increased start-up overhead).
Cheers, Nick. So would it be reasonable as a first step to get the PyPy runner(s) into operation by modifying the nightly runs to download from the latest nightly builds [1], [2]? We can deal with reporting cold/warm statistics later. As people have said, they are really two orthogonal issues.
[1] http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 for python 2.7 [2] http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
Matti
No responses, maybe I asked the wrong question.
I think the people who have traditionally maintained speed.python.org are just not available to answer the question, not that it was the wrong question.
I would be willing to issue a pull request to get PyPy runners into operation on "the beast" so it can report results to speed.python.org. Which repo holds the code that stages
performanceruns and reports to speed.pypy.org?
Unfortunately I don't know.
-Brett
Matti
Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
participants (2)
-
Brett Cannon -
Matti Picus