[Python-Dev] Benchmark results across all major Python implementations

[Zachary Ware]

On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin <brian at python.org> wrote:
> On Monday, November 16, 2015, Brett Cannon <brett at python.org> wrote:

>>>
>>> Hi Brett
>>>
>>> Any thoughts on improving the benchmark set (I think all of
>>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>>
>> We should probably start a mailing list
>
>
> There is/was a speed at python.org list.

Is, but seems to be heavily moderated without active moderation.  I
sent an offer to speed-owner this morning to help with moderation, but
as yet no response.  I know I have a couple of messages waiting in the
queue :)

On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski <fijall at gmail.com> wrote:
> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".

Setup is done in my dev environment, it's now blocking on people more
qualified than me to review and merge, then final tweaking of the
buildbot setup.  For those interested:

The part in most need of review is the changes to the PSF Salt
configuration to set up and run Codespeed on speed.python.org.  The
changes can be found in PR #74 on the psf-salt Github repo[0].

The Codespeed instance is housed in a Github repo owned by me[1]
currently.  There's one small patch to the codespeed code (which was
merged upstream this morning), the rest of the changes in my fork are
adapting a copy of the sample_project to be our own instance.  I've
been told that the grant proposal from long ago expected the use of
codespeed2 instead of codespeed.  I chose codespeed over codespeed2
because it appeared to be easier to get set up and running (which may
or may not have actually been true), but also because I've not seen
codespeed2 in action anywhere.

The final piece that could use review is the changes to our benchmark
repository, currently available in a sandbox repo on hg.python.org[2].
My original plan had been to use PyPy's benchmark suite, but as you
can tell from the logs of the existing buildslave, cpython doesn't run
that suite especially well, and the cpython suite has the advantage of
working with Python 2 and 3 out of the box.

Please have a look, give it a try if you can, and let me know what
needs improvement!

[0] https://github.com/python/psf-salt/pull/74
[1] https://github.com/zware/codespeed
[2] https://hg.python.org/sandbox/zware-benchmarks

-- 
Zach