On Tue, Dec 1, 2015 at 6:36 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi

Thanks for doing the work! I'm on of the pypy devs and I'm very
interested in seeing this getting somewhere. I must say I struggle to
read the graph - is red good or is red bad for example?

I'm keen to help you getting anything you want to run it repeatedly.

PS. The intel stuff runs one benchmark in a very questionable manner,
so let's maybe not rely on it too much.

​Hi Maciej,

Great, it'd be awesome having data on multiple Python VMs (my latest target is really having a way to compare across multiple VMs/versions easily and help each implementation keep a focus on performance). Ideally, a single, dedicated machine could be used just to run the benchmarks from multiple VMs (one less variable to take into account for comparisons later on, as I'm not sure it'd be reliable to normalize benchmark data from different machines -- it seems Zach was the one to contact from that, but if there's such a machine already being used to run PyPy, maybe it could be extended to run other VMs too?).

As for the graph, it should be easy to customize (and I'm open to suggestions). In the case, as it is, red is slower and blue is faster (so, for instance in https://www.speedtin.com/reports/1_CPython27x_Performance_Over_Time,  the fastest CPython version overall was 2.7.3 -- and 2.7.1 was the baseline). I've updated the comments to make it clearer (and changed the second graph to compare the latest against the fastest version (2.7.rc11 vs 2.7.3) for the individual benchmarks.

Best Regards,

Fabio

 

On Mon, Nov 30, 2015 at 3:52 PM, R. David Murray <rdmurray@bitdance.com> wrote:
> On Mon, 30 Nov 2015 09:02:12 -0200, Fabio Zadrozny <fabiofz@gmail.com> wrote:
>> Note that uploading the data to SpeedTin should be pretty straightforward
>> (by using https://github.com/fabioz/pyspeedtin, so, the main issue would be
>> setting up o machine to run the benchmarks).
>
> Thanks, but Zach almost has this working using codespeed (he's still
> waiting on a review from infrastructure, I think).  The server was not in
> fact running; a large part of what Zach did was to get that server set up.
> I don't know what it would take to export the data to another consumer,
> but if you want to work on that I'm guessing there would be no objection.
> And I'm sure there would be no objection if you want to get involved
> in maintaining the benchmark server!
>
> There's also an Intel project posted about here recently that checks
> individual benchmarks for performance regressions and posts the results
> to python-checkins.
>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/fabiofz%40gmail.com