[Python-Dev] Avoiding CPython performance regressions

[Fabio Zadrozny]

On Tue, Dec 1, 2015 at 6:36 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:

> Hi
>
> Thanks for doing the work! I'm on of the pypy devs and I'm very
> interested in seeing this getting somewhere. I must say I struggle to
> read the graph - is red good or is red bad for example?
>
> I'm keen to help you getting anything you want to run it repeatedly.
>
> PS. The intel stuff runs one benchmark in a very questionable manner,
> so let's maybe not rely on it too much.
>

​Hi Maciej,

Great, it'd be awesome having data on multiple Python VMs (my latest target
is really having a way to compare across multiple VMs/versions easily and
help each implementation keep a focus on performance). Ideally, a single,
dedicated machine could be used just to run the benchmarks from multiple
VMs (one less variable to take into account for comparisons later on, as
I'm not sure it'd be reliable to normalize benchmark data from different
machines -- it seems Zach was the one to contact from that, but if there's
such a machine already being used to run PyPy, maybe it could be extended
to run other VMs too?).

As for the graph, it should be easy to customize (and I'm open to
suggestions). In the case, as it is, red is slower and blue is faster (so,
for instance in
https://www.speedtin.com/reports/1_CPython27x_Performance_Over_Time,  the
fastest CPython version overall was 2.7.3 -- and 2.7.1 was the baseline).
I've updated the comments to make it clearer (and changed the second graph
to compare the latest against the fastest version (2.7.rc11 vs 2.7.3) for
the individual benchmarks.

Best Regards,

Fabio



>
> On Mon, Nov 30, 2015 at 3:52 PM, R. David Murray <rdmurray at bitdance.com>
> wrote:
> > On Mon, 30 Nov 2015 09:02:12 -0200, Fabio Zadrozny <fabiofz at gmail.com>
> wrote:
> >> Note that uploading the data to SpeedTin should be pretty
> straightforward
> >> (by using https://github.com/fabioz/pyspeedtin, so, the main issue
> would be
> >> setting up o machine to run the benchmarks).
> >
> > Thanks, but Zach almost has this working using codespeed (he's still
> > waiting on a review from infrastructure, I think).  The server was not in
> > fact running; a large part of what Zach did was to get that server set
> up.
> > I don't know what it would take to export the data to another consumer,
> > but if you want to work on that I'm guessing there would be no objection.
> > And I'm sure there would be no objection if you want to get involved
> > in maintaining the benchmark server!
> >
> > There's also an Intel project posted about here recently that checks
> > individual benchmarks for performance regressions and posts the results
> > to python-checkins.
> >
> > --David
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fabiofz%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151201/0292a0a8/attachment.html>