On Sat, Sep 1, 2012 at 9:45 PM, Gregory P. Smith <greg@krypto.org> wrote:

On Sat, Sep 1, 2012 at 10:21 AM, Brett Cannon <brett@python.org> wrote:

...

Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, I'm ready to start updating the benchmarks. This involves two parts.

One is moving benchmarks from PyPy over to the unladen repo on hg.python.org/benchmarks. But I wanted to first make sure people don't view the benchmarks as immutable (e.g. as Octane does: https://developers.google.com/octane/faq). Since the benchmarks are always relative between two interpreters their immutability isn't critical compared to if we were to report some overall score. But it also means that any changes made would throw off historical comparisons. For instance, if I take PyPy's Mako benchmark (which does a lot more work), should it be named mako_v2, or should we just replace mako wholesale?

I dislike benchmark immutability. The rest of the world including your local computing environment where benchmarks run continues to change around benchmarks which really makes using historical benchmark data from a run on an old version for comparison to a recent modern run pointless.

So far we (pypy) managed to maintain enough of the environment under control to have meaningful historical data. We have the same machine and we monitor whether changes introduce something new or not. Of course ideally, it's impossible, but for real world what we're doing is good enough.

Cheers, fijal