[Speed] merging PyPy and Python benchmark suite

[Alex Gaynor]

On Tue, Jul 24, 2012 at 10:10 AM, Brett Cannon <brett at python.org> wrote:

>
>
> On Mon, Jul 23, 2012 at 7:34 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:
>
>> On Mon, Jul 23, 2012 at 11:46 PM, Brett Cannon <brett at python.org> wrote:
>> >
>> >
>> > On Mon, Jul 23, 2012 at 4:39 PM, Armin Rigo <arigo at tunes.org> wrote:
>> >>
>> >> Hi Brett,
>> >>
>> >> On Mon, Jul 23, 2012 at 10:15 PM, Brett Cannon <brett at python.org>
>> wrote:
>> >> > That's what I'm trying to establish; how much have they diverged and
>> if
>> >> > I'm
>> >> > looking in the proper place.
>> >>
>> >> bm_mako.py is not from Unladen Swallow; that's why it is in
>> >> pypy/benchmarks/own/.  In case of doubts, check it in the history of
>> >> Hg.  The PyPy version was added from virhilo, which seems to be the
>> >> name of his author, on 2010-12-21, and was not changed at all since
>> >> then.
>> >
>> >
>> > OK. Maciej has always told me that a problem with the Unladen
>> benchmarks was
>> > that some of them had artificial loop unrolling, etc., so I had assumed
>> you
>> > had simply fixed those instances instead of creating entirely new
>> > benchmarks.
>>
>> No we did not use those benchmarks. Those were mostly completely
>> artificial microbenchmarks (call, call_method etc.). We decided we're
>> not really that interested in microbenchmarks.
>>
>> >
>> >>
>> >>
>> >> Hg tells me that there was no change at all in the 'unladen_swallow'
>> >> subdirectory, apart from 'unladen_swallow/perf.py' and adding some
>> >> __init__.py somewhere.  So at least these benchmarks did not receive
>> >> any pypy-specific adapatations.  If there are divergences, they come
>> >> from changes done to the unladen-swallow benchmark suite after PyPy
>> >> copied it on 2010-01-15.
>> >
>> >
>> > I know that directory wasn't changed, but I also noticed that some
>> > benchmarks had the same name, which is why I thought they were forked
>> > versions of the same-named Unladen benchmarks.
>>
>> Not if they're in own/ directory.
>>
>
> OK, good to know. I realized I can't copy code wholesale from PyPy's
> benchmark suite as I don't know the code's history and thus if the
> contributor signed Python's contributor agreement. Can the people who are
> familiar with the code help move benchmarks over where the copyright isn't
> in question?
>
> I can at least try to improve the Python 3 situation by doing things like
> pulling in Vinay's py3k port of Django, etc. to fill in gaps. I will also
> try to get the benchmarks to work with a Python 2.7 control and a Python 3
> "experimental" target for comparing performance since that's what I need
> (or at least be able to run the benchmarks on their own and writing out the
> results for later comparison).
>
> Anything else that should be worked on?
>
> _______________________________________________
> Speed mailing list
> Speed at python.org
> http://mail.python.org/mailman/listinfo/speed
>
>
The important thing is that once a benchmark is in the repo it can *never*
change including all the versions of dependencies, only Python can vary,
otherwise you kill the ability to actually do science with the numbers.

So, e.g., I wouldn't pull in Vijnay's fork, since that's going to be
utterly obsolete in a few weeks probably, I'd wait to have django on py3k
for that work to all be merged into django itself.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/speed/attachments/20120724/afa2fce5/attachment.html>