On Wed, Feb 1, 2012 at 06:33, Maciej Fijalkowski <fijall(a)gmail.com> wrote:
> On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <jnoller(a)gmail.com> wrote:
> >
> >
> > On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote:
> >
> >>>
> >>> I think pickle was mostly for unladen's pickle performance patches
> (trying
> >>> saying that three times fast =), so I don't really care about that one.
> >>
> >> This is up for discussion whether pickle's performance matters or not
> >> (we have it disabled for example, but we might reenable it one day)
> >>
> >>>
> >>> Would it make sense to change the pypy repo to make the unladen_swallow
> >>> directory an external repo from hg.python.org/benchmarks? Because as
> it
> >>> stands right now there are two mako benchmarks that are not identical.
> >>> Otherwise we should talk at PyCon and figure this all out before we
> end up
> >>> with two divergent benchmark suites that are being independently
> maintained
> >>> (since we are all going to be running the same benchmarks on
> >>> speed.python.org).
> >>
> >> No, I think it's a bad idea. First benchmarks should not change. It's
> >> fine to have a py3k benchmark next to py2 one, but we have 0 checkins
> >> to US benchmarks once we imported them.
> >>
> >> Second, I find some of US benchmarks (those with hand unrolled loops,
> >> like json from the newer one) complete nonsense. If it does make any
> >> sense, it makes it only for cpython so we have no interest in having
> >> those benchmarks at all. If someone unrolls loops by hand and have a
> >> performance problem on pypy it's his own problem :)
> >
> > Great attitude for a shared, common set of benchmarks. Remind me not to
> provide additional resources.
>
> Jesse, do you really write code like that:
>
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(DICT)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
> json.dumps(TUPLE)
>
> sorry for the long paste, but this is how the benchmark looks like. I
> would like to have a common set of benchmarks, but a *reasonable* one.
> Don't complain if I call it nonsense.
>
I think Jesse's point has nothing to do with thinking the unladen
benchmarks are perfect as-is and everything to do with divergence; instead
of diverging from the unladen benchmarks it would have been nicer to simply
fix them instead of having PyPy containing some changes that are legitimate
but never added back to the unladen benchmark repo that the work started
from. We have revision history so there is no need to keep a pristine
version anywhere if that was the thinking. Otherwise I'm going to assume
it's because of unladen being on svn back in the day or some historical
reason.
So, to prevent this from either ending up in a dead-end because of this, we
need to first decide where the canonical set of Python VM benchmarks are
going to live. I say hg.python.org/benchmarks for two reasons. One is that
Antoine has already done work there to port some of the benchmarks so there
is at least some there that are ready to be run under Python 3 (and the
tooling is in place to create separate Python 2 and Python 3 benchmark
suites). Two, this can be a test of having the various VM contributors work
out of hg.python.org if we are ever going to break the stdlib out for
shared development. At worst we can simply take the changes made at
pypy/benchmarks that apply to just the unladen benchmarks that exists, and
at best merge the two sets (manually) into one benchmark suite so PyPy
doesn't lose anything for Python 2 measurements that they have written and
CPython doesn't lose any of its Python 3 benchmarks that it has created.
How does that sound?
Maciej Fijalkowski wrote:
>> Try running the json or pickle benchmarks with a profiler
>> (oprofile or similar)
>> and you will see that they spend most of their time in a single
>> C module, not much of a benchmark for testing the VM.
>
> For what is worth I do believe that it is an aspect of VM performance
> (for good or bad). FYI json module in pypy is pure python anyway.
We should benchmark all the modules or none of them,
why should json and pickle be special?
Cheers,
Mark.
Maciej Fijalkowski wrote:
> On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <jnoller(a)gmail.com> wrote:
>>
>> On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote:
>>
>>>> I think pickle was mostly for unladen's pickle performance patches (trying
>>>> saying that three times fast =), so I don't really care about that one.
>>> This is up for discussion whether pickle's performance matters or not
>>> (we have it disabled for example, but we might reenable it one day)
>>>
>>>> Would it make sense to change the pypy repo to make the unladen_swallow
>>>> directory an external repo from hg.python.org/benchmarks? Because as it
>>>> stands right now there are two mako benchmarks that are not identical.
>>>> Otherwise we should talk at PyCon and figure this all out before we end up
>>>> with two divergent benchmark suites that are being independently maintained
>>>> (since we are all going to be running the same benchmarks on
>>>> speed.python.org).
>>> No, I think it's a bad idea. First benchmarks should not change. It's
>>> fine to have a py3k benchmark next to py2 one, but we have 0 checkins
>>> to US benchmarks once we imported them.
>>>
>>> Second, I find some of US benchmarks (those with hand unrolled loops,
>>> like json from the newer one) complete nonsense. If it does make any
>>> sense, it makes it only for cpython so we have no interest in having
>>> those benchmarks at all. If someone unrolls loops by hand and have a
>>> performance problem on pypy it's his own problem :)
Doesn't PyPy have a loop re-rolling optimisation? :)
>> Great attitude for a shared, common set of benchmarks. Remind me not to provide additional resources.
>
> Jesse, do you really write code like that:
>
> json.dumps(DICT)
[snip - Manual loop unrolling (1970s style) :( ]
> json.dumps(TUPLE)
>
I think the PyPy benchmarks are better than the US ones,
some of which may have been aimed at a few particular cases
where Google wanted to speed up CPython.
Try running the json or pickle benchmarks with a profiler
(oprofile or similar)
and you will see that they spend most of their time in a single
C module, not much of a benchmark for testing the VM.
It might seem that the PyPy folks were cherry picking,
but I don't think that is the case.
Cheers,
Mark.
On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote:
>>
>> I think pickle was mostly for unladen's pickle performance patches (trying
>> saying that three times fast =), so I don't really care about that one.
>
> This is up for discussion whether pickle's performance matters or not
> (we have it disabled for example, but we might reenable it one day)
>
>>
>> Would it make sense to change the pypy repo to make the unladen_swallow
>> directory an external repo from hg.python.org/benchmarks? Because as it
>> stands right now there are two mako benchmarks that are not identical.
>> Otherwise we should talk at PyCon and figure this all out before we end up
>> with two divergent benchmark suites that are being independently maintained
>> (since we are all going to be running the same benchmarks on
>> speed.python.org).
>
> No, I think it's a bad idea. First benchmarks should not change. It's
> fine to have a py3k benchmark next to py2 one, but we have 0 checkins
> to US benchmarks once we imported them.
>
> Second, I find some of US benchmarks (those with hand unrolled loops,
> like json from the newer one) complete nonsense. If it does make any
> sense, it makes it only for cpython so we have no interest in having
> those benchmarks at all. If someone unrolls loops by hand and have a
> performance problem on pypy it's his own problem :)
Great attitude for a shared, common set of benchmarks. Remind me not to provide additional resources.
>
> I have no idea why those benchmarks diverged. Probably because we did
> not work on the python's hg so people hacked there.
>
> Cheers,
> fijal
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> http://mail.python.org/mailman/listinfo/speed
>
> I think pickle was mostly for unladen's pickle performance patches (trying
> saying that three times fast =), so I don't really care about that one.
This is up for discussion whether pickle's performance matters or not
(we have it disabled for example, but we might reenable it one day)
>
> Would it make sense to change the pypy repo to make the unladen_swallow
> directory an external repo from hg.python.org/benchmarks? Because as it
> stands right now there are two mako benchmarks that are not identical.
> Otherwise we should talk at PyCon and figure this all out before we end up
> with two divergent benchmark suites that are being independently maintained
> (since we are all going to be running the same benchmarks on
> speed.python.org).
No, I think it's a bad idea. First benchmarks should not change. It's
fine to have a py3k benchmark next to py2 one, but we have 0 checkins
to US benchmarks once we imported them.
Second, I find some of US benchmarks (those with hand unrolled loops,
like json from the newer one) complete nonsense. If it does make any
sense, it makes it only for cpython so we have no interest in having
those benchmarks at all. If someone unrolls loops by hand and have a
performance problem on pypy it's his own problem :)
I have no idea why those benchmarks diverged. Probably because we did
not work on the python's hg so people hacked there.
Cheers,
fijal
