Mailman 3 February 2012 - Speed

Re: [Speed] Buildbot Status
by Brett Cannon Feb. 1, 2012

Feb. 1, 2012

On Wed, Feb 1, 2012 at 06:33, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <jnoller(a)gmail.com> wrote: > > > > > > On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > > > >>> > >>> I think pickle was mostly for unladen's pickle performance patches > (trying > >>> saying that three times fast =), so I don't really care about that one. > >> > >> This is up for discussion whether pickle's performance matters or not > >> (we have it disabled for example, but we might reenable it one day) > >> > >>> > >>> Would it make sense to change the pypy repo to make the unladen_swallow > >>> directory an external repo from hg.python.org/benchmarks? Because as > it > >>> stands right now there are two mako benchmarks that are not identical. > >>> Otherwise we should talk at PyCon and figure this all out before we > end up > >>> with two divergent benchmark suites that are being independently > maintained > >>> (since we are all going to be running the same benchmarks on > >>> speed.python.org). > >> > >> No, I think it's a bad idea. First benchmarks should not change. It's > >> fine to have a py3k benchmark next to py2 one, but we have 0 checkins > >> to US benchmarks once we imported them. > >> > >> Second, I find some of US benchmarks (those with hand unrolled loops, > >> like json from the newer one) complete nonsense. If it does make any > >> sense, it makes it only for cpython so we have no interest in having > >> those benchmarks at all. If someone unrolls loops by hand and have a > >> performance problem on pypy it's his own problem :) > > > > Great attitude for a shared, common set of benchmarks. Remind me not to > provide additional resources. > > Jesse, do you really write code like that: > > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(DICT) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > json.dumps(TUPLE) > > sorry for the long paste, but this is how the benchmark looks like. I > would like to have a common set of benchmarks, but a *reasonable* one. > Don't complain if I call it nonsense. > I think Jesse's point has nothing to do with thinking the unladen benchmarks are perfect as-is and everything to do with divergence; instead of diverging from the unladen benchmarks it would have been nicer to simply fix them instead of having PyPy containing some changes that are legitimate but never added back to the unladen benchmark repo that the work started from. We have revision history so there is no need to keep a pristine version anywhere if that was the thinking. Otherwise I'm going to assume it's because of unladen being on svn back in the day or some historical reason. So, to prevent this from either ending up in a dead-end because of this, we need to first decide where the canonical set of Python VM benchmarks are going to live. I say hg.python.org/benchmarks for two reasons. One is that Antoine has already done work there to port some of the benchmarks so there is at least some there that are ready to be run under Python 3 (and the tooling is in place to create separate Python 2 and Python 3 benchmark suites). Two, this can be a test of having the various VM contributors work out of hg.python.org if we are ever going to break the stdlib out for shared development. At worst we can simply take the changes made at pypy/benchmarks that apply to just the unladen benchmarks that exists, and at best merge the two sets (manually) into one benchmark suite so PyPy doesn't lose anything for Python 2 measurements that they have written and CPython doesn't lose any of its Python 3 benchmarks that it has created. How does that sound?

1 0

Re: [Speed] Buildbot Status
by Mark Shannon Feb. 1, 2012

Feb. 1, 2012

Maciej Fijalkowski wrote: >> Try running the json or pickle benchmarks with a profiler >> (oprofile or similar) >> and you will see that they spend most of their time in a single >> C module, not much of a benchmark for testing the VM. > > For what is worth I do believe that it is an aspect of VM performance > (for good or bad). FYI json module in pypy is pure python anyway. We should benchmark all the modules or none of them, why should json and pickle be special? Cheers, Mark.

2 1

Re: [Speed] Buildbot Status
by Mark Shannon Feb. 1, 2012

Feb. 1, 2012

Maciej Fijalkowski wrote: > On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <jnoller(a)gmail.com> wrote: >> >> On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >> >>>> I think pickle was mostly for unladen's pickle performance patches (trying >>>> saying that three times fast =), so I don't really care about that one. >>> This is up for discussion whether pickle's performance matters or not >>> (we have it disabled for example, but we might reenable it one day) >>> >>>> Would it make sense to change the pypy repo to make the unladen_swallow >>>> directory an external repo from hg.python.org/benchmarks? Because as it >>>> stands right now there are two mako benchmarks that are not identical. >>>> Otherwise we should talk at PyCon and figure this all out before we end up >>>> with two divergent benchmark suites that are being independently maintained >>>> (since we are all going to be running the same benchmarks on >>>> speed.python.org). >>> No, I think it's a bad idea. First benchmarks should not change. It's >>> fine to have a py3k benchmark next to py2 one, but we have 0 checkins >>> to US benchmarks once we imported them. >>> >>> Second, I find some of US benchmarks (those with hand unrolled loops, >>> like json from the newer one) complete nonsense. If it does make any >>> sense, it makes it only for cpython so we have no interest in having >>> those benchmarks at all. If someone unrolls loops by hand and have a >>> performance problem on pypy it's his own problem :) Doesn't PyPy have a loop re-rolling optimisation? :) >> Great attitude for a shared, common set of benchmarks. Remind me not to provide additional resources. > > Jesse, do you really write code like that: > > json.dumps(DICT) [snip - Manual loop unrolling (1970s style) :( ] > json.dumps(TUPLE) > I think the PyPy benchmarks are better than the US ones, some of which may have been aimed at a few particular cases where Google wanted to speed up CPython. Try running the json or pickle benchmarks with a profiler (oprofile or similar) and you will see that they spend most of their time in a single C module, not much of a benchmark for testing the VM. It might seem that the PyPy folks were cherry picking, but I don't think that is the case. Cheers, Mark.

2 1

Re: [Speed] Buildbot Status
by Jesse Noller Feb. 1, 2012

Feb. 1, 2012

On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >> >> I think pickle was mostly for unladen's pickle performance patches (trying >> saying that three times fast =), so I don't really care about that one. > > This is up for discussion whether pickle's performance matters or not > (we have it disabled for example, but we might reenable it one day) > >> >> Would it make sense to change the pypy repo to make the unladen_swallow >> directory an external repo from hg.python.org/benchmarks? Because as it >> stands right now there are two mako benchmarks that are not identical. >> Otherwise we should talk at PyCon and figure this all out before we end up >> with two divergent benchmark suites that are being independently maintained >> (since we are all going to be running the same benchmarks on >> speed.python.org). > > No, I think it's a bad idea. First benchmarks should not change. It's > fine to have a py3k benchmark next to py2 one, but we have 0 checkins > to US benchmarks once we imported them. > > Second, I find some of US benchmarks (those with hand unrolled loops, > like json from the newer one) complete nonsense. If it does make any > sense, it makes it only for cpython so we have no interest in having > those benchmarks at all. If someone unrolls loops by hand and have a > performance problem on pypy it's his own problem :) Great attitude for a shared, common set of benchmarks. Remind me not to provide additional resources. > > I have no idea why those benchmarks diverged. Probably because we did > not work on the python's hg so people hacked there. > > Cheers, > fijal > _______________________________________________ > Speed mailing list > Speed(a)python.org > http://mail.python.org/mailman/listinfo/speed

2 1

Re: [Speed] Buildbot Status
by Maciej Fijalkowski Feb. 1, 2012

Feb. 1, 2012

> > I think pickle was mostly for unladen's pickle performance patches (trying > saying that three times fast =), so I don't really care about that one. This is up for discussion whether pickle's performance matters or not (we have it disabled for example, but we might reenable it one day) > > Would it make sense to change the pypy repo to make the unladen_swallow > directory an external repo from hg.python.org/benchmarks? Because as it > stands right now there are two mako benchmarks that are not identical. > Otherwise we should talk at PyCon and figure this all out before we end up > with two divergent benchmark suites that are being independently maintained > (since we are all going to be running the same benchmarks on > speed.python.org). No, I think it's a bad idea. First benchmarks should not change. It's fine to have a py3k benchmark next to py2 one, but we have 0 checkins to US benchmarks once we imported them. Second, I find some of US benchmarks (those with hand unrolled loops, like json from the newer one) complete nonsense. If it does make any sense, it makes it only for cpython so we have no interest in having those benchmarks at all. If someone unrolls loops by hand and have a performance problem on pypy it's his own problem :) I have no idea why those benchmarks diverged. Probably because we did not work on the python's hg so people hacked there. Cheers, fijal

1 0