<br><br><div class="gmail_quote">On Wed, Feb 1, 2012 at 06:33, Maciej Fijalkowski <span dir="ltr"><<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Wed, Feb 1, 2012 at 1:24 PM, Jesse Noller <<a href="mailto:jnoller@gmail.com">jnoller@gmail.com</a>> wrote:<br>
><br>
><br>
> On Feb 1, 2012, at 4:43 AM, Maciej Fijalkowski <<a href="mailto:fijall@gmail.com">fijall@gmail.com</a>> wrote:<br>
><br>
>>><br>
>>> I think pickle was mostly for unladen's pickle performance patches (trying<br>
>>> saying that three times fast =), so I don't really care about that one.<br>
>><br>
>> This is up for discussion whether pickle's performance matters or not<br>
>> (we have it disabled for example, but we might reenable it one day)<br>
>><br>
>>><br>
>>> Would it make sense to change the pypy repo to make the unladen_swallow<br>
>>> directory an external repo from <a href="http://hg.python.org/benchmarks" target="_blank">hg.python.org/benchmarks</a>? Because as it<br>
>>> stands right now there are two mako benchmarks that are not identical.<br>
>>> Otherwise we should talk at PyCon and figure this all out before we end up<br>
>>> with two divergent benchmark suites that are being independently maintained<br>
>>> (since we are all going to be running the same benchmarks on<br>
>>> <a href="http://speed.python.org" target="_blank">speed.python.org</a>).<br>
>><br>
>> No, I think it's a bad idea. First benchmarks should not change. It's<br>
>> fine to have a py3k benchmark next to py2 one, but we have 0 checkins<br>
>> to US benchmarks once we imported them.<br>
>><br>
>> Second, I find some of US benchmarks (those with hand unrolled loops,<br>
>> like json from the newer one) complete nonsense. If it does make any<br>
>> sense, it makes it only for cpython so we have no interest in having<br>
>> those benchmarks at all. If someone unrolls loops by hand and have a<br>
>> performance problem on pypy it's his own problem :)<br>
><br>
> Great attitude for a shared, common set of benchmarks. Remind me not to provide additional resources.<br>
<br>
</div>Jesse, do you really write code like that:<br>
<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(DICT)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
json.dumps(TUPLE)<br>
<br>
sorry for the long paste, but this is how the benchmark looks like. I<br>
would like to have a common set of benchmarks, but a *reasonable* one.<br>
Don't complain if I call it nonsense.<br></blockquote><div><br></div><div>I think Jesse's point has nothing to do with thinking the unladen benchmarks are perfect as-is and everything to do with divergence; instead of diverging from the unladen benchmarks it would have been nicer to simply fix them instead of having PyPy containing some changes that are legitimate but never added back to the unladen benchmark repo that the work started from. We have revision history so there is no need to keep a pristine version anywhere if that was the thinking. Otherwise I'm going to assume it's because of unladen being on svn back in the day or some historical reason.</div>
<div><br></div><div>So, to prevent this from either ending up in a dead-end because of this, we need to first decide where the canonical set of Python VM benchmarks are going to live. I say <a href="http://hg.python.org/benchmarks">hg.python.org/benchmarks</a> for two reasons. One is that Antoine has already done work there to port some of the benchmarks so there is at least some there that are ready to be run under Python 3 (and the tooling is in place to create separate Python 2 and Python 3 benchmark suites). Two, this can be a test of having the various VM contributors work out of <a href="http://hg.python.org">hg.python.org</a> if we are ever going to break the stdlib out for shared development. At worst we can simply take the changes made at pypy/benchmarks that apply to just the unladen benchmarks that exists, and at best merge the two sets (manually) into one benchmark suite so PyPy doesn't lose anything for Python 2 measurements that they have written and CPython doesn't lose any of its Python 3 benchmarks that it has created.</div>
<div><br></div><div>How does that sound?</div></div>