I'm definitely interested and willing to clean up + contribute our benchmarks.
On a side note, I'm a bit skeptical that there can be a single benchmark suite that satisfies everyone. I would imagine that there will still be projects with specific use-cases they prioritize (such as Pyston with webserver workloads), or that have some idea that their users will be "non-representative" in some way. One example of that is the emphasis on warmup vs steady-state performance, which can be reflected in different measurement methodologies -- I don't think there's a single right answer to the question "how much does warmup matter".
But anyway, I'm still definitely +1 on the idea of merging all the
benchmarks together, and I think that that will be better than the current
situation. I'm imagining that we can at least have a common language for
discussing these things ("Pyston prefers to use the flags --webserver --include-warmup"). I also see quite a few blog posts / academic papers
on Python performance that seem to get led astray by the confusing
benchmark situation, and I think having a blessed set of benchmarks (even
if different people use them in different ways) would still be a huge step
forward.
kmod
On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <brett@python.org> wrote:
On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fijall@gmail.com> wrote:
On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 13 Apr 2016 20:57:35 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi
I have a radical idea: to take a pypy benchmark suite, update the libraries to newer ones and replace python benchmarks with that. The main reason being that pypy has a much better coverage of things that are not microbenchmarks, the list (in json):
So why not consolidate all benchmarks together, instead of throwing away work already done?
Regards
Antoine.
Yeah, you can call it that too.
I also reached out to Pyston at https://gitter.im/dropbox/pyston over the weekend to see if they would want to participate as well.
So are we actually going to try and make this happen? I guess we should get people to vote on whether they like the idea enough before we hash out how we want to structure the new repository and benchmark suite.
I'm +1 on the idea, but I currently don't have the time to help beyond helping drive the email conversation.
Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed