I'm definitely interested and willing to clean up + contribute our benchmarks.

On a side note, I'm a bit skeptical that there can be a single benchmark suite that satisfies everyone. I would imagine that there will still be projects with specific use-cases they prioritize (such as Pyston with webserver workloads), or that have some idea that their users will be "non-representative" in some way. One example of that is the emphasis on warmup vs steady-state performance, which can be reflected in different measurement methodologies -- I don't think there's a single right answer to the question "how much does warmup matter".

But anyway, I'm still definitely +1 on the idea of merging all the benchmarks together, and I think that that will be better than the current situation. I'm imagining that we can at least have a common language for discussing these things ("Pyston prefers to use the flags --webserver --include-warmup"). I also see quite a few blog posts / academic papers on Python performance that seem to get led astray by the confusing benchmark situation, and I think having a blessed set of benchmarks (even if different people use them in different ways) would still be a huge step forward.

kmod

On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <brett@python.org> wrote:

On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fijall@gmail.com> wrote:

...

On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

On Wed, 13 Apr 2016 20:57:35 +0200 Maciej Fijalkowski <fijall@gmail.com> wrote:

...

Hi

I have a radical idea: to take a pypy benchmark suite, update the libraries to newer ones and replace python benchmarks with that. The main reason being that pypy has a much better coverage of things that are not microbenchmarks, the list (in json):

So why not consolidate all benchmarks together, instead of throwing away work already done?

Regards

Antoine.

Yeah, you can call it that too.

I also reached out to Pyston at https://gitter.im/dropbox/pyston over the weekend to see if they would want to participate as well.

So are we actually going to try and make this happen? I guess we should get people to vote on whether they like the idea enough before we hash out how we want to structure the new repository and benchmark suite.

I'm +1 on the idea, but I currently don't have the time to help beyond helping drive the email conversation.

---

Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed