On Thu, 11 Feb 2016 18:35:29 +0000 Brett Cannon brett@python.org wrote:

Maybe we should just have a requirements.txt file for Python 2 and another for Python 3 that are pegged to specific versions? We could even install things into a venv for isolation.

How does this impact interaction with the benchmarks suite? E.g. does it increase the time of running a couple of benchmarks? Does it make it easier or harder to benchmark a work-in-progress patch for whatever interpreter?

If we go this route then we could make the benchmark suite a package on PyPI and have people install the benchmark suite and then have instructions to run pip on the requirements files that we embed in the package.

I'm not fond of encouraging random users to run the benchmarks suite without understanding what they're doing, and starting throwing around pointless numbers and misconceptions about performance (which are then very hard to fight since people tend to be irrationally captivated by "performance numbers"). The benchmarks suite is mostly a tool for developers of Python implementations, not the greater public.

Having the benchmarks suite only available through hg or git kind of discourages those tendencies.

Regards

Antoine.