On Fri, Jul 29, 2016 at 12:03 PM, Brett Cannon <brett@python.org> wrote:

On Thu, 28 Jul 2016 at 10:25 Victor Stinner <victor.stinner@gmail.com> wrote:

...

• Should we remove vendor copies of libraries and work with virtual environments? Not all libraries are available on PyPI :-/ See the requirements.txt file and TODO.

If they are not on PyPI then we should just drop the benchmark. And I say we do use virtual environments to keep the repo size down.

I think rather than using virtual environments which aren't truly supported by <3.3 anyway, we should instead make use of pip's --target, --root, and/or --prefix flags (whatever combination it takes, I haven't looked into it deeply) to install the packages into a particular dir which is then added to each benchmarked interpreter's PYTHONPATH. This way, we're sure that each interpreter is running exactly the same code.

Either way, I'm for not vendoring libraries. If the library disappears from PyPI, it's probably not an important workload anymore anyway.

...

Reminder: My final goal is to merge again all benchmarks suites (CPython, PyPy, Pyston, Pyjion, ...) into one unique project!

I hope this happens!

Me too! I'd also like to get the benchmark runner for speed.python.org set up to build and benchmark as many interpreters as possible.

-- Zach