stefan_ml at behnel.de
Sat Apr 30 09:09:24 CEST 2011
Vitja Makarov, 30.04.2011 08:08:
> Can I run this tests at home?
Don't expect it to come shrink wrapped. It took me a while to get the setup
running on Hudson/Jenkins, and the scripts that do it aren't completely
trivial. You can take a look at the config of the job that runs them:
I'm currently using the PyPy test suite at
instead of the future CPython one at
simply because the PyPy suite has more benchmarks (that work for us ;).
The major problem with the test suite is that the unladen swallow people
who wrote the test runner somehow got the paranoid idea that they had to
control which environment variables were set during the run, so they chose
to drop the entire environment and use their own one. They clearly never
tried to run distutils in a benchmark, which requires some intimate
knowledge about the environment it runs in.
So you basically need a wrapper around cythonrun that properly sets up the
environment for you. Take a look at what I did in the build job, it
generates a script that sets up the environment, and then passes that into
the benchmark runner.
You also need a "usercustomize.py" script that installs pyximport (but only
for the Cython runs, not for plain CPython!), so that it's not only the
benchmark script that gets compiled but also its dependencies. I'm
currently blacklisting the stdlib modules ['threading', 'socket',
'gettext', 'locale'] in pyximport.PyxImporter.blocked_modules when I run it
locally, simply because they don't work but have a substantial impact on
the running code.
The basic command I'm using to run the suite is:
python runner.py -a ",-Xauto_cpdef=True" \
-p ./cythonrun.sh --baseline=/path/to/python \
--fast -o results.json
The CPython test runner is a tad better here, because it allows you to
provide at least a whitelist of environment variables that are being passed
through. I'm currently trying to set up a second benchmark job to run the
(smaller) CPython benchmark suite in Py3 mode. The command to run that
suite is more like
python perf.py -a ',-Xauto_cpdef=true' \
/path/to/python ./cythonrun.sh \
and it doesn't seem to have json output. (Seriously, I can't believe they
even broke "PATH" in their runner script!).
There is also a GSoC project that aims to fix up the benchmark suite for
the different Python implementations in order to set up a
"speed.python.org" site that compares them. We should stay involved in that.
More information about the cython-devel