[pypy-dev] Benchmark that pypy doesn't do well on

Kevin Modzelewski kmod at dropbox.com
Tue Apr 22 21:29:51 CEST 2014


Hi all, I've finally gotten around to open-sourcing some old code of mine
(coincidentally, the predecessor to Pyston) which doesn't perform that well
on PyPy.  It's not the best benchmark, since it's not deterministic and
produces different results on PyPy and CPython (due to dict ordering, I
think), but PyPy seems to be consistently 20-30% slower than CPython, so I
think the overall effect is reliable even if the exact numbers aren't
meaningful.  The benchmark is "real code" in that I simply added a
benchmark mode to a real project, where the benchmark runs the exact same
thing as a normal invocation (except for disabling some disk output); I'm
definitely not trying to claim that the benchmark is "representative" in
any sense, though.

To run it:
cd ~ # needs to be in the home directly unfortunately
git clone https://github.com/kmod/icbd
# set up python env
time bash icbd/icbd/type_analyzer/run.sh bench

On my machine, CPython runs it in 60s, but PyPy takes 75s; this is on PyPy
1.8 since I can't get a newer version, but I get similar results with
2.2.1.  I should mention that the program doesn't use any non-stdlib
modules, at least in "bench" mode.


I've also tried to extract a part of the program that seemed to run
significantly slower under PyPy, and got this microbenchmark:
https://github.com/dropbox/pyston/blob/master/microbenchmarks/polymorphism.py

It takes about 13s for PyPy, and about 4s for CPython.  This microbenchmark
is only based on the final phase of the program (the part right before
printing all the colored output), so I don't think it's necessarily that
representative, it's just something I happened to notice was much slower.

Just wanted to send it over in case you guys were interested.

kmod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140422/c498e1c1/attachment.html>


More information about the pypy-dev mailing list