Hi all, I've finally gotten around to open-sourcing some old code of mine (coincidentally, the predecessor to Pyston) which doesn't perform that well on PyPy. It's not the best benchmark, since it's not deterministic and produces different results on PyPy and CPython (due to dict ordering, I think), but PyPy seems to be consistently 20-30% slower than CPython, so I think the overall effect is reliable even if the exact numbers aren't meaningful. The benchmark is "real code" in that I simply added a benchmark mode to a real project, where the benchmark runs the exact same thing as a normal invocation (except for disabling some disk output); I'm definitely not trying to claim that the benchmark is "representative" in any sense, though.
To run it:
# set up python env
time bash icbd/icbd/type_analyzer/run.sh bench
On my machine, CPython runs it in 60s, but PyPy takes 75s; this is on PyPy 1.8 since I can't get a newer version, but I get similar results with 2.2.1. I should mention that the program doesn't use any non-stdlib modules, at least in "bench" mode.
I've also tried to extract a part of the program that seemed to run significantly slower under PyPy, and got this microbenchmark:
It takes about 13s for PyPy, and about 4s for CPython. This microbenchmark is only based on the final phase of the program (the part right before printing all the colored output), so I don't think it's necessarily that representative, it's just something I happened to notice was much slower.
Just wanted to send it over in case you guys were interested.
kmod