Benchmark that pypy doesn't do well on
Hi all, I've finally gotten around to open-sourcing some old code of mine (coincidentally, the predecessor to Pyston) which doesn't perform that well on PyPy. It's not the best benchmark, since it's not deterministic and produces different results on PyPy and CPython (due to dict ordering, I think), but PyPy seems to be consistently 20-30% slower than CPython, so I think the overall effect is reliable even if the exact numbers aren't meaningful. The benchmark is "real code" in that I simply added a benchmark mode to a real project, where the benchmark runs the exact same thing as a normal invocation (except for disabling some disk output); I'm definitely not trying to claim that the benchmark is "representative" in any sense, though. To run it: cd ~ # needs to be in the home directly unfortunately git clone https://github.com/kmod/icbd # set up python env time bash icbd/icbd/type_analyzer/run.sh bench On my machine, CPython runs it in 60s, but PyPy takes 75s; this is on PyPy 1.8 since I can't get a newer version, but I get similar results with 2.2.1. I should mention that the program doesn't use any non-stdlib modules, at least in "bench" mode. I've also tried to extract a part of the program that seemed to run significantly slower under PyPy, and got this microbenchmark: https://github.com/dropbox/pyston/blob/master/microbenchmarks/polymorphism.p... It takes about 13s for PyPy, and about 4s for CPython. This microbenchmark is only based on the final phase of the program (the part right before printing all the colored output), so I don't think it's necessarily that representative, it's just something I happened to notice was much slower. Just wanted to send it over in case you guys were interested. kmod
Hi Kevin, On 22 April 2014 21:29, Kevin Modzelewski <kmod@dropbox.com> wrote:
This suffers from extreme warm-up times. If I run it as posted, I get 9.1s on PyPy versus 4.4s on CPython. If I increase the number of iterations from 1k to 10k, I get 18.9s on PyPy versus 44s on CPython. The reasons for the large warm-up times are multiple: it's a highly recursive example; a core function in this recursion contains a loop that runs only twice; and the overall process is repeated 1000 times --- just before the JIT triggers compilation from the outermost level. It's an example where we should ideally detect that the loop typically runs twice, and unroll it. If we change the source code to manually unroll it, the four timings above become respectively 3.9s, 3.9s, 6.1s, 39s. A bientôt, Armin.
Hi Kevin, On 22 April 2014 21:29, Kevin Modzelewski <kmod@dropbox.com> wrote:
This suffers from extreme warm-up times. If I run it as posted, I get 9.1s on PyPy versus 4.4s on CPython. If I increase the number of iterations from 1k to 10k, I get 18.9s on PyPy versus 44s on CPython. The reasons for the large warm-up times are multiple: it's a highly recursive example; a core function in this recursion contains a loop that runs only twice; and the overall process is repeated 1000 times --- just before the JIT triggers compilation from the outermost level. It's an example where we should ideally detect that the loop typically runs twice, and unroll it. If we change the source code to manually unroll it, the four timings above become respectively 3.9s, 3.9s, 6.1s, 39s. A bientôt, Armin.
participants (2)
-
Armin Rigo -
Kevin Modzelewski