Poor performance for Krakatau
In some cases, Pypy performs worse than CPython when running Krakatau, and it has gotten a lot worse from 2.3 to 2.5 My benchmark is https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (for future reference, commit d889c7f44723e6d66a3630681f0385f173317dc9) You can run it with pypy Krakatau\benchmark.py -path <path to Java rt.jar> I get the following timings CPython: 17.43s Pypy 2.3: 38.48s (build pypy-c-jit-71056-c8e3b8cbc843-win32) Pypy 2.4: 39.92s (build pypy-c-jit-72200-375133966c12-win32) Pypy 2.5: 49.94s (build pypy-c-jit-74404-9fd586fe0fe5-win32) Not only was it slower than CPython to begin with, but it got much worse recently.
Hi Robert. I've been looking at krakatau performance for a while, it's almost exclusively warmup time. We are going to address it, I hope rather sooner than later :-) On Mon, Nov 10, 2014 at 12:41 AM, Robert Grosse <n210241048576@gmail.com> wrote:
In some cases, Pypy performs worse than CPython when running Krakatau, and it has gotten a lot worse from 2.3 to 2.5
My benchmark is https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (for future reference, commit d889c7f44723e6d66a3630681f0385f173317dc9)
You can run it with pypy Krakatau\benchmark.py -path <path to Java rt.jar>
I get the following timings
CPython: 17.43s Pypy 2.3: 38.48s (build pypy-c-jit-71056-c8e3b8cbc843-win32) Pypy 2.4: 39.92s (build pypy-c-jit-72200-375133966c12-win32) Pypy 2.5: 49.94s (build pypy-c-jit-74404-9fd586fe0fe5-win32)
Not only was it slower than CPython to begin with, but it got much worse recently.
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Robert, On 10 November 2014 at 05:27, Maciej Fijalkowski <fijall@gmail.com> wrote:
I've been looking at krakatau performance for a while, it's almost exclusively warmup time. We are going to address it, I hope rather sooner than later :-)
We added Krakatau to the official benchmark suite. It turns out not to be exclusively warmup time: after the program is fully warmed up, it is still almost 2 times slower than CPython. We'll look at it at some point now that it's on speed.pypy.org and annoying us regularly :-) A bientôt, Armin & Carl Friedrich
I decided to update the benchmark today and compare it with pypy2.6. If anything, the performance issues with Pypy seem to have gotten worse with 2.6. As before, you can see the benchmark at https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (commit 234edc936b958596d843b91b963c4f61f56f2410) I also decided to compare it with different number of iterations for warmup. CPython: 12.65s Pypy2.6 100 iters: 36.83s 500 iters: 28.68s 1000 iters: 25.56s So performance does improve when increasing the amount of time spent warming up, but even with 1000 iterations, which is over a full minute, it is still half the speed of CPython. On Mon, Dec 8, 2014 at 7:49 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Robert,
On 10 November 2014 at 05:27, Maciej Fijalkowski <fijall@gmail.com> wrote:
I've been looking at krakatau performance for a while, it's almost exclusively warmup time. We are going to address it, I hope rather sooner than later :-)
We added Krakatau to the official benchmark suite. It turns out not to be exclusively warmup time: after the program is fully warmed up, it is still almost 2 times slower than CPython. We'll look at it at some point now that it's on speed.pypy.org and annoying us regularly :-)
A bientôt,
Armin & Carl Friedrich
you *eventually* get somewhere. Try running this: PYPYLOG=jit-summary:- pypy <other args> and see with what number of iterations the warmup time stops growing. Also we have vmprof these days so maybe it's a good time to look into the performance again, I'll keep that in mind On Sun, Aug 9, 2015 at 5:51 AM, Robert Grosse <n210241048576@gmail.com> wrote:
I decided to update the benchmark today and compare it with pypy2.6. If anything, the performance issues with Pypy seem to have gotten worse with 2.6.
As before, you can see the benchmark at https://github.com/Storyyeller/Krakatau/tree/pypy_benchmark (commit 234edc936b958596d843b91b963c4f61f56f2410)
I also decided to compare it with different number of iterations for warmup.
CPython: 12.65s Pypy2.6 100 iters: 36.83s 500 iters: 28.68s 1000 iters: 25.56s
So performance does improve when increasing the amount of time spent warming up, but even with 1000 iterations, which is over a full minute, it is still half the speed of CPython.
On Mon, Dec 8, 2014 at 7:49 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Robert,
On 10 November 2014 at 05:27, Maciej Fijalkowski <fijall@gmail.com> wrote:
I've been looking at krakatau performance for a while, it's almost exclusively warmup time. We are going to address it, I hope rather sooner than later :-)
We added Krakatau to the official benchmark suite. It turns out not to be exclusively warmup time: after the program is fully warmed up, it is still almost 2 times slower than CPython. We'll look at it at some point now that it's on speed.pypy.org and annoying us regularly :-)
A bientôt,
Armin & Carl Friedrich
participants (3)
-
Armin Rigo -
Maciej Fijalkowski -
Robert Grosse