# [pypy-svn] r60567 - in pypy/extradoc/talk/ecoop2009: . benchmarks

antocuni at codespeak.net antocuni at codespeak.net
Thu Dec 18 14:15:46 CET 2008

Author: antocuni
Date: Thu Dec 18 14:15:45 2008
New Revision: 60567

Modified:
Log:
add tables showing the results. They need some restyling as they are very
ugly.

==============================================================================
+++ pypy/extradoc/talk/ecoop2009/benchmarks.tex	Thu Dec 18 14:15:45 2008
@@ -75,23 +75,79 @@
return b
\end{lstlisting}

+\anto{these tables are ugly}

-
-Table XXX and figure XXX show the time spent to calculate the factorial of
-various numbers, with and without the JIT.  Table XXX and figure XXX show the
-same informations for the Fibonacci program.
-
-Note that do get meaningful timings, we had to calculate the factorial and
-Fibonacci of very high numbers.  This means that the results are incorrect due
-to overflow, but since all the runnings overflows in the very same way, the
-timings are still comparable. \anto{I think we should rephrase this sentence}.
-
-As we can see, the code generated by the JIT is almost 500 times faster than
-the non-jitted case, and it is only about 1.5 times slower than the same
-algorithm written in C\#: the difference in speed it is probably due to both
-the fact that the current CLI backend emits slightly non-optimal code and that
-the underyling .NET JIT compiler is highly optimized to handle bytecode
-generated by C\# compilers.
+\begin{table}[ht]
+  \begin{tabular}{|l|r|r|r|r||r|r|}
+    \hline
+    \textbf{n} &
+    \textbf{Interp} &
+    \textbf{JIT} &
+    \textbf{JIT 2} &
+    \textbf{C\#} &
+    \textbf{Interp/JIT 2} &
+    \textbf{JIT 2/C\#} \\
+    \hline
+
+    $10$    &   0.031  &  0.422  &  0.000  &  0.000  &      N/A  &    N/A \\
+    $10^7$  &  30.984  &  0.453  &  0.047  &  0.031  &  661.000  &  1.500 \\
+    $10^8$  &     N/A  &  0.859  &  0.453  &  0.359  &      N/A  &  1.261 \\
+    $10^9$  &     N/A  &  4.844  &  4.641  &  3.438  &      N/A  &  1.350 \\
+
+    \hline
+
+  \end{tabular}
+  \caption{Factorial benchmark}
+  \label{tab:factorial}
+\end{table}
+
+
+\begin{table}[ht]
+  \begin{tabular}{|l|r|r|r|r||r|r|}
+    \hline
+    \textbf{n} &
+    \textbf{Interp} &
+    \textbf{JIT} &
+    \textbf{JIT 2} &
+    \textbf{C\#} &
+    \textbf{Interp/JIT 2} &
+    \textbf{JIT 2/C\#} \\
+    \hline
+
+    $10$    &   0.031  &  0.453  &  0.000  &  0.000  &       N/A  &  N/A   \\
+    $10^7$  &  29.359  &  0.469  &  0.016  &  0.016  &  1879.962  &  0.999 \\
+    $10^8$  &     N/A  &  0.688  &  0.250  &  0.234  &       N/A  &  1.067 \\
+    $10^9$  &     N/A  &  2.953  &  2.500  &  2.453  &       N/A  &  1.019 \\
+
+    \hline
+
+  \end{tabular}
+  \caption{Fibonacci benchmark}
+  \label{tab:fibo}
+\end{table}
+
+
+Tables \ref{tab:factorial} and \ref{tab:fibo} show the time spent to calculate
+the factorial and Fibonacci for various $n$.  As we can see, for small values
+of $n$ the time spent running the JIT compiler is much higher than the time
+spent to simply interpret the program.  This is an expected result, as till
+now we only focused on optimizing the compiled code, not the compilation
+process itself.
+
+On the other hand, to get meaningful timings we had to use very high values of
+$n$.  This means that the results are incorrect due to overflow, but since all
+the runnings overflow in the very same way, the timings are still
+comparable. \anto{I think we should rephrase this sentence}.  For $n$ greater
+than $10^7$, we did not run the interpreted program as it would have took too
+much time, without adding anything to the discussion.
+
+As we can see, the code generated by the JIT can be up to ~1800 times faster
+than the non-jitted case.  Moreover, it often runs at the same speed as the
+equivalent program written in C\#, being only 1.5 slower in the worst case.
+
+The difference in speed it is probably due to both the fact that the current
+CLI backend emits slightly non-optimal code and that the underyling .NET JIT
+compiler is highly optimized to handle bytecode generated by C\# compilers.

\subsection{Object-oriented features}

@@ -142,10 +198,36 @@
as a local variable.  As a result, the generated code results in a simple loop

-Table XXX show the time spent to run the benchmark with various input
-arguments. Again, we can see that the jitted code is up to 500 times faster
-than the interpreted one.  Moreover, the code generated by the JIT is
-\textbf{faster} than the equivalent C\# code.
+\begin{table}[ht]
+  \begin{tabular}{|l|r|r|r|r||r|r|}
+    \hline
+    \textbf{n} &
+    \textbf{Interp} &
+    \textbf{JIT} &
+    \textbf{JIT 2} &
+    \textbf{C\#} &
+    \textbf{Interp/JIT 2} &
+    \textbf{JIT 2/C\#} \\
+    \hline
+
+    $10$    &   0.031  &  0.453  &  0.000  &  0.000  &      N/A  &  N/A   \\
+    $10^7$  &  43.063  &  0.516  &  0.047  &  0.063  &  918.765  &  0.750 \\
+    $10^8$  &     N/A  &  0.875  &  0.453  &  0.563  &      N/A  &  0.806 \\
+    $10^9$  &     N/A  &  4.188  &  3.672  &  5.953  &      N/A  &  0.617 \\
+
+    \hline
+
+  \end{tabular}
+  \caption{Accumulator benchmark}
+  \label{tab:accumulator}
+\end{table}
+
+
+Table \ref{tab:accumulator} show the results for the benchmark.  Again, we can
+see that the speedup of the JIT over the interpreter is comparable to the
+other two benchmarks.  However, the really interesting part is the comparison
+with the equivalent C\# code, as the code generated by the JIT is
+\textbf{faster}.

Probably, the C\# code is slower because:

==============================================================================
+++ pypy/extradoc/talk/ecoop2009/benchmarks/results.txt	Thu Dec 18 14:15:45 2008
@@ -1,21 +1,24 @@
Factorial
N               Interp       JIT1          JIT2        C#
-10              0,03125      0,421875      0           0
-10000000        30,98438     0,453132      0,046875    0,03125
-100000000       N/A          0,859375      0,453125    0,35937
-1000000000      N/A          4,84375       4,640625    3,4375
+10              0.03125      0.421875      0           0
+10000000        30.98438     0.453132      0.046875    0.03125
+100000000       N/A          0.859375      0.453125    0.35937
+1000000000      N/A          4.84375       4.640625    3.4375

Fibonacci
N               Interp       JIT1          JIT2        C#
-10              0,0312576    0,45313       0           0
-10000000        29,359367    0,46875       0,015617    0,015625
-100000000       N/A          0,6875        0,25001     0,234375
-1000000000      N/A          2,9531        2,5         2,453125
+10              0.0312576    0.45313       0           0
+10000000        29.359367    0.46875       0.015617    0.015625
+100000000       N/A          0.6875        0.25001     0.234375
+1000000000      N/A          2.9531        2.5         2.453125

Accumulator
N               Interp       JIT1          JIT2        C#
-10              0,03125      0,453132      0           0
-10000000        43,0625      0,515625      0,04687     0,0625
-100000000       N/A          0,87500       0,453125    0,5625
-1000000000      N/A          4,18750       3,67187     5,953125
+10              0.03125      0.453132      0           0
+10000000        43.0625      0.515625      0.04687     0.0625
+100000000       N/A          0.87500       0.453125    0.5625
+1000000000      N/A          4.18750       3.67187     5.953125
+
+
+