Performance measurements

[Cedric Adjih]

Cameron Laird <claird at starbase.neosoft.com> wrote:
> In article <lcd7a9rldg.fsf at gaffa.mit.edu>,
> Douglas Alan  <nessus at mit.edu> wrote:
> 			.
> 			.
> 			.
>>I'm just quoting Guido from a while back.  This is, of course, for
>>doing number crunching using only bare Python and normal loops.  If
>>you use NumPy or PyAPL or some other highly optimized C-coded
>>extension to do matrix operations on large matrices, then you can of
>>course do as well as C, because your number crunching is really being
>>done by C.  This is assuming you can figure out a way to turn your
>>number crunching problem into operations on large matrices.
>>
>>All this should come as no shock -- 100 times slower for an
>>interpreted language is just par for a good interpreter.  Before the
>>spiffy Self compiler, state of the art was 10x slower than C for a
>>really bitchin' highly optimized Smalltalk compiler.
>>
>>|>oug
>
> Help me, Mr. Alan.  I remember a time when "100 times
> slower for an interpreted language [was] just par for
> a good interpreter."  I have a lot of trouble repro-
> ducing that today.  If it'll help, I can pull some of
> my '80s era hosts and interpreters from the closet. 
> Yes, I get a factor of between 100 and 300 for empty
> loops, but, as soon as they do anything useful, the
> multiple drops down to a range more like ten--definitely
> not a hundred.

I respectfully disagree. For most code that really need speed, a 
factor of 100 is plausible. x10 is probably the difference when you
are using the python libraries. Myself I've tried (again) to write
a alpha/beta solver for a game: the C++ version was x100 faster 
(quickly optimized to x1000), even though the board itself
was already in C++. That's okay because I expected it.

Another dramatic example of Python not top speed on some cases
is the raytracer entry CuttySark at last ICFP programming contest:
<http://www.crepuscule.com/icfp/> (eliminated because too slow) ;
CuttySark is 25000+ times slower than the top entry (partly
because of lack of algorithmic optimization), and 300+
times slower than a C++ entry, which, from the results, didn't
look optimized either. [and for a good laugh, just try to run
it on JPython.]

Or even an image compression algorithm I prototyped in Python:
it was 1000x slower than the first C++ version (now 10000x
slower than the somewhat optimized version).

Or even in the C parser I tried to interface with Python
(<http://www.crepuscule.com/ctool/>): it takes forever.

Then again, in all those cases, it's not a problem, since
python is used to write quickly a prototype, and low performance
is expected, but if Python is advertised to be 10x slower,
this will generate lots of disappointement.

> What I remember Guido saying is something like, "Maybe
> Python is only one one-hundredth as fast on do-nothing
> loops; I can live with that, that is, I have more im-
> portant things to spend my time on" (that's heavily
> paraphrased).
>
> I'll summarize:  I don't understand what point is
> being made when someone says, "Interpreters are 1% as
> fast as compilers", and the evidence all has to do with
> empty loops.  It makes me think I'm missing something.

I'd think it's true on any purely CPU intensive task in Python,
and it can be easily be worse.

> If you mollify the claim to 10% as fast, I ... well, in
> that case, I can at least believe we're looking at the
> same data.

I'd think it's true when taking into account Python lib,
including operations of strings, of course tasks
with lots of I/O, or generally data-structures fitting
those of Python, such as the following pi calculation for 
instance:
---
Prec=1000 ; I=pow(10L,Prec) ; vPi=0L ; k=0
while 1:
   v=(((I<<2)/(8*k+1)-(I<<1)/(8*k+4)-I/(8*k+5)-I/(8*k+6))>>(4*k))
   vPi=vPi+v
   k=k+1
   if v==0: break
print "iterations=", k, "pi=",vPi
---

> Again, I recognize the Lisp community has a long and
> interesting history with performance comparisons.  I'm
> not reflecting on that at all.  I'm talking about cur-
> rent Python and, let's say, gcc--I take those to be
> consistent with your intent.

Basically it somewhat depends on what you are doing.

-- Cedric