[Python-Dev] Re: new bytecode results

[Tim Peters]

[Terry Reedy]
> ...
> The problems of teaching to the test and programming to the benchmark
> are well-known.
> ...
> Your contest, my questions and comments.

Not to worry -- I made my living for 15 years writing optimizing compilers
in the highly competitive (at the time -- it competed itself to death)
supercomputer biz.  I've forgotten more dirty tricks than even a Perl guy
can pick up in a lifetime <wink>.

One of my favorites:  one of the rules in a particular competitive benchmark
was that editing the source code was not allowed.  Everyone had to run the
source file verbatim from the prospective customer.  One of Cray's
competitors (and, no, I'm not making this up) implemented a simple sed-like
subsystem driven by text on the compiler *command line*, which effectively
allowed them to edit the source code without physically changing the file.
They used this to dump all sorts of compiler directives into the benchmark
code, and even to replace a particular critical loop nest with a call to a
highly optimized library subroutine.

Their results were quite impressive.  But ours were even better:  general
global analysis deduced that the computed results weren't used for anything,
so we optimized away all the expensive loops.  Our results weren't all to
the good, though:  the program ran so fast then that the "operations per
second" output at the end overflowed to a negative integer.

Heh.  Believe it or not, customers were once so naive that they actually
looked at benchmark results as if they measured something relevant to their
business <wink>.

beauty-is-a-more-objective-measure-ly y'rs  - tim