make faster Richards benchmark

Peter Hansen peter at engcorp.com
Thu May 13 15:34:42 EDT 2004


Jack Diederich wrote:

> On Thu, May 13, 2004 at 09:15:58AM -0700, Duncan Lissett wrote:
>>The pointless set/get methods are pointless in the other language
>>implementations as well - that's the point. 
> 
> I would leave python and some other languages out of the comparison.
> You can do near line-for-line translations for languages in the same class
> - say C++ and Java, or Assembly and C.  Requiring the python/perl/ruby
> versions to look like a C++ program just measures how badly C++ maps to 
> Python, and not much else.
> 
> I've even seen some line-for-line translations in production use that
> use python list-of-strings where the orignal version used char arrarys.
> You can imagine what that does for performance *shudder*.
> 
> Writing benchmarks is just hard, if you allow people to solve the problem
> in whatever way they like you end up measuring how good a coder the
> language A guy is compared to the language B submitter.

In the case of the Richards benchmark, it's better to say "specifying
benchmarks is hard".  The problem that I saw with it is that it
describes the problem in terms that are inappropriate for many of the
languages it has been translated to, effectively constraining the
implementation in ways that are more suitable to certain languages
than others.

One key example is the requirement for "global registers" which act
similarly to how the low-level registers in the CPU have to be handled,
specifically by saving and restoring them in a task-local context
whenever a task switch occurs.  This is clearly what Richards really
wanted, but I'd argue that the approach is ill-conceived, especially
in this century...

On the other hand, if the specification had been more thoroughly
thought out, or perhaps modernized is a more appropriate way of looking
at it, then it would describe the expected *use and behaviour* of the
tasks in terms of black box testability... in other words, it shouldn't
matter that "global registers" are used, but merely that the
benchmark produces certain results.

Without those tests, it's fairly open to interpretation just how
much deviation from the original BCPL implementation is appropriate.
Duncan L's ideas for a generator-based approach are interesting,
though there are probably those who would argue that it wouldn't
follow the benchmark specs exactly.  (Especially if it showed
performance much closer to C. <wink>)

-Peter



More information about the Python-list mailing list