2013/5/30 Nathan Hurst <njh@njhurst.com>
> Does it still make sense to add programs like this to our benchmarks?
> So far, our benchmarks are "real-life" examples.  The benchmarks like
> above are completely missing the point of PyPy, as they don't stress
> at all the Python interpreter part.  There are also other cases where
> PyPy's performance is very bad, like cpyext on an extension module
> with lots of small C API calls.  I believe that it would still make
> sense to list such cases in the official benchmark, and have the
> descriptions of the benchmarks explain what's wrong with them.

I agree that you should include them, I disagree that they are
'wrong'.  They measure the overhead of a C call.  Why should a C call
be slower in pypy than cpython?  Presumably it could be compiled down
to the appropriate instructions and then out-perform cpy.

The C API here is the one of the CPython interpreter (PyLong_FromLong &co)
To support it PyPy has to emulate many aspects, specially the fact that pypy
objects are movable memory, and a PyObject* pointer is not supposed to change.

To get fair benchmarks, those extension modules should be rewritten,
with cffi for example: its C calls have very little overhead,
and it integrates very well with the rest of the PyPy interpreter.

Amaury Forgeot d'Arc