[Speed] Cython's view on a common benchmark suite

[Stefan Behnel]

Maciej Fijalkowski, 02.02.2012 12:12:
> On Thu, Feb 2, 2012 at 1:09 PM, Maciej Fijalkowski wrote:
>> On Thu, Feb 2, 2012 at 10:21 AM, Stefan Behnel wrote:
>>> We would like to join in on speed.python.org, once it's clear how the
>>> benchmarks will be run and how the data uploads work and all that. It
>>> already proved a bit tricky to get Cython integrated with the benchmark
>>> runner on our side, and I'm planning to rewrite that integration at some
>>> point, but it should already be doable to get "something" to work now.
>>
>> Can you come up with a script that does "cython <a python program>"?
>> that would simplify a lot

Yes, I have something like that, but it's a whole bunch of "do this, add
that, then run something". It mostly works (as you can see from the link
above), but it needs some serious reworking.

Basically, it compiles and starts the main program, and then enables
on-the-fly compilation of modules in sitecustomize.py by registering an
import hook. I'll see if I can get the script wrapped up a tiny bit so that
it becomes usable for speed.python.org.

Any way I could get an account on the machine? Would make it easier to test
it there.


>>> I should also note that we don't currently support the whole benchmark
>>> suite, so there must be a way to record individual benchmark results even
>>> in the face of failures in other benchmarks. Basically, speed.python.org
>>> would be useless for us if a failure in a single benchmark left us without
>>> any performance data at all, because it will still take us some time to get
>>> to 100% compliance and we would like to know if anything on that road has a
>>> performance impact. Currently, we apply a short patch that adds a
>>> try-except to the benchmark runner's main loop before starting the
>>> measurements, because otherwise it would just bail out completely on a
>>> single failure. Oh, and we also patch the benchmarks to remove references
>>> to __file__ because of CPython issue 13429, although we may be able to work
>>> around that at some point, specifically when doing on-the-fly compilation
>>> during imports.
>>
>> I think it's fine to mark certain benchmarks not to be runnable under
>> certain platforms. For example it's not like jython will run twisted
>> stuff.

... oh, and we'd like to know when it suddenly starts working. ;)

So, I think catching and ignoring (or logging) errors is the best way to go
about it.


>>> Another test that is useless for us is the "mako" benchmark, because most
>>> of what it does is to run generated code. There is currently no way for
>>> Cython to hook into that, so we're out of the game here.
>>
>> Well, if you want cython to be considered python I think this is a
>> pretty crucial feature no?

Oh, we have that feature, it's called CPython. The thing is that Cython
doesn't get to see the generated sources, so it won't compile them and
instead, CPython ends up executing the code at normal interpreted speed. So
there's nothing gained by running the benchmark at all. And even if we
found a way to hook into this machinery, I doubt that the static compiler
overhead would make this any useful. The whole purpose of generating code
is that it likely will not look the same the next time you do it (well,
outside of benchmarks, that is), so even a cache is unlikely to help much
for real code. It's like PyPy running code in interpreting mode before it
gets compiled, except that Cython will never compile this code, even if it
turns out to be worth it.

Personally, I rather consider it a feature that users can employ exec()
from their Cython code to run code in plain CPython (for whatever reason).

Stefan