Re: [Speed] Cython's view on a common benchmark suite
Maciej Fijalkowski, 02.02.2012 12:12:
On Thu, Feb 2, 2012 at 1:09 PM, Maciej Fijalkowski wrote:
On Thu, Feb 2, 2012 at 10:21 AM, Stefan Behnel wrote:
We would like to join in on speed.python.org, once it's clear how the benchmarks will be run and how the data uploads work and all that. It already proved a bit tricky to get Cython integrated with the benchmark runner on our side, and I'm planning to rewrite that integration at some point, but it should already be doable to get "something" to work now.
Can you come up with a script that does "cython <a python program>"? that would simplify a lot
Yes, I have something like that, but it's a whole bunch of "do this, add that, then run something". It mostly works (as you can see from the link above), but it needs some serious reworking.
Basically, it compiles and starts the main program, and then enables on-the-fly compilation of modules in sitecustomize.py by registering an import hook. I'll see if I can get the script wrapped up a tiny bit so that it becomes usable for speed.python.org.
Any way I could get an account on the machine? Would make it easier to test it there.
I should also note that we don't currently support the whole benchmark suite, so there must be a way to record individual benchmark results even in the face of failures in other benchmarks. Basically, speed.python.org would be useless for us if a failure in a single benchmark left us without any performance data at all, because it will still take us some time to get to 100% compliance and we would like to know if anything on that road has a performance impact. Currently, we apply a short patch that adds a try-except to the benchmark runner's main loop before starting the measurements, because otherwise it would just bail out completely on a single failure. Oh, and we also patch the benchmarks to remove references to __file__ because of CPython issue 13429, although we may be able to work around that at some point, specifically when doing on-the-fly compilation during imports.
I think it's fine to mark certain benchmarks not to be runnable under certain platforms. For example it's not like jython will run twisted stuff.
... oh, and we'd like to know when it suddenly starts working. ;)
So, I think catching and ignoring (or logging) errors is the best way to go about it.
Another test that is useless for us is the "mako" benchmark, because most of what it does is to run generated code. There is currently no way for Cython to hook into that, so we're out of the game here.
Well, if you want cython to be considered python I think this is a pretty crucial feature no?
Oh, we have that feature, it's called CPython. The thing is that Cython doesn't get to see the generated sources, so it won't compile them and instead, CPython ends up executing the code at normal interpreted speed. So there's nothing gained by running the benchmark at all. And even if we found a way to hook into this machinery, I doubt that the static compiler overhead would make this any useful. The whole purpose of generating code is that it likely will not look the same the next time you do it (well, outside of benchmarks, that is), so even a cache is unlikely to help much for real code. It's like PyPy running code in interpreting mode before it gets compiled, except that Cython will never compile this code, even if it turns out to be worth it.
Personally, I rather consider it a feature that users can employ exec() from their Cython code to run code in plain CPython (for whatever reason).
Stefan
Oh, we have that feature, it's called CPython. The thing is that Cython doesn't get to see the generated sources, so it won't compile them and instead, CPython ends up executing the code at normal interpreted speed. So there's nothing gained by running the benchmark at all. And even if we found a way to hook into this machinery, I doubt that the static compiler overhead would make this any useful. The whole purpose of generating code is that it likely will not look the same the next time you do it (well, outside of benchmarks, that is), so even a cache is unlikely to help much for real code. It's like PyPy running code in interpreting mode before it gets compiled, except that Cython will never compile this code, even if it turns out to be worth it.
Personally, I rather consider it a feature that users can employ exec() from their Cython code to run code in plain CPython (for whatever reason).
Yes, ok, but I believe this should mean "Cython does not give speedups on this benchmark" and not "we should modify the benchmark".
Cheers, fijal
participants (2)
-
Maciej Fijalkowski
-
Stefan Behnel