[Cython] PyCon-DE wrap-up by Kay Hayen

[Stefan Behnel]

mark florisson, 09.10.2011 19:57:
> On 9 October 2011 18:35, Stefan Behnel wrote:
>> One of the impressions I took out of the technical discussions with Kay is
>> that there isn't really a good reason why Cython should refuse to duplicate
>> some of the inner mechanics of CPython for optimisation purposes. Nuitka
>> appears to be somewhat more aggressive here, partly because Kay doesn't
>> currently care all that much about portability (e.g. to Python 3).
>
> Interesting. What kind of (significant) optimizations could be made by
> duplicating code? Do you want to duplicate entire functions or do you
> want to inline parts of those?

I was mainly referring to things like direct access to type/object struct 
fields and little things like that. They can make a difference especially 
in loops, compared to calling into a generic C-API function. For example, 
we could have our own interned implementation of PyDict_Next(). I'm not 
very impressed by the performance of that C-API function - repeated calls 
to GetItem can be faster than looping over a dict with PyDict_Next()!

That being said, I wasn't referring to any specific changes. It was more of 
a general remark about the invisible line that we currently draw in Cython.


> I actually think we should not get too tied to CPython, e.g. what if
> PyPy gets a CPython compatible API

It already implements a part of the C-API:

http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html

However, if we really want to support it at that level, there's likely more 
to do than just removing low-level optimisations. And that would take the 
normal route that we always use: macros and conditionally compiled inline 
functions. The mere fact that we try to support different targets doesn't 
mean that we should stop optimising for specific targets. The same is true 
for different versions of CPython, where we often use better optimisations 
in newer releases, without sacrificing backwards compatibility.

Personally, I think that supporting PyPy at the Python level is a lot more 
interesting, although it may be easier to get it working at the cpyext level.


> or possibly a subset like PEP 384?

That's currently not very interesting since there are basically no C 
extensions around (generated or hand written) that restrict themselves to 
that API. Supporting it in Cython would mean that we have to rewrite huge 
parts of the generated C code. It's not even clear to me yet that we *can* 
implement all of Cython's features based on PEP 384. For example, fast 
indexing into lists and tuples is basically a no-no in the restricted 
C-API. There are tons of rather unexpected restrictions like this.

Stefan