[Cython] Cython-ctypes branch

[Stefan Behnel]

mark florisson, 18.10.2011 18:50:
> On 18 October 2011 09:06, Stefan Behnel wrote:
>> I know your branch isn't "ready" in the sense that it's useful for the real
>> world, but I'd like to find a way to get it merged, and to find a time frame
>> for that. Otherwise, it will just bit-rot, which is certainly not what
>> anyone wants.
>
> I think you're more concerned about Cython playing a role in numpypy
> than in bit-rot :)

Both, in a way. But I think I'm really more concerned about the code dying 
from branch divergence. It's a really nice feature that's worth keeping and 
growing.

Obviously, it also means it'll be more work for us to add language 
features, as we have an additional backend to support. But it's also a good 
proof that we're prepared to deal with that, and may eventually lead to 
more backends being added.


> I certainly agree though, it would be great to have
> some decent functionality in, if said functionality actually covers a
> large subset of the Cython language, otherwise users might be tempted
> to restrict themselves to certain functionality only.

Well, it's a pretty experimental feature. Users should expect it to be 
limited in functionality and have bugs. If they want to use it, they'll 
have to accept some drawbacks for the time being. I think that's fine.


>> You will also notice that Cython gained a couple of new features and syntax
>> since you started, specifically fused types, an extended array syntax for
>> memoryviews and parallel OpenMP loops. I'm not sure how (or even if) they
>> will translate to the Python backend. I think all of them will need a
>> dedicated implementation in some way, which is very unfortunate. But I don't
>> think that has to bother us for the moment.
>
> For OpenMP you might not actually need to do anything at all, it
> should already be supported in pure mode. Fused types and memoryviews
> are harder (as is the older buffer support). I'm not even sure if/how
> pypy's buffer support works.

I have no idea. It doesn't even have to have such a feature. It's not 
required for language compliance, for one.


> There is also support for pure-mode fused types, but to a very limited
> extend, i.e. you can do cython.fused_type(my-type-list) to create a
> fused type, but you don't actually generate any actual specializations
> unless you compile it with Cython.
>
> As for actual fused types support, I think you can replace-and-wrap
> fused functions at runtime with an instance of a generated
> FusedFunction class that is indexable and callable and does the
> necessary instance checks (in case of 'def' or 'cpdef'). You will also
> know at compile-time whether a certain cdef or cpdef call is valid,
> and you can basically do the same trick as we do in C: generate
> multiple specializations and choose which one to call. But seeing that
> compile time checks will ensure that you can only do an intersection
> of all possible operations, I think you might only need to do this for
> the case where you may either get a ctypes object or a normal python
> object (if you make weird combinations of types to fuse).

Yes, I was expecting problems when using C types. However, just because you 
pass different ctypes wrapped C values into generic code doesn't mean you 
have to split the function. It depends on what the function actually does, 
i.e. if the different types lead to different *Python* code. However, I 
guess that will quickly be the case as soon as you use typed variables in 
side of the function that need a ctypes initialisation in the generated code.


> In any case, I agree that leaving buffers and fused types for now in
> the ctypes support is probably the best idea.

Certainly helps in getting this out of the door.


> I'm not really familiar with RPython, but would it in any way be
> feasible to have Cython generate RPython code? That may just make
> things easier and more efficient to implement.

I never used RPython either, but my guess is that this would be quite 
involved. You'd loose the more or less 1:1 mapping from Cython code to 
Python code and would have to replace some constructs or code patterns by 
different Python code.

Anyway, I think PyPy can optimise Python code just fine.

Stefan