[Cython] Cython-ctypes branch

[mark florisson]

On 18 October 2011 09:06, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Romain,
>
> I know your branch isn't "ready" in the sense that it's useful for the real
> world, but I'd like to find a way to get it merged, and to find a time frame
> for that. Otherwise, it will just bit-rot, which is certainly not what
> anyone wants.

I think you're more concerned about Cython playing a role in numpypy
than in bit-rot :) I certainly agree though, it would be great to have
some decent functionality in, if said functionality actually covers a
large subset of the Cython language, otherwise users might be tempted
to restrict themselves to certain functionality only.

> How would you judge your availability for this in the near
> future? I hope you're interested. :)
>
> For those who didn't follow the project, the branch lives here:
>
> https://github.com/hardshooter/CythonCTypesBackend
>
> The first thing that (IMHO, let's see if the others agree) needs to happen
> is that you should try to rebase it on the latest master branch. There were
> changes in the meantime that will not make this go clean. For example, the
> pipeline code was factored out of Main.py into a separate module
> Pipeline.py, so you will have to migrate your pipeline changes manually.
> That shouldn't be too hard, though, and it's the only major conflict that I
> currently anticipate.
>
> There's a test runner change in the master branch that will allow you to
> select the tested backends with a positive list, i.e. as in
>
>    runtests.py --backends=c,cpp
>
> You'd want to add the ctypes backend here. The "--no-cpp" etc. set of
> switches become very unwieldy as new backends are added.
>
> You will also notice that Cython gained a couple of new features and syntax
> since you started, specifically fused types, an extended array syntax for
> memoryviews and parallel OpenMP loops. I'm not sure how (or even if) they
> will translate to the Python backend. I think all of them will need a
> dedicated implementation in some way, which is very unfortunate. But I don't
> think that has to bother us for the moment.

For OpenMP you might not actually need to do anything at all, it
should already be supported in pure mode. Fused types and memoryviews
are harder (as is the older buffer support). I'm not even sure if/how
pypy's buffer support works.

There is also support for pure-mode fused types, but to a very limited
extend, i.e. you can do cython.fused_type(my-type-list) to create a
fused type, but you don't actually generate any actual specializations
unless you compile it with Cython.

As for actual fused types support, I think you can replace-and-wrap
fused functions at runtime with an instance of a generated
FusedFunction class that is indexable and callable and does the
necessary instance checks (in case of 'def' or 'cpdef'). You will also
know at compile-time whether a certain cdef or cpdef call is valid,
and you can basically do the same trick as we do in C: generate
multiple specializations and choose which one to call. But seeing that
compile time checks will ensure that you can only do an intersection
of all possible operations, I think you might only need to do this for
the case where you may either get a ctypes object or a normal python
object (if you make weird combinations of types to fuse).

In any case, I agree that leaving buffers and fused types for now in
the ctypes support is probably the best idea.

I'm not really familiar with RPython, but would it in any way be
feasible to have Cython generate RPython code? That may just make
things easier and more efficient to implement.

> I recreated the Jenkins build and test jobs for your branch:
>
> https://sage.math.washington.edu:8091/hudson/view/dev-romain/
>
> There's currently a unit test failure in the build job that keeps me from
> trying the subsequent test runs. It looks trivial, though, so if you could
> push a fix, I can make sure the build and test jobs work as expected. That
> will give us an idea about the current status of your code.
>
> I also noticed that the ctypes_configure script is not Py3 clean, so we
> can't currently test your code on that platform. 2to3 may be able to do the
> job, but the package needs fixing upstream.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>