[pypy-dev] Python FFI

[Stefan Behnel]

Armin Rigo, 16.05.2012 08:36:
> On Wed, May 16, 2012 at 7:01 AM, Stefan Behnel wrote:
>> I'm with Jean-Paul here.
> 
> ...for a reason that might be unrelated to Jean-Paul's original
> complain: it's not like Cython gives first-class object manipulation
> on the result of parsing its .pyx files...

I wasn't talking about .pyx files but .pxd files. While the first contain
actual (Cython) source code, which the tool we are discussing here would
want to see in a pure Python source file, .pxd files are meant to contain
C/C++ declarations that the Cython compiler uses to learn about external
code. (Also about C-level APIs of Cython generated modules, BTW, using a
PyCapsule based export mechanism. We'd eventually like to remove the need
for that with the CEP that Mark mentioned.)

Here is an example:

http://docs.cython.org/src/tutorial/clibraries.html

The first page or so should give you an idea. You can stop reading at the
point where I start implementing the Queue wrapper class, as that's
something that you would want to do in plain Python code instead.

There are ways to generate .pxd files from header files as a one-shot
operation, so that users can get going quickly and then start adapting the
declaration to what they want them to look like when writing their own code
against them. So this indirection allows them to take advantage of some
ambiguities given by C.


>> I don't think there's anything wrong with requiring declarations for
>> external C code to live in external modules.
> 
> I'm not against the idea of recognizing also Cython syntax extensions
> in the declarations, and indeed external files are a better idea than
> big inlined strings then.  But I would also like to try out different
> paths.  In one of them, these inline declarations would contain mostly
> just "#include <foo.h>" and the real header would be processed with
> the help of the C compiler.

Then take a look at some of the tools that Cython users have written to
generate .pxd files from header files. I heard of at least two that use
gccxml and clang respectively. Seems like Wim Lavrijsen has also started
doing the same thing all over again with cppyy. SWIG also does it, but
basically all on its own and without the helpful hand of a real C/C++ compiler.

I would really appreciate it if there was one way to do this that everyone
could just settle on and use, instead of throwing massive amounts of
separate developer time on the same issue again and again, that only lead
to tools that "work for what the author needed right now".

We actually have a GSoC project this year that tries to get a .pxd
generation tool up to a point where it becomes generally usable, based on
the gcc Python plugin, which sounds like a good idea to me. There's still
some time left to refocus this into something that also makes it more
widely usable, outside of a plain Cython context.

Stefan