[pypy-dev] Python FFI

[Stefan Behnel]

Maciej Fijalkowski, 15.05.2012 17:39:
> On Tue, May 15, 2012 at 5:16 PM, Stefan Behnel wrote:
>> Armin Rigo, 15.05.2012 16:23:
>>> Fijal and me would like to raise interest among various groups of
>>> people about building a better ctypes replacement for Python.
>>> [...]
>>> The working alternatives nowadays are Cython and ctypes.  For Cython
>>> you have to learn a whole new language; we can argue infinitely about
>>> preferences, but from my point of view it's mostly just a nicer way to
>>> write CPython C extensions using --- instead of C --- a custom
>>> language.
>>
>> Well, it's basically Python with optional static types, which are mostly
>> provided in C(-like) notation. I find it surprisingly similar to LuaJIT's
>> FFI, since you mentioned that anyway.
>>
>>
>>> Also, because it would parse strings, we can also add support for
>>> parsing strings in the format of Cython declarations.  This would
>>> still lower the entry barrier, and have the nice effect that the user
>>> can sometimes copy-and-paste Cython declarations, instead of
>>> copy-and-pasting carefully chosen pieces of a potentially messy .h
>>> file full of custom macros (see for example /usr/include/zlib.h).  The
>>> benefit would be that, unlike Cython, he gets to call the declared
>>> functions directly from his regular Python code.
>>
>> I assume you are aware of Cython's pure Python mode? It doesn't currently
>> support calling into C, but that's a matter of doing it, not so much of
>> researching how it could be done.
>>
>> Personally, I've always been thinking of Cython as the best FFI that Python
>> could ever have. Why reinvent the wheel? Do you really think you could come
>> up with something as powerful as Cython, but substantially simpler?
> 
> Cython's pure-python mode requires quite a bit of work to be able to work
> and design I think, although feel free to prove me wrong.

As I said, it lacks a way to use external C declarations. This could be
done by enabling cimports to read these declarations from an external .pxd
file, or, less nicely, because it won't play well with editors: read them
from a string.

I'm not a fan of relying on parsing header files directly, because It Will
Not Work. And it it works, it will Not Do The Right Thing.


> Most importantly,
> it's not only about calls - cython deals with it just fine. I want to be
> able to manipulate the resulting data structures and arguments from pure
> python and do all of that without compiling anything.

I see your point, although I think you will eventually have to compile
something (or generate assembly) in one way or another, at least for C++
support. Yes, I'm well aware that a C/C++ compiler is a major dependency to
carry around.

However, I'm not sure how you want this to work (efficiently?) with
CPython. The reason why it works in LuaJIT is that LuaJIT has a JIT
compiler that can translate it down to native code. PyPy could obviously do
that, too. I'm inclined to think that Jython won't be able to do it all
that well, because it runs inside of a VM that requires at least a JNI call
in between. No idea how IronPython would deal with it.

However, CPython doesn't have a JIT compiler, and that's one of the biggest
drawbacks for (something like) ctypes. There's a reason we compile real
code in Cython, not just a thin glue layer. So, if you want something
that's usable outside of PyPy, you will have to come up with something
that's useful in CPython as well.


> This is part one,
> however part two is that you need some extra step - you need at the very
> least preprocessing that would remove stuff from files that's incorrect
> python, if I understood you correctly. This is quite different from what I
> have in mind where you don't have to have an extra compile/preprocess step
> anywhere.

I think you want some preprocessing, if only for performance reasons. And
while some CPython users may really want to call non-performance critical C
stuff from time to time, you will have a hard time convincing everyone that
they need to port their whole legacy environment to PyPy just to speed up
their calls into external code. I'm sure they'll quite happily stick with
Cython instead.

Stefan