[pypy-dev] Python FFI

[mark florisson]

On 15 May 2012 17:14, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Maciej Fijalkowski, 15.05.2012 17:39:
>> On Tue, May 15, 2012 at 5:16 PM, Stefan Behnel wrote:
>>> Armin Rigo, 15.05.2012 16:23:
>>>> Fijal and me would like to raise interest among various groups of
>>>> people about building a better ctypes replacement for Python.
>>>> [...]
>>>> The working alternatives nowadays are Cython and ctypes.  For Cython
>>>> you have to learn a whole new language; we can argue infinitely about
>>>> preferences, but from my point of view it's mostly just a nicer way to
>>>> write CPython C extensions using --- instead of C --- a custom
>>>> language.
>>>
>>> Well, it's basically Python with optional static types, which are mostly
>>> provided in C(-like) notation. I find it surprisingly similar to LuaJIT's
>>> FFI, since you mentioned that anyway.
>>>
>>>
>>>> Also, because it would parse strings, we can also add support for
>>>> parsing strings in the format of Cython declarations.  This would
>>>> still lower the entry barrier, and have the nice effect that the user
>>>> can sometimes copy-and-paste Cython declarations, instead of
>>>> copy-and-pasting carefully chosen pieces of a potentially messy .h
>>>> file full of custom macros (see for example /usr/include/zlib.h).  The
>>>> benefit would be that, unlike Cython, he gets to call the declared
>>>> functions directly from his regular Python code.
>>>
>>> I assume you are aware of Cython's pure Python mode? It doesn't currently
>>> support calling into C, but that's a matter of doing it, not so much of
>>> researching how it could be done.
>>>
>>> Personally, I've always been thinking of Cython as the best FFI that Python
>>> could ever have. Why reinvent the wheel? Do you really think you could come
>>> up with something as powerful as Cython, but substantially simpler?
>>
>> Cython's pure-python mode requires quite a bit of work to be able to work
>> and design I think, although feel free to prove me wrong.
>
> As I said, it lacks a way to use external C declarations. This could be
> done by enabling cimports to read these declarations from an external .pxd
> file, or, less nicely, because it won't play well with editors: read them
> from a string.
>
> I'm not a fan of relying on parsing header files directly, because It Will
> Not Work. And it it works, it will Not Do The Right Thing.
>

I think a parser may not always be able to do the right thing in case
of macros etc, but I think the compiled wrapper approach will work
quite well, especially since the user writes the code explicitly.

>> Most importantly,
>> it's not only about calls - cython deals with it just fine. I want to be
>> able to manipulate the resulting data structures and arguments from pure
>> python and do all of that without compiling anything.
>
> I see your point, although I think you will eventually have to compile
> something (or generate assembly) in one way or another, at least for C++
> support. Yes, I'm well aware that a C/C++ compiler is a major dependency to
> carry around.
>
> However, I'm not sure how you want this to work (efficiently?) with
> CPython. The reason why it works in LuaJIT is that LuaJIT has a JIT
> compiler that can translate it down to native code. PyPy could obviously do
> that, too. I'm inclined to think that Jython won't be able to do it all
> that well, because it runs inside of a VM that requires at least a JNI call
> in between. No idea how IronPython would deal with it.
>
> However, CPython doesn't have a JIT compiler, and that's one of the biggest
> drawbacks for (something like) ctypes. There's a reason we compile real
> code in Cython, not just a thin glue layer. So, if you want something
> that's usable outside of PyPy, you will have to come up with something
> that's useful in CPython as well.
>
>
>> This is part one,
>> however part two is that you need some extra step - you need at the very
>> least preprocessing that would remove stuff from files that's incorrect
>> python, if I understood you correctly. This is quite different from what I
>> have in mind where you don't have to have an extra compile/preprocess step
>> anywhere.
>
> I think you want some preprocessing, if only for performance reasons. And
> while some CPython users may really want to call non-performance critical C
> stuff from time to time, you will have a hard time convincing everyone that
> they need to port their whole legacy environment to PyPy just to speed up
> their calls into external code. I'm sure they'll quite happily stick with
> Cython instead.

I don't think that's the point though, nor the problem they are trying
to solve. CTypes is a pretty tough interface to deal with these
problems, and any better interface can benefit many Python users,
including the people currently using ctypes, and including people that
have a simple api they need to call or just prefer portability over
using Cython. It would also perhaps allow better sharing of certain
stdlib modules between Python implementations (?).

> Stefan
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev