[pypy-dev] Python FFI

[David Naylor]

On Tuesday, 15 May 2012 16:23:14 Armin Rigo wrote:
> Hi all,

Hi,
 
> Fijal and me would like to raise interest among various groups of
> people about building a better ctypes replacement for Python.

I have a suggestion, and it (possibly) involves using ctypes...

> Opinions?  Interests?  This mail is deliberately low on details about
> how we think we can do it.  Instead I'm seeking general reactions for
> now and would like to move this soon to its own project, independent
> of PyPy.

First, I think when one interfaces with a C(++) library a limited 
understanding of C(++) is required, at least the minimum of data types, 
structures and (to an extent) memory management (think pointers and who owns 
which structures).  

Using that understanding one could then use tool X to create a pythonic 
wrapper around the underlying C CPI and expose a "pure" python API to the rest 
of the world.  

For me, the ideal would be (using a trivial [bad] example):
<module name="sysctl.py>
import ctypes
# See sysctl(3) for details (from FreeBSD)
# Note 1
from c.sys import sysctl as _sysctl  #include <sys/sysctl.h>

__all__ = ["get", "set"]

def get(name):
	# Assume all values are c_int (for now)
	cint = ctypes.c_int()
	# Note 2
	_sysctl.sysctl(name, len(name), cint, ctypes.sizeof(cint), None, 0)
	return cint.value

def set(name, value):
	# Assume all values are c_int (for now)
	# Note 3
	_sysctl.sysctl(name, len(name), None, 0, value, 
ctypes.sizeof(ctypes.c_int))

</module>

Notes:
 1) The module "c" reads the C header file <sys/sysctl.h> and generates (or 
loads a cached result, possible bundled with the program) glue that provides a 
low level interface to the C API described in that header file.  
 2) See (*) for the description of _sysctl.sysctl.  The wrapper for sysctl(3) 
knows that the third parameter is needed by reference (and that we want the 
stored value), the 5th parameter None translates to NULL.  
 3) See (*), the wrapper knows the 5th parameter is needed by reference (and 
that we don't want the result).

I think the trick here is automating the tedious description of the C API in 
ctypes (note, the above doesn't need to be done in ctypes, but can be thought 
of a pythonic interface to low-level C).  AFAIK LLVM provides python binding 
for the language and (with clang) could possibly provide all the information 
required to parse the C header files, then python can be used to generate the 
interface provided by LLVM (this, to my understanding, is starting to sound 
like rpython...).  

I think it is quite possible that the interface modules for the C API could be 
pre-generated (in rpython, cython, etc and compiled) and distributed with the 
python codes (similarly to how C/Python mixed code works).  

Which, all the above, is very similar to the Lua FFI library, just on a 
different scale.  

Regards,

David

P.S. Would it not be possible for the pypy jit to inline c macros, given the 
right information ;-)

(*) int sysctl(const int *name, u_int namelen, void *oldp, size_t *oldlenp, 
const void *newp, size_t newlen);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20120516/f6fbb6bf/attachment.pgp>