[Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

[Stefan Behnel]

Guido van Rossum, 30.08.2011 19:05:
> On Tue, Aug 30, 2011 at 9:49 AM, "Martin v. Löwis" wrote:
>>>> I can understand how that works when building a CPython extension.
>>>> But what about creating Jython/IronPython modules with Cython?
>>>> At what point get the header files considered there?
>>>
>>> I had written a bit about this here:
>>>
>>> http://thread.gmane.org/gmane.comp.python.devel/126340/focus=126419
>>
>> I see. So there is potential for error there.
>
> To elaborate, with CPython it looks pretty solid, at least for
> functions and constants (does it do structs?).

Sure. They even coerce from Python dicts and accept keyword arguments in 
Cython.


> You must manually
> declare the name and signature of a function, and Pyrex/Cython emits C
> code that includes the header and calls the function with the
> appropriate types. If the signature you declare doesn't match what's
> in the .h file you'll get a compiler error when the C code is
> compiled. If (perhaps on some platforms) the function is really a
> macro, the macro in the .h file will be invoked and the right thing
> will happen. So far so good.

Right.


> The problem lies with the PyPy backend -- there it generates ctypes
> code, which means that the signature you declare to Cython/Pyrex must
> match the *linker* level API, not the C compiler level API. Thus, if
> in a system header a certain function is really a macro that invokes
> another function with a permuted or augmented argument list, you'd
> have to know what that macro does. I also don't see how this would
> work for #defined constants: where does Cython/Pyrex get their value?
> ctypes doesn't have their values.
>
> So, for PyPy, a solution based on Cython/Pyrex has many of the same
> downsides as one based on ctypes where it comes to complying with an
> API defined by a .h file.

Right again. The declarations that Cython uses describe the API at the C or 
C++ level. They do not describe the ABI. So the situation is the same as 
with ctypes, and the same solutions (or work-arounds) apply, such as 
generating additional glue code that calls macros or reads compile time 
constants, for example. That's the approach that the IronPython backend has 
taken. It's a lot more complex, but also a lot more versatile in the long run.

Stefan