[Python-Dev] Cython, ctypes and the stdlib

Stefan Behnel stefan_ml at behnel.de
Mon Aug 29 16:14:53 CEST 2011


Hi,

I agree that this is getting off-topic for this list. I'm answering here in 
a certain detail to lighten things up a bit regarding thin and thick 
wrappers, but please move further usage related questions to the 
cython-users mailing list.

Paul Moore, 29.08.2011 12:37:
> On 29 August 2011 10:39, Stefan Behnel wrote:
>> In the CPython backend, the header files are normally #included by the
>> generated C code, so they are used at C compilation time.
>>
>> Cython has its own view on the header files in separate declaration files
>> (.pxd). Basically looks like this:
>>
>>     # file "mymath.pxd"
>>     cdef extern from "aheader.h":
>>         double PI
>>         double E
>>         double abs(double x)
>>
>> These declaration files usually only contain the parts of a header file that
>> are used in the user code, either manually copied over or extracted by
>> scripts (that's what I was referring to in my reply to Terry). The complete
>> 'real' content of the header file is then used by the C compiler at C
>> compilation time.
>>
>> The user code employs a "cimport" statement to import the declarations at
>> Cython compilation time, e.g.
>>
>>     # file "mymodule.pyx"
>>     cimport mymath
>>     print mymath.PI + mymath.E
>>
>> would result in C code that #includes "aheader.h", adds the C constants "PI"
>> and "E", converts the result to a Python float object and prints it out
>> using the normal CPython machinery.
>
> One thing that would make it easier for me to understand the role of
> Cython in this context would be to see a simple example of the type of
> "thin wrapper" we're talking about here. The above code is nearly
> this, but the pyx file executes "real code".

Yes, that's the idea. If all you want is an exact, thin wrapper, you are 
better off with SWIG (well, assuming that performance is not important for 
you - Cython is a *lot* faster). But if you use it, or any other plain glue 
code generator, chances are that you will quickly learn that you do not 
actually want a thin wrapper. Instead, you want something that makes the 
external library easily and efficiently usable from Python code. Which 
means that the wrapper will be thin in some places and thick in others, 
sometimes very thick in selected places, and usually growing thicker over time.

You can do this by using a glue code generator and writing the rest in a 
Python wrapper on top of the thin glue code. It's just that Cython makes 
such a wrapper much more efficient (for CPython), be it in terms of CPU 
performance (fast Python interaction, overhead-free C interaction, native C 
data type support, various Python code optimisations), or in terms of 
parallelisation support (explicit GIL-free threading and OpenMP), or just 
general programmer efficiency, e.g. regarding automatic data conversion or 
ease and safety of manual C memory management.


> For example, how do I simply expose pi and abs from math.h? Based on
> the above, I tried a pyx file containing just the code
>
>      cdef extern from "math.h":
>          double pi
>          double abs(double x)
>
> but the resulting module exported no symbols.

Recent Cython versions have support for directly exporting C values (e.g. 
enum values) at the Python module level. However, the normal way is to 
explicitly implement the module API as you guessed, i.e.

     cimport mydecls   # assuming there is a mydecls.pxd

     PI = mydecls.PI
     def abs(x):
         return mydecls.abs(x)

Looks simple, right? Nothing interesting here, until you start putting 
actual code into it, as in this (totally contrived and untested, but much 
more correct) example:

     from libc cimport math

     cdef extern from *:
         # these are defined by the always included Python.h:
         long LONG_MAX, LONG_MIN

     def abs(x):
         if isinstance(x, float):    # -> C double
             return math.fabs(x)
         elif isinstance(x, int):    # -> may or may not be a C integer
             if LONG_MIN <= x <= LONG_MAX:
                 return <unsigned long> math.labs(x)
             else:
                 # either within "long long" or raise OverflowError
                 return <unsigned long long> math.llabs(x)
         else:
             # assume it can at least coerce to a C long,
             # or raise ValueError or OverflowError or whatever
             return <unsigned long> math.labs(x)

BTW, there is some simple templating/generics-like type merging support 
upcoming in a GSoC to simplify this kind of type specific code.


> This is probably a bit off-topic, but it seems to me that whenever
> Cython comes up in these discussions, the implications of
> Cython-as-an-implementation-of-python obscure the idea of simply using
> Cython as a means of writing thin library wrappers.

Cython is not a glue code generator, it's a full-fledged programming 
language. It's Python, with additional support for C data types. That makes 
it great for writing non-trivial wrappers between Python and C. It's not so 
great for the trivial cases, but luckily, those are rare. ;)


> I've kept python-dev in this response, on the assumption that others
> on the list might be glad of seeing a concrete example of using Cython
> to build wrapper code. But anything further should probably be taken
> off-list...

Agreed. The best place for asking about Cython usage is the cython-users 
mailing list.


> PS This would also probably be a useful addition to the Cython wiki
> and/or the manual. I searched both and found very little other than a
> page on wrapping C++ classes (which is not very helpful for simple C
> global functions and constants).

Hmm, ok, I guess that's because it's too simple (you actually guessed how 
it works) and a somewhat rare use case. In most cases, wrappers tend to use 
extension types, as presented here:

http://docs.cython.org/src/tutorial/clibraries.html

Stefan



More information about the Python-Dev mailing list