ctype performance benchmark
nick at craig-wood.com
Fri Jul 17 14:29:59 EDT 2009
Wai Yip <aurora00 at gmail.com> wrote:
> I started with ctypes because it is the battery included with the
> Python standard library. My code is very fluid and I'm looking for
> easy opportunity to optimize it. One example is to find the longest
> common prefix among two strings. Right now I am comparing it character
> by character with pure Python. It seems like an ideal low hanging
> fruit. With ctype I can rewriting a few lines of Python into a few
> lines of C. All the tools are available and no third party library is
Yes ctypes is very easy for this and I've used it like this in the
past too. It makes interfacing C code very easy.
You might consider just writing a plain bit of C code. If you make a
setup.py too (not hard) then it is very easy to distribute and use on
The Python C API is very easy to use - documentation is extensive.
You need to be very careful with references and the error checking is
boring and tedious!
> It turned out the performance fail to match pure Python. This function
> is called million of times in an inner loop. The overhead overwhelm
> any performance gain with C. Eventually I have found success grouping
> the data into larger chunk for each API call. This is what I
> originally planned to do anyway. I am only sharing my experience here
> that doing fine grained ctype function call has its limitation.
Interesting but not too unexpected - that translation of types is
> I have looked into Pyrex before and I was quite confused by the fusion
> of C and Python langauage. Perhaps it is time for me to give it a
> second look. I just heard of Cython now and it also look interesting.
> I think it is helpful for both project to articulate more clearly what
> they are, how they achieve the performance gain and to give more
> guidance on good use cases. On the other hand, perhaps it is just me
> are confused because I don't know enough of the Python internal.
Pyrex evolved/was forked into Cython.
Cython allows you to write a subset of python code that is compiled
into C. You can also optionally provide some type annotations to get
it to produce better C code at the cost of making your code look less
I find Cython alternatively brilliant and frustrating! Sometimes it
works really well, and sometimes I spend hours working out why my
program doesn't compile, or to find out the exact syntax I need to
interface with such and such a library.
I did a C library interfacing project with both ctypes and cython
recently as a comparison. It came out to be about the same number of
lines of code for both. I decided to run with the ctypes version as
it was part python.
Here is a short Cython example which interfaces with opendir /
closedir / readdir which python doesn't do natively. Note the way you
interface with C structures (the real ones are used in the C code -
the annotations just tell cython how to use them). Note also that
Directory is a class defined in C which I don't think you can't do
with ctypes without a wrapper class.
This compiles into 1320 lines of C, which in turn compile into 11380
bytes of shared object (when stripped).
cdef extern from "dirent.h":
ctypedef struct DIR
DIR *opendir(char *name)
int closedir(DIR *dirp)
dirent *readdir(DIR *dirp)
cdef extern from "errno.h":
cdef extern from "string.h":
char *strerror(int errnum)
cdef class Directory:
"""Represents an open directory"""
cdef DIR *handle
def __init__(self, path):
self.handle = opendir(path)
if self.handle is NULL:
raise OSError(errno, "Failed to open directory: %s" % strerror(errno))
"""Read the next name in the directory"""
cdef dirent *p
p = readdir(self.handle)
if p is NULL:
"""Close the directory"""
if self.handle is not NULL:
self.handle = NULL
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick
More information about the Python-list