[Python-Dev] reference counting in Py3K
Phillip J. Eby
pje at telecommunity.com
Wed Sep 7 08:01:01 CEST 2005
At 09:58 PM 9/6/2005 -0700, Guido van Rossum wrote:
>On 9/6/05, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > A better plan would be to build something akin to
> > Pyrex into the scheme of things, so that all the
> > refcount/GC issues are taken care of automatically.
>That sounds exciting. I have to admit that despite hearing many
>enthusiastic reviews, I've never used it myself -- in fact I've
>written very little C code in the last few years, and zero new
>extension modules. (Lots of Java, but that's another story. :-)
>I expect that many standard extensions could benefit from a rewrite in
>Pyrex, although this might take a lot of work and in some cases not
>necessarily result in better code (_tkinter comes to mind -- though I
>don't really know why this would be). So this shouldn't be the goal
>(yet). Instead, we should encourage folks to write *new* extensions
Just an FYI; Pyrex certainly makes it relatively painless to write code
that interfaces with C, but it doesn't do much for performance, and
naively-written Pyrex code can actually be slower than carefully-optimized
Python code. So, for existing modules that were written in C for
performance reasons, Pyrex isn't currently a substitute.
One of the reasons for this is that Pyrex code uses the generic Python/C
APIs, like PySequence_GetItem, even in cases where PyList_GetItem or its
macro form would be more appropriate. Pyrex has no way currently to say,
"this is type X's C API, so use it when you have a variable that's of type
X, instead of using the generic object protocols."
There are other issues that contribute to the inefficiency as well, like
redundant refcounting, assigning None to temporary variables, etc. I
haven't used the absolute latest version of Pyrex, but older versions also
used C strings for attribute lookups, which was horribly slow. I think the
latest version now creates string objects at module initialization to avoid
this issue, though.
Anyway, don't get me wrong - Pyrex is by *far* my preferred way of writing
C extensions for Python. And I'd love to see a version that used your
proposed static typing syntax in place of the "cdef" syntax it currently
uses, thus paving the way to selective translation of Python to C. I'd
just like to set expectations appropriately, for what it is and isn't good at.
Sadly, the current state of Pyrex is such that to write efficient code, you
have to use the Python C API from Pyrex, which tends to result in something
that looks like C code with Python-like syntax. But you don't have to
worry about memory allocation, exception labels, or reference counting, so
it's more compact and less error-prone than C too.
More information about the Python-Dev