[Python-Dev] reference counting in Py3K

[Phillip J. Eby]

At 09:58 PM 9/6/2005 -0700, Guido van Rossum wrote:
>On 9/6/05, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > A better plan would be to build something akin to
> > Pyrex into the scheme of things, so that all the
> > refcount/GC issues are taken care of automatically.
>
>That sounds exciting. I have to admit that despite hearing many
>enthusiastic reviews, I've never used it myself -- in fact I've
>written very little C code in the last few years, and zero new
>extension modules. (Lots of Java, but that's another story. :-)
>
>I expect that many standard extensions could benefit from a rewrite in
>Pyrex, although this might take a lot of work and in some cases not
>necessarily result in better code (_tkinter comes to mind -- though I
>don't really know why this would be). So this shouldn't be the goal
>(yet). Instead, we should encourage folks to write *new* extensions
>using PyRex.

Just an FYI; Pyrex certainly makes it relatively painless to write code 
that interfaces with C, but it doesn't do much for performance, and 
naively-written Pyrex code can actually be slower than carefully-optimized 
Python code.  So, for existing modules that were written in C for 
performance reasons, Pyrex isn't currently a substitute.

One of the reasons for this is that Pyrex code uses the generic Python/C 
APIs, like PySequence_GetItem, even in cases where PyList_GetItem or its 
macro form would be more appropriate.  Pyrex has no way currently to say, 
"this is type X's C API, so use it when you have a variable that's of type 
X, instead of using the generic object protocols."

There are other issues that contribute to the inefficiency as well, like 
redundant refcounting, assigning None to temporary variables, etc.  I 
haven't used the absolute latest version of Pyrex, but older versions also 
used C strings for attribute lookups, which was horribly slow.  I think the 
latest version now creates string objects at module initialization to avoid 
this issue, though.

Anyway, don't get me wrong - Pyrex is by *far* my preferred way of writing 
C extensions for Python.  And I'd love to see a version that used your 
proposed static typing syntax in place of the "cdef" syntax it currently 
uses, thus paving the way to selective translation of Python to C.  I'd 
just like to set expectations appropriately, for what it is and isn't good at.

Sadly, the current state of Pyrex is such that to write efficient code, you 
have to use the Python C API from Pyrex, which tends to result in something 
that looks like C code with Python-like syntax.  But you don't have to 
worry about memory allocation, exception labels, or reference counting, so 
it's more compact and less error-prone than C too.