[Python-Dev] reference counting in Py3K

[Josiah Carlson]

Fernando Perez <fperez.net at gmail.com> wrote:
> Josiah Carlson wrote:
> > Here's a perspective "from the trenches" as it were.
> > 
> > I've been writing quite a bit of code, initially all in Python (27k
> > lines in the last year or so).  It worked reasonably well and fast. It
> > wasn't fast enough. I needed a 25x increase in performance, which would
> > have been easily attainable if I were to rewrite everything in C, but
> > writing a module in pure C is a bit of a pain (as others can attest), so
> > I gave Pyrex a shot (after scipy.weave.inline, ick).
> 
> Would you care to elaborate on the reasons behind the 'ick'?  I'm a big fan of
> weave.inline and have used it very successfully for my own needs, so I'm
> genuinely curious (as I tend to teach its use, I like to know of potential
> problems I may not have seen).  

1. Mixing multiple languages in a single source file is bad form, yet it
seems to be encouraged in weave.inline and other such packages (it
becomes a big deal when the handful of Python becomes 20+ lines of C).

2. I experienced some minor but annoying issues in regards to automatic
type conversions (strings, mmaps, buffers, and arrays if I remember
correctly, it has been since February or March).

There were other things, but I'm not sure if I am remembering them
correctly or not (I spent around 12 hours over two days wrestling with
weave.inline, but in 10 minutes I was using Pyrex effectively).

> I should also add that a while ago a number of extremely annoying spurious
> recompilation bugs were finally fixed, in case this was what bothered you. 
> Those bugs (hard to find) made weave in certain cases useless, as it
> recompiled everything blindly, thus killing its whole purpose.

I was actually finding that weave wasn't recompiling /enough/.  I'd
change some source, and get old behavior.  I'd delete the various cache
files, then see the recompilation and new behavior.  With Pyrex and a
bit of magic, I get auto-recompilation (though will seriously consider
switching to Pyximport as another suggested).  It also seemed to have
some issues with interactive sessions, but I may be misremembering.


> Feel free to reply off-list if you feel this is not appropriate for python-dev,
> though I think that a survey of the c-python bridges may be of interest to
> others.

Agreed.  I admit that some of my issues would likely be lesser if I were
to start to use inline now, with additional experience with such things.
But with a few thousand lines of Pyrex and C working right now, I'm hard
pressed to convince anyone (including myself) that such a switch is
worthwhile.


 - Josiah