Accessing c objects in python

[Alex Martelli]

Duncan Booth wrote:
        ...
> Thomas Heller <theller at python.net> wrote in news:smocwafb.fsf at python.net:
> 
>> Would you care to write a more detailed summary of your findings? This
>> would probably make a good page in the main Python wiki...
>> 
> 
> I did an evaluation of the various ways to link to Python for the ACCU
> conference this year, although I must say my conclusion was slightly
> different than Alex's: I would use Pyrex in preference to SWIG even if I
> didn't want to embed much code in the interface.

Interesting.  I do agree with your observation that, with SWIG,
responsibility for allocating and freeing tends to show up at the
Python level.  However, SWIG generates such a Python level itself,
as it wraps 'foo' into a low-level _foo.so and a higher-level foo.py.

Thus, the _USER_ of the overall wrapper need not be concerned with
memory allocation and freeing -- unless he WANTS to be, for performance
reasons, in which case, with SWIG, he gets that *option*.  That, in
summary, is the epiphany I had on this latest round of encounter with
SWIG, which made me chose it to complete the wrapping in preference
to Boost and pyrex: the python user of the wrapper could use it in
the simplest and safest of ways _OR_, when performance mattered a
lot (as it might in some cases for these wrappings, which dealt with
multimedia-related libraries), the user could also CHOOSE to take
more direct control -- e.g. explicitly allocate a buffer at the start
of a loop, reuse the buffer N times in the loop, and free it after
the loop, rather than rely on automatic allocation and freeing and
thus pay the resulting performance cost -- in that spot of two in a
huge program where such a strategy pays off.

"Trust the programmer" and "don't stop the programmer from doing 
what needs to be done" are two of the "spirit of C" principles (from
the Rationale of the C standard) which Python respects (more so, in
fact, than the other principle from the same source that is so often
quoted, "provide only one way to perform a task").  With SWIG I get
such an arrangement "almost by default" -- and I think that's quite
different from saying that the python user "has" to deal with the
issues of memory management... it is, rather, a _choice_ he or she
gets, just about "for free" from the POV of the guy who is authoring
(and those who in the future will be maintaining) the wrapper.

I could, of course, get the same choice for the user by coding in
Boost -- but then, since I'm wrapping a C library [and not a C++
one with classes already provided and decently well designed], I'd
have a lot more boilerplate-coding to do myself.  (I could probably
concoct incredibly subtle C++ templates with some macros in support,
to lighten my own boiler-plate coding task -- but I fear that would 
leave far too heavy a maintenance burden on future maintainers of 
those wrappers).

I suspect I could most likely get the same result in pyrex, too -- but
then, again, I'd have to code a lot of boilerplate, _without_ even the 
help of C++ templates, OR even of C-preprocessor macros... that's too 
much raw boilerplate for me;-)  Quite possibly I haven't grasped "the 
pyrex way to metaprogramming" (just as I guess I hadn't grasped the 
SWIG one on earlier encounters with it -- Boost never gave me any 
trouble in this regard, as I already understood C++ templates & the 
like in sufficient depth before I met it).

I'd better underline yet again that this applies to _wrapping C libraries_!
I do NOT claim I have any real grasp yet about the SWIG way wrt C++
(templates in particular) -- when I next need to wrap a C++ library, I'll
reach for Boost and Pyste, without doubts; and if I ever need to provide
Python users with very speedy code, psyco can't supply it (e.g. because I
need to target non-Intel-like CPU's), and Boost is not applicable (e.g.
because I can't require future maintainers to know C++), I'm most 
definitely going to give pyrex another chance -- apart from the boilerplate
issues (==lack of [or insufficient grasp of mine for] metaprogramming and
code generation facilities), I found it just delightful.


Alex