[Python-Dev] SWIG (was Re: Ctypes and the stdlib)

[David Cournapeau]

On Mon, Aug 29, 2011 at 7:14 PM, Eli Bendersky <eliben at gmail.com> wrote:
> <snip>
>>
>> I've sometimes thought it might be interesting to create a Swig
>> replacement purely in Python.  When I work on the PLY project, this is often
>> what I think about.   In that project, I've actually built a number of the
>> parsing tools that would be useful in creating such a thing.   The only
>> catch is that when I start thinking along these lines, I usually reach a
>> point where I say "nah, I'll just write the whole application in Python."
>>
>> Anyways, this is probably way more than anyone wants to know about Swig.
>> Getting back to the original topic of using it to make standard library
>> modules, I just don't know.   I think you probably could have some success
>> with an automatic code generator of some kind.  I'm just not sure it should
>> take the Swig approach of parsing C++ headers.  I think you could do better.
>>
>
> Dave,
>
> Having written a full C99 parser (http://code.google.com/p/pycparser/) based
> on your (excellent) PLY library, my impression is that the problem is with
> the problem, not with the solution. Strange sentence, I know :-) What I mean
> is that parsing C++ (even its headers) is inherently hard, which is why the
> solutions tend to grow so complex. Even with the modest C99, clean and
> simple solutions based on theoretical approaches (like PLY with its
> generated LALR parsers) tend to run into walls [*]. C++ is an order of
> magnitude harder.
>
> If I went to implement something like SWIG today, I would almost surely base
> my implementation on Clang (http://clang.llvm.org/). They have a full C++
> parser (carefully hand-crafted, quite admirably keeping a relatively
> comprehensible code-base for such a task) used in a real compiler front-end,
> and a flexible library structure aimed at creating tools. There are also
> Python bindings that would allow to do most of the interesting
> Python-interface-specific work in Python - parse the C++ headers using
> Clang's existing parser into ASTs - then generate ctypes / extensions from
> that, *in Python*.
>
> The community is also gladly accepting contributions. I've had some fixes
> committed for the Python bindings and the C interfaces that tie them to
> Clang, and got the impression from Clang's core devs that further
> contributions will be most welcome. So whatever is missing from the Python
> bindings can be easily added.

Agreed, I know some people have looked into that direction in the
scientific python community (to generate .pxd for cython). I wrote one
of the hack Stefan refered to (based on ctypeslib using gccxml), and
using clang makes so much more sense.

To go back to the initial issue, using cython to wrap C code makes a
lot of sense. In the scipy community, I believe there is a broad
agreement that most of code which would requires C/C++ should be done
in cython instead (numpy and scipy already do so a bit). I personally
cannot see man situations where writing wrappers in C by hand works
better than cython (especially since cython handles python2/3
automatically for you).

cheers,

David