SWIG and Callbacks

[David Abrahams]

"Jim" <jbublitzNO at SPAMnwinternet.com> wrote in message
news:3D0E2606.5050001 at SPAMnwinternet.com...
> David Abrahams wrote:

> > Hmm, I wonder about that. The C++ type system is pretty sophisticated
(to
> > the point of being able to represent entire compile-time programs!) and
all
> > of the parser-based wrapping systems I've seen eventually fail to handle
> > some constructs I use, especially if I'm wrapping template-based code.
> > That's part of the reason I chose to use the compiler itself in
> > Boost.Python. The compiler (in theory) has no problem with C++ syntax
;-)
>
> > On the other hand, I realize there's a large category of simple C++
> > interfaces which can be parsed will less-than-complete machinery.
>
> Template-based code does require special treatment with sip.
> In most cases where I've wrapped template-based types or
> classes, I manually write C++ code for conversion C++ <-> Py
> (sip has some functions that make that easier) and implement
> it as a sip 'mapped type', which lets me refer to the same
> conversion code from any return value or arg.

That's a given with Boost.Python; with v2 conversions are automatically
registered in a global repository so they can be re-used automatically
across extension modules.

> I generally
> assign a new name (eg QList<int> -> IntList) but wouldn't
> have to (sip will parse QList<int> and look for a mapped
> type definition corresponding).
>
> sip won't handle all possible C++ syntax, and it sometimes
> takes some creativity to shoehorn something in, but something
> well over 80% of h file code translates directly (after
> removing variable names), and only 1 or 2% really takes
> major effort.

    .../your/ h file code...

> The real problem wouldn't seem to be C++ types as such, but
> finding a useful (to the Python programmer) representation
> via PyObject, and still maintaining the range of abilities
> built into the C++ code (esp things like subclassing).

That's built-in to Boost.Python. V1 had to implement a complicated Python
class look-alike itself; V2 uses Python's new-style classes so subclassing
"just works". Making virtual functions overridable in Python requires a
small amount of boilerplate on the part of the programmer. I often wish I
had a good C++ parser front-end which I could use to automate that part.

> >>The other features that are helpful on large projects are the
> >>built-in support for versioning
>
> > How so? Can you say more about this?
>
> sip uses directives: %If (A - B) ... %End to wrap a block
> of code for versioning over a range. It could also be
> %If (A - ) (>= A) or %If ( - B) (< B). Acceptable values
> for A and B are defined in a %Timeline declaration; actual
> values for A and B are passed to sip via the command line,
> usually  derived from a version #define in the C++ source
> code via configure (at least on Linux). Evaluation is by
> lexical ordering of the A and B values. A and B are
> strings like 'KDE_3_0_0'.
>
> Not very different from #ifdefs, but maybe a little
> easier.

OK. The C++ code you're wrapping still has to get by with #ifdef, I
suppose...

> > Do you mean docstring generation?
>
> No - similar, but sip provides %ExportDoc ... %End directives
> inside which you could place just about anything you want I
> suppose. For PyQt/PyKDE the directives wrap SGML marked-up text.
> You can construct an input file which along with the proper sip
> command line switch, will collect and concatenate all of the
> doc blocks distributed over the description files. The
> concatenated file can be processed (in the case of SGML) to
> provide HTML docs, or whatever you require.

Oh, there are plenty of automated tools for extracting doc from C++. Lots of
people use Doxygen with Boost.Python. No need to invent a new way to do it!

> > Boost.Python works on a similar range of platforms.
> > I'm not sure if this configuration stuff is an appropriate job for a
> > wrapping system to provide specific support for; what's the argument for
> > integrating it with the wrapping system?
>
> PyKDE consists of about 400 description files spread over 10-12
> libraries depending on version and not necessarily with the same
> description file <-> library correspondence in each version. PyQt
> is slightly smaller, but supports more platforms. The alternative
> to embedding version/platform/feature info in the description files
> is to maintain separate sets of files. In the case of PyKDE (depends
> on PyQt) for KDE < 3.0.0, it might build with 2.3.0 <= Qt <= 2.3.2
> depending on what the user has. If the user compiled Qt himself, he
> might have a different feature set than the official distributed Qt
> for his platform. The number of permutations is fairly large and that
> would be multiplied by the number of files.
>
> On the other hand, changes or bug fixes are usually the same across
> versions/platforms/features. So in addition to only supporting one
> fileset instead of many, it's only necessary to repair/edit one
> fileset instead of many. Similarly, it isn't possible to test every
> possible platform - the potential for simple typos crippling a
> particular version alone is pretty large, and most code is common.
>
> Again, #ifdefs probably provide similar functionality - this is
> just a little more formalized/standardized and supported by
> the tools (so error checking is possible, for instance).

I see.

> > Not sure about this; we'd need to compare specific functionality.
>
> My guess is that Boost does at least some of this a little more
> naturally. I'm not sure the extent that Boost can modify things
> like return values or args without resorting to C++

There is no "resorting to C++" with Boost.Python -- Boost.Python *embraces*
C++! Since you're coding the wrappers in C++ anyway, it's trivial to make a
"thin wrapper function" which calls the function being wrapped if you want
to change the interface somehow. Then you just wrap the thin wrapper.

> >>I suppose those are a series of small advantages, but the end
> >>result seems to me that you write less code
>
> > Are you counting the IDL file? That's code, IMO.
>
> Yes, but I don't write it - I just edit it mostly to correct
> mistakes my description file generator creates and to handwrite
> code where the generator has flagged that. Crude but effective (TM)
> Even without automation, the process is easiest if you start by
> saving the h file with a .sip extension and then (mostly) delete
> the things that sip doesn't want/need, so it's still mostly
> editing (sometimes A LOT of editing).

That seems to be the key difference. It's not clear wether it's really an
advantage, but I suppose that time will tell.

> >>Boost (IIRC) also has a big advantage in being able to bind much
> >>more selectively.
>
> > Can you explain what you mean by that?
>
> I should probably go back and review Boost. I had thought that
> with Boost it was possible to wrap as much or as little of a
> C++ library as you want to.

That's true.

> > Heh, yeah. Unfortunately, documentation is often harder than coding.
Even
> > the current official release of Boost.Python is not very
well-documented.
> > We're working hard to correct that for v2, which is in pre-release.
>
> Well, you have docs ... I found the web site to be pretty good at
> explaining how to use Boost (and perhaps haven't remembered it all
> correctly).

Yeah, it's a reasonable tutorial, but there are still a lot of undocumented
features in v1.

> >>Could you automate code generation to some extent?
>
> > Not sure what you have in mind here. In a sense, the metaprogramming
> > techniques used by Boost.Python already *do* code generation.
>
> Yeah - 'code generation' is ambiguous. sip itself is a code generator
> (it produces C++/Python files which are the actual bindings or Python
> lib). What I meant in this case is generating the description files
> directly from the h files. I currently take a set of h files, run them
> through a handwritten parser and generate *.sip files which sip then
> uses to generate the actual C++ code for the bindings.

Oh, sure, these guys are doing the same thing with Boost.Python:

http://mambo.peabody.jhu.edu/~karlmac/publications/gamera_python_2001/gamera
.html

> I had the
> impression SWIG was supposed to do most of that in a single step. Just
> wondering if I could implement something like that with Boost, or
> need to code a line manually for each method/class/whatever.

If you have something which can mostly parse your headers, you can generate
much of the wrapping code automatically.

> It's pretty amazing to find out who's using Python and things
> like Boost, pyGTK. PyQt, PyKDE, etc.

Yep. BTW, Boost is much more than just my Python binding library, so please,
for the sake of the other authors: "Boost.Python" when discussing my lib.

> I left your web site with the impression that I (personally) would
> find sip easier for large projects, but Boost would have advantages
> on smaller projects (unfortunately I remember the impression better
> than the reasons for it). That's especially interesting to me at
> the moment, because I have ideas for a lot of smaller projects. I expect
> I'll stick with sip, but will have to take another look at Boost, if
> only to steal good ideas :)

Not sure that you'll find much to steal; they're so different in approach...

Regards,
Dave