[C++-sig] Weave Scipy Inline C++

[eric jones]

> --- David Abrahams <dave at boost-consulting.com> wrote:
> > Boost doesn't provide any facilities for on-demand compilation of
C++
> > code from Python, if that's what you're getting at.
> 
> Boost.Python is a high-throughput factory for Python "built-in" types.
> Instead of using the few native types (list, tuple, dict) you can
easily 
> make types that are tailored for specific needs.

I think there is a key difference that needs to be understood here.  You
may understand this already, but it will help in the overall discussion.


Weave's current incarnation is roughly divided into four parts:

0. Fancy translation of Python->C/C++ code in blitz() or accelerate()
1. Extension function and module code generation for C/C++ snippets
2. On demand compilation machinery.
3. A "Catalog" that keeps track of extension functions and handles
dynamic typing issues.

(0) is not used by inline() which is the most heavily used part of weave
currently.  (0) is the really hard part and is getting attention from
Pat Miller at LLNL.  Also, I'd like to look at rolling some of Pyrex in
for this sort of thing.

The semi-overlapping portions of weave and boost are in (1), but they
definitely have different purposes.  Boost wraps existing C++ code and
exposes the classes as Python types.  Weave allows you to inline C/C++
in Python with the variables in Python automatically converted to
meaningful C++ types (when possible).  Based on this the type mapping
process for weave is to map Python types/classes into C++ types/classes.
This is the opposite of what boost or SWIG which map C/C++ code types to
Python types.  Most of the machinery in boost is explicitly for this
purpose.  Weave *might* be augmented to handling the task of wrapping an
existing C++ class to be exposed as a Python type/class, but it would be
some serious work (as yall know) and the task is already well handled by
boost and SWIG.  So isn't the niche it is trying to fill.

So, to be more explicit.  You would never wrap wxWindows with weave.
However, since weave is wxPython aware (on windows anyway), you can
inline  snippets of C/C++ code that manipulate wxWindows objects passed
from Python into your C/C++ snippet.  For example, we needed a faster
method of drawing Numeric arrays to the screen on windows.  If you don't
want to go in and rebuild all of wxPython, so weave is an option.  The
following does it at about 10 times the speed of the standard wxPython
method:

def polyline(dc,line):
    shp = line.shape
    assert(len(shp)==2 and shp[1] == 2)
    if (line.typecode() != Int or not line.iscontiguous()):
        line = line.astype(Int)
    
    polyline_code = """
                    HDC hdc = (HDC) dc->GetHDC();                    
                    Polyline(hdc,(POINT*)line,Nline[0]);
                    """
    # call the weave version of the function.
    weave.inline(polyline_code,['dc','line'])

This sort of thing and, writing computational kernels, is where weave
excels.  The user doesn't have to know much about Python's API to get
this to work.

> I am using Boost.Python in combination with SCons, which is much more
> mature than the version number suggests. SCons is suitable as a 
> "compilation on demand" tool.

This is part (2) of what weave does.  We currently use distutils, and it
works fine.  Weave actually does very little work here... Well, it does
some work with managing what extra files etc. need to be compiled/linked
to support special types such as wxPython, but distutils does the actual
compiling.   Hopefully distutils will get
re-implemented/refactored/reworked with SCons as the underpinnings at
some point.  

> I am wondering how much of a run-time difference it can make if some
> trivial
> pieces of compiled code working on objects through the (slow b/o
dynamic
> typing) Python API are embedded in Python. 

Not much for stuff that has to go through the API -- maybe a factor of
2-4 if you're really lucky.  Where you win is when you know the C/C++
format of a Python object.  Numeric arrays are a key example where one
or two orders of magnitude speed up are common.

> If speed is a premium you can
> gain
> much more if the C++ code operates on native C++ types, without the
> overhead of
> the Python API for accessing data items. This is precisely what
> Boost.Python
> gives you. Typically, I cross the language boundary only once to do an
> "expensive", non trivial operation in pure C++, and the result is a
> wrapped C++
> object that can conveniently be inspected from Python.

The same is true in weave if it is aware of your type.  Here is how you
might use it.  Say create a class called Particle in C++.  Further, you
have a C++ container called Vector<Particle> or something like that.
Boost is used to wrap this and expose it Python.  You make the extension
module and distribute it to your users.  Now, a user comes along and
needs to calculate a weighted centroid of a group of particles, but you
didn't define that function.  Their options are to learn how to
recompile your module (perhaps daunting), or, if you send along weave
aware type converters (simple to make), the could do something like the
following:

def weighted_centroid(particles):
    centroid = zeros(3,Float32) # Numeric array
    code = """
           float sum = 0;
           for(int i=0; i < particles.length(); i++)
           {
               Particle p = particles[i];
               float w = p.weight;
               sum += w; 
               centroid[0] = w * p.x;
               centroid[1] = w * p.y;
               centroid[2] = w * p.z;
           }
           centroid[0] /= sum; centroid[1] /= sum; centroid[2] /= sum;
           """
    weave.inline(code,['particles','centroid'])
    return centroid

<hope I got the code right, but you get the idea>

Now the end user can extend your boost library without needing to have
or know how to use boost and without fear of breaking the library .
After prototyping the function, they might send it back to you for
incorporation in the main library and they can get rid of weave snippet,
but while they are waiting they have something that works and runs at
the speed of C -- the same speed as anything else in your library.  

Hopefully this shows that boost and weave serve different but related
purposes and can work together.
     
> 
> Does it make sense to put Weave and Boost.Python together? Let me try
an
> analogy: if you choose Weave you are choosing a bicycle, and if you
choose
> Boost.Python you are choosing a car. Obviously the world needs both
> bicycles
> and cars, but would you use a car as a building block for a bicycle?

<David alludes to potential offense caused by analogy>

No offense taken.

I asked a similar question on this list a year or so ago when weave was
released.  This is a much livelier discussion than that one was.

As I said earlier, there are some areas where the two tools overlap, but
there are many regions where they do not.  I'm not sure merging them (at
least at this point) buys a whole lot.  The code re-use would be
minimal.

On the other hand, the synergy might be really nice.  I think boost is
probably about 80-90% of the way to its goal (is this correct?).  Weave
also does (1)-(3) fairly well in the list above.  However, there is a
*huge* amount of work that can happen in (0), and its potential is
exciting much like Pyrex and psyco are exciting.  (0) is more compiler
stuff.  I'm not sure how boost fits, but it may -- I haven't spent
enough time with boost yet to know.

I guess the other thing that would be really nice is a boost_converter
base class in weave that others who want to expose there boost wrapped
classes to weave can sub-class.  I already have this for SWIG (at least
the old SWIG).  Again, doing so is likely to be trivial.

Regards,
eric