[C++-sig] weave plus boost?

[eric]

Hey Martin,

Good to hear from you!  Hope all is well.  Any chance I'll see you at the
Python conference?

> It is impossible to know which instances of vector are going to be needed
> before runtime with this method, so one option is to have vector(..)
generate
> an
> instance of vector<string> on the fly, compile it and load it in. Each
> instance extension
> would proabaly be cached, keyed by the type so subsequent requests for
> vector("std::string") wouldn't have to redo the process.
>
> I'm not sure how practical this would be, but at one time I had a sloppy
but
> working prototype and it was.. if nothing else.. neat.

You've pretty much described how weave.inline works.  Because python is
dynamic and C is statically typed, weave  handles all argument types
this way.  The case of templates fits seamlessly and nicely into this
scheme.

To describe how it works, lets first look a simple code fragment and see
how weave handles it.

    >>> import weave
    >>> code = "std::cout << a << std::endl;"
    >>> a = 1
    >>> weave.inline(code, ['a'])
    <there is a delay for the compile>
   1
    >>> a = 'string'
    >>> weave.inline(code, ['a'])
    < another delay for the compile>
    string
    >>> weave.inline(code, ['a'])
   <no delay>
   1

In the example, the first time weave is called, it has never seen the C++
code fragment contained in 'code'.  As a result, it generates a wrapper
function, fires up the compiler, and builds the extension function,
<ext_mod1.compiled_func>.  weave enters this function in a "catalog"
of all the functions its compiled.  The catalog is a dictionary that relates
a given C++ code fragment with a *list* of extension functions.

    catalog[code] = [<ext_mod1.compiled_func>]

The second time weave.inline is called, it is called with the exact same
code fragment. weave looks in its catalog and sees that it has compiled
functions for this code fragment before.  weave then starts at the front of
the list of functions and starts calling them one by one.  The compiled
functions have type checks in them that throw a ConversionError
exception (well techincally a TypeError) when the incoming argument
types don't match the argument types the function was compiled for.  If you
get to the end of the list of functions and they have all thrown
ConversionError, weave will compile a new function for the new
argument types.  In the 2nd case above where 'code' is called with a
string, , 'code' would get recompiled for strings to a second function
<ext_mod2.compiled_func> and added to the front of the catalog.

    catalog[code] = [<ext_mod2.compiled_func>,<ext_mod1.compiled_func>]

The same thing happens for template arguments.  one of the
"type conversion factories" used to generate the code for converting Python
Numeric arrays to C++ data types uses blitz++ arrays on the C++ side.
blitz++
arrays have two template parameters, their data type and their size.

    template<type,N>
    class array
    ...

There is an example in the weave/examples directory called
cast_copy_transpose.py that is useful in handling some of the issues that
come
up in using Python with Fortran code.  We'll show a modified version of this
example to illustrate how this can work:

def copy_transpose(a_2d):
    assert(len(shape(a_2d)) == 2)
    new_array = zeros(shape(a_2d),a.typecode())
    code = """
           for(int i = 0; i < _Na_2d[0]; i++)
               for(int j = 0; j < _Na_2d[1]; j++)
                   new_array(i,j) = a_2d(j,i);
           """
    weave.inline(code,['new_array','a_2d'],
                        type_factories = blitz_type_factories,
                        compiler='gcc')
    return new_array

Numeric arrays come in about 10 different types (byte, int, float , double,
complex float, ...) and this function works for all of them -- weave
generates
them on the fly.

The catalog is one of the more involved pieces of code in weave.  It handles
persistence of functions so that you don't have to re-compile every time you
restart the interpreter, and it also does some caching to keep overhead of
calling the same function multiple times in an inner loop to a minimum.

So, within the context of inline, the catalog handles what you are
describing
pretty well.  If you see anything missing that your implementation did well,
please let me know, and we'll try to work it in somehow.

thanks,
eric

----- Original Message -----
From: "Martin Casado" <casado2 at llnl.gov>
To: <c++-sig at python.org>
Sent: Sunday, January 20, 2002 5:53 PM
Subject: Re: [C++-sig] weave plus boost?


> On Saturday 19 January 2002 18:28, you wrote:
> > Hey group,
> >
> > I've just signed up to listen in on the group.  I haven't followed boost
to
> > closely, but am interested in catching up with what this community is
> > doing.
> >
> > My main interest here is to see if there is anyway that weave can
benefit
> > from the boost effort -- I'm almost sure there is.  weave
> > (www.scipy.org/site_content/weave) is a tool for, among other things,
> > inlining C/C++ within Python.  It also has a module called ext_tools
that
> > allows you to build C/C++ extension modules using Python.  Currently,
> > ext_tools only supports building modules that contain functions.  I'd
like
> > to eventually extend it so that it handles extension classes as well.
It
> > looks like it won't be to bad, but boost might make some aspects of this
> > more elegant and easier for the end user.
>
> Hi Eric,
>
>  Here is an idea I toyed around with last summer.  One of the
> challenges with making extension types that represent C++ classes is
> trying to come up with adequate ways to support templated classes.  Pretty
> much all packages I've looked at to automate the wrapping process  require
you
> to create a seperate class for each template instantiation... which makes
> sense
> since template instantion is done at compile time.  However, if I have a
> templated class, vector<typename>, I would really like to construct an
> instance
> of vector in python, like.
>
> myVector = vector("std::string")
>
> It is impossible to know which instances of vector are going to be needed
> before runtime with this method, so one option is to have vector(..)
generate
> an
> instance of vector<string> on the fly, compile it and load it in. Each
> instance extension
> would proabaly be cached, keyed by the type so subsequent requests for
> vector("std::string") wouldn't have to redo the process.
>
> I'm not sure how practical this would be, but at one time I had a sloppy
but
> working prototype and it was.. if nothing else.. neat.
>
>
>                         Cheers,
>                         ~~m
>
> _______________________________________________
> C++-sig mailing list
> C++-sig at python.org
> http://mail.python.org/mailman/listinfo/c++-sig
>