adding methods on the fly

[Alex Martelli]

Renzo Tomaselli wrote:

> Alex Martelli <aleax at aleax.it> wrote in message
> news:<DGnW8.46007$Jj7.1292356 at news1.tin.it>...
> 
>> Py_CompileString (careful about underscore placement!-) won't let you
>> easily build a function (or unbound-method) object, which is what you
>> need (though you can get there eventually, but it's a long way 'round).
> 
> Fairly disappointing. Since I plan to store such used-defined methods
> into a OO database, I would expect it being much more efficient to
> store them in a precompiled form.

It keeps looking to me as if you're making a lot of implicit assumptions,
and that many of them are wrong.  What makes you believe, for example,
that just exec'ing "def foo(self): pass" does NOT leave you "a compiled
form" of function foo (in the locals dictionary used for executing)?
You can marshal such objects to bytestrings, too, for storing as BLOBs
or whatever.

> I still wonder why it's so hard (from a C/C++ perspective) to walk
> from a "def foo(self): pass" string to a callable object to be added
> to a class.

But it's not.  exec that with a locals dictionary, and in the dictionary
at key 'foo' you find exactly a Python callable object (a function
object, to be precise).  What IS "so hard" about this?  def is an
executable statement, used to create function objects, so, if you
start with a def statement and want a function object, you, of course,
execute the statement.  What COULD possibly be easier or mote natural?

Maybe you have some unspoken assumption that "compiling" is what
gives you an executable object.  But it's not necessarily so, in
Python.  In Python, you get an executable object (a function)
from executing a def statement.  Compiling (using the compile
built-in) gives you a *code* object, which is "executable" in
a sense but is not directly *callable* -- and you do need a
*callable* object for your purposes (a function is callable).


> Ok, I'm not forced to fit all in C++, after all I define the features
> my platform is going to offer. An extra layer in Python is acceptable,
> provided I hide it from common usage (e.g. adding/invoking methods).

My point exactly.

> So let's see this "easy" way in more details.

Sure.

> Let's assume I created a C++ extension in module myExt, which exports
> a type for each C++ class to be wrapped (and this type is defined so
> that it can be inherited from). Then I would need to defined an empty

OK, here's a specific example -- inhe.c exposing an inheritable type
with no other functionality (simplified mostly to remove a zillion
zeros from the type struct, you can reconstruct it from comments):

#include "Python.h"

typedef struct {
    PyObject_HEAD
} inhe;

static void
inhe_dealloc(PyObject *op)
{
    op->ob_type->tp_free(op);
}

static PyTypeObject inhe_t = {
    PyObject_HEAD_INIT(0)
    0,
    "inhe",
    sizeof(inhe),
    0,
    inhe_dealloc,
    ...
    0,                                  /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE,
    "inheritable type",
    0,                                  /* tp_traverse */
    ...
    0,                                  /* tp_init */
    PyType_GenericAlloc,                /* tp_alloc */
    PyType_GenericNew,                  /* tp_new */
    _PyObject_Del,                      /* tp_free */
};

static PyMethodDef no_methods[] = { {0} };

void
initinhe(void)
{
    PyObject* self;
    inhe_t.ob_type = &PyType_Type;
    self = Py_InitModule("inhe", no_methods);
    PyObject_SetAttrString(self, "inhe", (PyObject*)&inhe_t);
}

Save this as inhe.c, make a setup.py in the same directory with:

from distutils.core import setup, Extension

setup(name = "inhe",
    version = "1.0",
    description = "inheritable type",
    maintainer = "Alex Martelli",
    maintainer_email = "aleaxit at yahoo.com",

    ext_modules = [ Extension('inhe', sources=['inhe.c']) ]
)

cd to that directory and do "python setup.py install" and
you'll be able to try out some Python code using it, e.g.

import inhe

class gop(inhe.inhe): pass

g = gop()
g.pop=23
print g.pop
del g
print 'OK'


> class like:
> 
> #P1: wrapping
> import myExt
> 
> class foo(myFoo):   # one class per object to wrap
>     pass

Definitely *NOT* per-object, but rather per CLASS (or type) to wrap.
And if myFoo is a class exposed from myExt, you need to use myExt.myFoo
as the base class.

> myExt.hook(foo())

Not quite sure what this hook function is, but presumably it's
something your module myExt uses to keep track of subclasses of
the classes it exposes, which seems OK (you could do it slightly
differently, by tweaking the metaclass for example, but as you
don't intend to let endusers freely subclass your extension's
classes the simpler mechanism appears to be the better one here).


> Then from time to time someone adds a new method to class foo:
> 
> #P2: declaration
> def m1(self, args):
>     do_something
> 
> #P3: appending
> foo.m1 = m1
> 
> #P4: execution
> and run it from C++ or from Python.

"From C++" is unclear to me, unless it's via PyRun_* or the like.


> Now, the entire game is C++ driven. P1, P2, P3 can be merged together
> in a string(P2 is an user-defined string) and run through
> PyRun_SimpleString. Now new method is attached to class foo and it can
> be run through PyObject_CallMethod on the hooked class instance.

Or in other Python-oriented ways, yes.

> I still prefer a PyClass_New/PyMethod_New/PyInstance_New sequence from
> C++, but here we are back to the point of feeding PyMethod_New with
> appropriate value from pass P2 above. This is the missing link about
> which I couln't find anything on the list (dealing with callables is
> general is a black hole).

And I still don't see why you "still prefer" a more complex solution
to a simpler one.  Except, perhaps, due to some unspoken (and wrong)
assumptions, as I hypothesized above.

There's nothing particularly mysterious about "dealing with callables":
the only difference between a callable and any other Python object
is that the former has a non-null entry in tp_call, which changes
just about nothing else except allowing Python code to call it!
(Classic-class instances typically have nonnull entries everywhere
and need more complicated tests to check if they're callable and
so on -- but you don't need to worry about classic classes here).

> Missing precompilation is another missing feature of this solution.

Just because you choose to "merge together in a string", above,
several steps that you might perfectly well keep separate.  As I
have by now repeated a few too many times, but don't seem to be
getting through to you: if you exec JUST the def statement, and
supply an initially empty local dict, then you can fish out of
that dict the function object, and do various things with it --
including marshaling it into a bytestring to be saved and restored
at will, as well as setting it as an attribute of one or more
classes.  Nothing stops you from building the classes with
PyClass_New, if for some reason you think that's preferable to
simpler Python ways (it's not, but it's not terribly worse either,
so, suit yourself) -- 


>> Another possibility is to look at Boost Python, if the schedule for their
>> new release 2 (supporting Python 2.2 fully) is compatible with your
>> timing constraints
> 
> Several people suggested me to look at Boost, but I feel it deals
> primarily with extending, while I'm on the opposite side.

_Everything_ you'll find "deals primarily with extending", because
that's where all the interesting / difficult / challenging stuff IS.

Embedding is very simple once you have totally mastered extending.

Your problems might stem from trying to skip the "totally mastered"
step.  Once you have a complete working solution in Python with
extensions, moving some or all of the solution back into C or C++
code is invariably rather trivial, although verbose/boring if indeed 
it's worth doing at all.  But starting with a Python-coded prototype is 
most often the best idea -- Python+extensions, in this case.

> On the other hand I don't feel good to add an entire package only for
> the purpose of adding new Python methods on the fly.

If you have tiny C++ programs to start with, the idea of adding 10KB
or code, of more, can indeed be unsettling -- I tend not to think
of that because the C++ programs I dealt with tended to be megabytes
rather than kilobytes, so a few tens of KB more or less didn't matter.

Further, Boost Python is hardly a SIMPLE piece of code -- the C API
are much simpler (although fussier and messier to deal with, as is
often the case for C-coded versus C++-coded functionality).  So, if
you don't need it (because you need to expose to Python only a very
few, simple classes, and would take no advantage of the many extras
that Boost Python embodies, such as setters/getters, ability to
override virtual functions from Python in such a way that the
override will also be seen by C++ callers, etc, etc), the simpler
solution, based on Python's C API, can surely be preferable.

The puzzle is that you seem persistently unsatisfied with said
"simpler solution", and I _still_ can't quite grasp WHY.


Alex