New subject: flextype.c -- extended type system

12 Sep 2002

      Hi Guido, py-dev,

preface:
--------
a week ago or so, I sent a patch to Guido that removes
the "etype" struct. This is a hidden structure that
extends types when they are allocated on the heap.
One restriction with this type was that types could
not be extended by metatypes for some internal reason.
I fixed this. Now meta-types can define extra slots
for types.

the point:
----------
I wasn't really after slots in types, but I wanted to
have a type that can be extended as the user likes to.
Using the re-worked etype (now named PyHeapType_Type),
I created a new meta-type with some cool new features
which give you C++ - like virtual methods and inheritance.
The new dynamic type (PyFlexType_Type) allows to clone
any existing type, and thereby to pass a virtual method
table which will be bound into the type. It is a bit like
slots and slot definitions, but the VMT definition is
written like a PyMethodDef list (in fact, I have PyCMethodDef),
and the created virtual function entries are spelled explicitly
in the type structure.

Structure of a VMT definition:

typedef struct _pycmethoddef {
     char        *name;      /* name to lookup in __dict__ */
     PyCFunction match;      /* to be found if non-overridden */
     void        *fast;      /* native C call */
     void        *wrap;      /* wrapped call into Python */
     int         offset;     /* slot offset in heap type */
} PyCMethodDef;

At creation time of a new flextype, all VMT entries in the
accumulated bases (accessed via the MRT) are scanned from
oldest to newest, and the new type's methods are retrieved
by the "name" entry. Then it is checked whether the
method descriptor still points to the original PyCFunction
entry (the "match" field). If it is still original, the
native C call (field "fast") is inserted into the VMT,
otherwise the wrapped Python callback (field "wrap") is
inserted.

As a result, it is now very cheap to use overridable small
methods in your C implementations, since it nearly comes to
no cost if the method isn't overridden.
It is also possible to have private methods, in the sense
that you can use inheritance between your flextypes without
publishing every virtual method to Python at all.

Here an example of my Stackless type system, where I made
my channel interface overridable:

(channelobject.h)
"""
#define CHANNEL_SEND_HEAD(func) \
     int func (PyChannelObject *self, PyObject *arg)

#define CHANNEL_SEND_EXCEPTION_HEAD(func) \
     int func (PyChannelObject *self, PyObject *klass, PyObject *value)

#define CHANNEL_RECEIVE_HEAD(func) \
     PyObject * func (PyChannelObject *self)

typedef struct _pychannel_heaptype {
     PyFlexTypeObject type;
     /* the fast callbacks */
     CHANNEL_SEND_HEAD(           (*send)             );
     CHANNEL_SEND_EXCEPTION_HEAD( (*send_exception)   );
     CHANNEL_RECEIVE_HEAD(	 (*receive)          );
} PyChannel_HeapType;

int init_channeltype(void);
"""

Here the VMT definition of channelobject.c:
"""
static PyCMethodDef
channel_cmethods[] = {
     CMETHOD_PUBLIC_ENTRY(PyChannel_HeapType, channel, send),
     CMETHOD_PUBLIC_ENTRY(PyChannel_HeapType, channel, send_exception),
     CMETHOD_PUBLIC_ENTRY(PyChannel_HeapType, channel, receive),
     {NULL}                       /* sentinel */
};

"""

where the CMETHOD_PUBLIC_ENTRY macro looks like this:
/*
  * a public entry defines
  * - the function name      "name"
  * - the PyCFunction        class_name       seen from Python,
  * - the fast function      impl_class_name  implements the method for C
  * - the wrapper function   wrap_class_name  that calls back into a 
Python override.
  */
#define CMETHOD_PUBLIC_ENTRY(type, prefix, name) \
     {#name, (PyCFunction)prefix##_##name, &impl_##prefix##_##name, 
&wrap_##prefix##_##name, \
     offsetof(type, name)}

So basically three functions are involved in a virtual method:
the PyCFunction, the C implementation and a wrapper.
Normally, the PyCFunction and the implementation can be
identical, but usually my C interface looks slightly
different from the Python interface, for convenience.

Here an excerpt from channel_send:
"""
int
PyChannel_Send(PyChannelObject *self, PyObject *arg)
{
	PyChannel_HeapType *t = (PyChannel_HeapType *) self->ob_type;
	return t->send(self, arg);
}

static CHANNEL_SEND_HEAD(impl_channel_send)
{
     PyThreadState *ts = PyThreadState_GET();
     PyTaskletObject *sender, *receiver;
.... implementation skipped ....
}

static CHANNEL_SEND_HEAD(wrap_channel_send)
{
     PyObject * ret = PyObject_CallMethod((PyObject *) self, "send", 
"(O)", arg);
     return slp_return_wrapper(ret);
}

static PyObject *
channel_send(PyObject *myself, PyObject *arg)
{
     if (impl_channel_send((PyChannelObject*)myself, arg))
         return NULL;
     Py_INCREF(Py_None);
     return Py_None;
}
"""

end of story.

Summary:
--------
Overridable methods have always been present in Python,
via the built-in method slots. My extension methods
give the same functionality to the user, at maximum possible
speed (only templates can be faster).
The benefit is that users can use much more flexibility
in C modules than before, without fear of speed loss.
I believe that virtual methods will be used more often,
since it is cheap, flexible and compatible with Python.

Please let me know if there is interest to use this techique
in the Python core. I'm also not sure how to show the complete
thing, since it is partially a patch to the existing type
implementation (concerning the etype), partially a new C
module flextype.c, and the rest is part of Stackless.
Does it make sense (would somebody look at it) if I create
a little demo application or something?

cheers - chris

-- 
Christian Tismer             :^)   mailto:tismer@tismer.com
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  pager +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/

flextype.c -- extended type system

Christian Tismer

Brett Cannon

Christian Tismer

Brett Cannon

Guido van Rossum

Christian Tismer

Christian Tismer

Christian Tismer

martin＠v.loewis.de

Thomas Heller

David Abrahams

David Abrahams

Thomas Heller

Christian Tismer

Christian Tismer

tags

participants (6)