Pickling an extension type subclasses problems

[Gaetan de Menten]

Hi all,

I am trying to write an (optional!) C extension for SQLAlchemy, and I
am struggling to make an extension type picklable *and* using the same
format as the pure Python version (so that computers with the C
extension can unpickle pickles from computers without it and
vice-versa). Also the extension type (MyBase in the example code
below) only serves as a base class for the type that get pickled
(MyPythonType). MyPythonType also has further subclasses and those
should be picklable too.

My setup is as follow:

On the Python side:
================

try:
    from cextension import MyBase
except ImportError:
    class MyBase(object):
        __slots__ = ('a', 'b', 'c')
        def __init__(self, a, b, c):
            self.a, self.b, self.c = a, b, c

        def __getstate__(self):
            ...

        def __setstate__(self, state):
            ...
        ...


class MyPythonType(MyBase):
    ...

On the .c side:
===========
I implemented a reduce method, which returns

Py_BuildValue("(O(s)N)", Py_TYPE(self), "__state__", state);

and in my BaseRowProxy_init method, I check the number of arguments,
whether the "__state__" marker is present and unpack the state if it
is. This makes the type nicely pickleable, but this is obviously not
the same "pickle format" as the Python version.

What would you recommend in this kind of situation? I'm a bit tired of
trial and error... My last hope is to define __reduce__ on the Python
side too, and make it use a module-level "reconstructor" function as
follow:

def mypythontype_reconstructor(cls, state):
    obj = object.__new__(cls, state):
    ....
    return obj

Will this work? Any other idea?

Thanks in advance,
-- 
Gaëtan de Menten