[C++-sig] Pickle python subclass of C++ interface

[John Reid]

John Reid wrote:
> John Reid wrote:
>> John Reid wrote:
>>>
>>>
>>> Ralf W. Grosse-Kunstleve wrote:
>>>>> Ok I'll give that a whirl. I was hoping to avoid doing 
>>>>> __getinitargs__()
>>>>> for each subclass as I have quite a few of them. If I didn't have a 
>>>>> C++
>>>>> base class then the pickling would just work as is. There's no way I
>>>>> can get back to that sort of situation with the C++ base class is 
>>>>> there?
>>>>
>>>> I don't know, but there may be. One (totally untested) idea would be to
>>>> give the base class __getstate__ and __setstate__ methods that 
>>>> inspect the
>>>> instance to re/store the state. Maybe you just need to return and 
>>>> restore
>>>> self.__dict__?
>>>> I guess you could mix-in the __getstate__, __setstate__ methods.
>>>>
>>>> Ralf
>>>
>>>
>>> I'm not sure what you mean by mix-in, but my first attempt involved 
>>> defining pickle suite getstate() and setstate() methods. I did not 
>>> define a getinitargs() method. Unfortunately when the derived object 
>>> was unpickled, __init__ was called with no arguments. As far as I can 
>>> see there's no way to use the boost.python pickle suite that does not 
>>> involve a call to __init__() on the unpickled object.
>>>
>>> I'll try having a go using the python pickling protocol's 
>>> __reduce__() method.
>>
>> Ok this seems to work by injecting a __reduce__() method into the C++ 
>> base class. Here ext is the extension module and A is the C++ base class:
>>
>>
>> import ext, cPickle, logging, copy_reg
>>
>> def __newobj__(cls, *args):
>>     return cls.__new__(cls, *args)
>>
>> def __reduce__(self):
>>     return (
>>         __newobj__,
>>         (self.__class__,),
>>         self.__dict__
>>     )
>> ext.A.__reduce__ = __reduce__
>>
>> class Derived(ext.A):
>>     def __init__(self, init_arg):
>>         self.data = init_arg
>>
>> derived = Derived(1)
>> pickled_repr = cPickle.dumps(derived)
>> unpickled = cPickle.loads(pickled_repr)
>> assert unpickled.data == derived.data
>>
> OK it seems I was a bit optimistic. When I pass an unpickled object back 
> to C++, I get this error:
> 
> Boost.Python.ArgumentError: Python argument types in
>     ext.method_that_takes_A(Derived)
> did not match C++ signature:
>     method_that_takes_A(A)
> 
> Does anyone have any ideas why my solution doesn't work when I try to 
> pass an unpickled object back to C++? I guess I need to learn a bit more 
> about how __new__() works.
> 
> 
Changing

def __newobj__(cls, *args):
      return cls.__new__(cls, *args)

to

def __newobj__(cls, *args):
     obj = cls.__new__(cls, *args)
     ext.A.__init__(obj)
     return obj

seems to have done the trick.