pickling instances of metaclass generated classes

[lars van gemerden]

On Dec 30, 4:56 pm, lars van gemerden <l... at rational-it.com> wrote:
> On Dec 30, 12:16 pm, lars van gemerden <l... at rational-it.com> wrote:
>
>
>
>
>
>
>
>
>
> > On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke... at gmail.com> wrote:
>
> > > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l... at rational-it.com> wrote:
>
> > > > Hello,
>
> > > > Can someone help me with the following:
>
> > > > I am using metaclasses to make classes and these classes to make
> > > > instances. Now I want to use multiprocessing, which needs to pickle
> > > > these instances.
>
> > > > Pickle cannot find the class definitions of the instances. I am trying
> > > > to add a line to the __new__ of the metaclass to add the new class
> > > > under the right name in the right module/place, so pickle can find
> > > > it.
>
> > > > Is this the right approach? Can anyone explain to me where/how to add
> > > > these classes for pickle to find and maybe why?
>
> > > It sounds like you're trying to do something like this?
>
> > > >>> class MetaClass(type):
>
> > > ...     pass
> > > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
> > > >>> instance
>
> > > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
> > > >>> pickle.dumps(instance)
>
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "c:\python27\lib\pickle.py", line 1374, in dumps
> > >     Pickler(file, protocol).dump(obj)
> > >   File "c:\python27\lib\pickle.py", line 224, in dump
> > >     self.save(obj)
> > >   File "c:\python27\lib\pickle.py", line 331, in save
> > >     self.save_reduce(obj=obj, *rv)
> > >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
> > >     save(args)
> > >   File "c:\python27\lib\pickle.py", line 286, in save
> > >     f(self, obj) # Call unbound method with explicit self
> > >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
> > >     save(element)
> > >   File "c:\python27\lib\pickle.py", line 295, in save
> > >     self.save_global(obj)
> > >   File "c:\python27\lib\pickle.py", line 748, in save_global
> > >     (obj, module, name))
> > > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
> > > it's not found as __main__.<Anonymous>
>
> > > Yeah, pickle's not going to work with anonymous classes.  As you
> > > suggest, you could dynamically add the classes to the module namespace
> > > so that pickle.dumps will find them, but bear in mind that they will
> > > also have to exist when calling pickle.loads, so you will need to be
> > > able to reconstruct the same anonymous classes before unpickling later
> > > on.
>
> > > Cheers,
> > > Ian
> > Ian also wrote:
>
> > '''
> > Actually, I was wrong, you probably don't need to do that.  I suggest
> > going with Robert Kern's suggestion to either register the class with
> > the copy_reg module, or (perhaps better since it won't leak
> > registrations) implement a __reduce__ method on the class.  For
> > example, this seems to work:
>
> > >>> def reconstructor(*metaclass_args):
>
> > ...     cls = MetaClass.build_class(*metaclass_args)
> > ...     self = cls.__new__(cls)
> > ...     return self
> > ...>>> class MetaClass(type):
>
> > ...     @classmethod
> > ...     def build_class(mcs, arg1, arg2, arg3):
> > ...         # Do something useful with the args...
> > ...         class _AnonymousClass(object):
> > ...             __metaclass__ = mcs
> > ...             def __reduce__(self):
> > ...                 return (reconstructor, ('foo', 'bar', 'baz'),
> > self.__dict__)
> > ...         return _AnonymousClass
> > ...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
> > >>> instance
>
> > <__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42
> > >>> import pickle
> > >>> s = pickle.dumps(instance)
> > >>> s
>
> > "c__main__\nreconstructor
> > \np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)
> > >>> inst2
>
> > <__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
> > 42
> > >>> inst2.__class__ is instance.__class__
>
> > False
>
> > Cheers,
> > Ian
>
> > '''
>
> Interesting, though I cannot say I completely understand this solution
> (looked up __reduce__, but still). I am trying to adapt this example
> to a situation where the metaclass generated classes are named at
> runtime (not anonymous), but cannot figure it out.
>
> Cheers, Lars

Found a way to name the classes:

def reconstructor(*metaclass_args):
    cls = MetaClass2.build_class(*metaclass_args)
    self = cls.__new__(cls)
    return self

class MetaClass(type):
    @classmethod
    def build_class(mcs, name, arg1, arg2, arg3):
        return mcs(name, (object,), {"__reduce__": lambda e:
(reconstructor2, (name, arg1, arg2, arg3), e.__dict__)})

I still wonder whether it might be easier to add the class to the
namespace. Can anyone help me with that?

Regards, Lars