pickling instances of metaclass generated classes

[Peter Otten]

lars van gemerden wrote:

> On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke... at gmail.com> wrote:
>> On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l... at rational-it.com>
>> wrote:
>>
>> > Hello,
>>
>> > Can someone help me with the following:
>>
>> > I am using metaclasses to make classes and these classes to make
>> > instances. Now I want to use multiprocessing, which needs to pickle
>> > these instances.
>>
>> > Pickle cannot find the class definitions of the instances. I am trying
>> > to add a line to the __new__ of the metaclass to add the new class
>> > under the right name in the right module/place, so pickle can find
>> > it.
>>
>> > Is this the right approach? Can anyone explain to me where/how to add
>> > these classes for pickle to find and maybe why?
>>
>> It sounds like you're trying to do something like this?
>>
>> >>> class MetaClass(type):
>>
>> ...     pass
>> ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
>> >>> instance
>>
>> <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
>> >>> pickle.dumps(instance)
>>
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "c:\python27\lib\pickle.py", line 1374, in dumps
>> Pickler(file, protocol).dump(obj)
>> File "c:\python27\lib\pickle.py", line 224, in dump
>> self.save(obj)
>> File "c:\python27\lib\pickle.py", line 331, in save
>> self.save_reduce(obj=obj, *rv)
>> File "c:\python27\lib\pickle.py", line 401, in save_reduce
>> save(args)
>> File "c:\python27\lib\pickle.py", line 286, in save
>> f(self, obj) # Call unbound method with explicit self
>> File "c:\python27\lib\pickle.py", line 562, in save_tuple
>> save(element)
>> File "c:\python27\lib\pickle.py", line 295, in save
>> self.save_global(obj)
>> File "c:\python27\lib\pickle.py", line 748, in save_global
>> (obj, module, name))
>> pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
>> it's not found as __main__.<Anonymous>
>>
>> Yeah, pickle's not going to work with anonymous classes.  As you
>> suggest, you could dynamically add the classes to the module namespace
>> so that pickle.dumps will find them, but bear in mind that they will
>> also have to exist when calling pickle.loads, so you will need to be
>> able to reconstruct the same anonymous classes before unpickling later
>> on.
>>
>> Cheers,
>> Ian
> 
> Thank you Ian for the minimal example. This is almost the case i was
> trying to discribe, however the classes will be named at runtime and
> the class definitions will be persisted in a database.
> 
> Can you help me with how to add the classes to the correct namespace?
> The application spans multiple modules (or compared to the example,
> the metaclass definition will be in another module then one where the
> class and instance will be generated).

If the metaclass is global in whatever module you don't need to bother about 
that. The problem is that you cannot access the classes (metaclass 
instances) under a dotted name. One workaround is a pseudo-module that does 
whatever is needed to recreate the class:

import pickle
import sys

class MetaClass(type):
    pass

class M(object):
    def __init__(self, module):
        self.__module = module
    def __getattr__(self, name):
            print "creating class", name
            class_ = MetaClass(name, (), {"__module__": self.__module})
            setattr(self, name, class_)
            return class_

sys.modules["m"] = M("m")
import m
c = m.x
s = pickle.dumps(c)
print repr(s)
d = pickle.loads(s)

assert c is d

sys.modules["m"] = M("m")
e = pickle.loads(s)

assert c is not e

The official way is probably what Robert mentioned, via the copy_reg module, 
but I didn't get it to work.