[Python-Dev] Pickling instances of nested classes

[Samuele Pedroni]

Walter Dörwald wrote:
> Samuele Pedroni wrote:
> 
>> Walter Dörwald wrote:
>>
>>> [User cases for pickling instances of nested classes]
>>> So is this change wanted? useful? implementable with reasonable 
>>> effort? Or
>>> just not worth it?
>>
>>
>> notice that in this cases often metaclasses are involved or could 
>> easely be, so if pickling would honor __reduce__ or __reduce_ex__ on 
>> metaclasses (which right now it doesn't treating their instances as 
>> normal classes) one could roll her own solution without the burden for 
>> the language of implementing pickling of nested classes in general, so 
>> I think that would make more sense, to add support to honor 
>> __reduce__/__reduce_ex__ for metaclasses.
> 
> 
> Sorry, I don't understand: In most cases it can be possible to
> work around the nested classes problem by implementing custom pickling 
> functionality (getstate/setstate/reduce/reduce_ex). But it is probably 
> impossible to implement this once and for all in a common base class, 
> because there's no way to find the real name of the nested class (or any 
> other handle that makes it possible to retrieve the class from the 
> module on unpickling).
> 
> And having the full name of the class available would certainly help in 
> debugging.
> 

that's probably the only plus point but the names would be confusing wrt
  modules vs. classes.

My point was that enabling reduce hooks at the metaclass level has
propably other interesting applications, is far less complicated than
your proposal to implement, it does not further complicate the notion of
what happens at class creation time, and indeed avoids the
implementation costs (for all python impls) of your proposal and still
allows fairly generic solutions to the problem at hand because the
solution can be formulated at the metaclass level.

If pickle.py is patched along these lines [*] (strawman impl, not much
tested but test_pickle.py still passes, needs further work to support
__reduce_ex__ and cPickle would need similar changes) then this example 
works:


class HierarchMeta(type):
   """metaclass such that inner classes know their outer class, with 
pickling support"""
   def __new__(cls, name, bases, dic):
       sub = [x for x in dic.values() if isinstance(x,HierarchMeta)]
       newtype = type.__new__(cls, name, bases, dic)
       for x in sub:
           x._outer_ = newtype
       return newtype

   def __reduce__(cls):
       if hasattr(cls, '_outer_'):
           return getattr, (cls._outer_, cls.__name__)
       else:
           return cls.__name__

# uses the HierarchMeta metaclass
class Elm:
   __metaclass__ = HierarchMeta

   def __init__(self, **stuff):
       self.__dict__.update(stuff)

   def __repr__(self):
       return "<%s %s>" % (self.__class__.__name__, self.__dict__)

# example
class X(Elm):
   class Y(Elm):
     pass

   class Z(Elm):
     pass

import pickle

x = X(a=1)
y = X.Y(b=2)
z = X.Z(c=3)

xs = pickle.dumps(x)
ys = pickle.dumps(y)
zs = pickle.dumps(z)

print pickle.loads(xs)
print pickle.loads(ys)
print pickle.loads(zs)

pedronis$ python2.4 example.py
<X {'a': 1}>
<Y {'b': 2}>
<Z {'c': 3}>



[*]:
--- pickle.py.orig	Wed Mar 30 20:37:14 2005
+++ pickle.py	Thu Mar 31 21:09:41 2005
@@ -298,12 +298,19 @@
              issc = issubclass(t, TypeType)
          except TypeError: # t is not a class (old Boost; see SF #502085)
              issc = 0
+        reduce = None
          if issc:
-            self.save_global(obj)
-            return
+            for x in t.__mro__:
+                if x is not object and '__reduce__' in x.__dict__:
+                    reduce = x.__dict__['__reduce__']
+                    break
+            else:
+                self.save_global(obj)
+                return

          # Check copy_reg.dispatch_table
-        reduce = dispatch_table.get(t)
+        if not reduce:
+            reduce = dispatch_table.get(t)
          if reduce:
              rv = reduce(obj)
          else: