Pickling classes (not class instances)

[Nicolas M. Thiéry]

	Dear python developers,

Purpose of this e-mail:
-----------------------

How to customize how a class (not an instance of a class!!!) is
pickled?


Example:
==============================================================================
class metaclass(type):
    def __new__(mcs, name, bases, dict):
        print " running metaclass.__new__"
        return type.__new__(mcs, name, bases, dict)
    #
    def __reduce_class__(self):
        print "reducing a class"
        # do the real work

c = metaclass("foo", (object,), dict())

import copy_reg
copy_reg.pickle(metaclass, metaclass.__reduce_class__)

pickle.dumps(c)
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call
last)
...
PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
__main__.foo
==============================================================================


Context:
--------

I am working on the Sage project (www.sagemath.org), and more
precisely on the category infrastructure. The code is building lots of
classes on the fly by composing preexisting classes by inheritance
(for the curious, see the doc of the class Category in
http://sage.math.washington.edu:2144/file/1567cea09170/categories-nt.patch).

Since those classes are built on the fly, they cannot be pickled with
the default mechanism of name lookup. A proper pickling would be to
rebuild the class anew. Nothing problematic, except for the question
above.


Discussion:
-----------

It sounds like copy_reg would be the natural way to go (as in the
example above). However, its documentation suggests that it explicitly
is not intended for pickling classes, e.g. first paragraph of:

http://docs.python.org/library/copy_reg.html#module-copy_reg

is:

  The copy_reg module provides support for the pickle and cPickle
  modules. The copy module is likely to use this in the future as
  well.  It provides configuration information about object
  constructors which are not classes. Such constructors may be factory
  functions or class instances.

And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
(and similarly in cpickle), the copy-reg dispatch is explicit bypassed
for metaclasses:


        # Check for a class with a custom metaclass; treat as regular
class
        try:
            issc = issubclass(t, TypeType)
        except TypeError: # t is not a class (old Boost; see SF
#502085)
            issc = 0
        if issc:
            self.save_global(obj)
            return

        # Check copy_reg.dispatch_table
        reduce = dispatch_table.get(t)

Which causes the failure above.


Is there a specific reason for this restriction?

Would it be thinkable to move up the copy reg dispatch before the
metaclass treatment in pickle and cPickle? I did it locally, and it
fixed my problem.

If not, what would be the right way to achieve this?

Many thanks in advance!

Best regards,
				Nicolas
--
Nicolas M. Thiéry "Isil" <nthiery at users.sf.net>
http://Nicolas.Thiery.name/