[Python-Dev] [Python 2.2 BUG] pickle/cPickle does not find __slots__

Kevin Jacobs jacobs@penguin.theopalgroup.com
Fri, 15 Feb 2002 09:57:19 -0500 (EST)


[I tried to post this on SourceForge, but as usual, it hates my guts]

I have been hacking on ways to make lighter-weight Python objects using the
__slots__ mechanism that came with Python 2.2 new-style class.  Everything
has gone swimmingly until I noticed that slots do not get pickled/cPickled
at all!

Here is a simple test case:

  import pickle,cPickle
  class Test(object):
    __slots__ = ['x']
    def __init__(self):
      self.x = 66666

  test = Test()

  pickle_str  = pickle.dumps( test )
  cpickle_str = cPickle.dumps( test )

  untest  = pickle.loads( pickle_str )
  untestc = cPickle.loads( cpickle_str )

  print untest.x    # raises AttributeError
  print untextc.x   # raises AttributeError

Clearly, this is incorrect behavior.  The problem is due a change in object
reflection semantics.  Previously (before type-class unification), a
standard Python object instance always contained a __dict__ that listed all
of its attributes.  Now, with __slots__, some classes that do store
attributes have no __dict__ or one that only contains what did not fit into
slots.

Unfortunately, there is no trivial way to know what slots a particular class
instance really has. This is because the __slots__ list in classes and
instances can be mutable!  Changing these lists _does not_ change the object
layout at all, so I am unsure why they are not stored as tuples and the
'__slots__' attribute is not made read-only.  To be pedantic, the C
implementation does have an immutable and canonical list(s) of slots, though
they are well buried within the C extended type implementation.

So, IMHO this bug needs to be fixed in two steps:

First, I propose that class and instance __slots__ read-only and the lists
made immutable.  Otherwise, pickle, cPickle, and any others that want to use
reflection will be SOL.  There is certainly good precedent in several places
for this change (e.g., __bases__, __mro__, etc.) I can submit a fairly
trivial patch to do so.  This change requires Guido's input, since I am
guessing that I am simply not privy to the method, only the madness.

Second, after the first issue is resolved, pickle and cPickle must then be
modified to iterate over an instance's __slots__ (or possibly its class's)
and store any attributes that exist.  i.e., some __slots__ can be empty and
thus should not be pickled.  I can also whip up patches for this, though I'll
wait to see how the first issue shakes out.

Regards,
-Kevin

PS:  There may be another problem when when one class inherits from another
     and both have a slot with the same name.

     e.g.:
       class Test(object):
         __slots__ = ['a']

       class Test2(Test):
         __slots__ = ['a']

       test=Test()
       test2=Test2()
       test2.__class__ = Test

    This code results in this error:

      Traceback (most recent call last):
        File "<stdin>", line 1, in ?
      TypeError: __class__ assignment: 'Test' object layout differs from 'Test2'

    However, Test2's slot 'a' entirely masks Test's slot 'a'.  So, either
    there should be some complex attribute access scheme to make both slots
    available OR (in my mind, the preferable solution) slots with the same
    name can simply be re-used or coalesced.  Now that I think about it,
    this has implications for pickling objects as well.  I'll likely leave
    this patch for Guido -- it tickles some fairly hairy bits of typeobject.

    Cool stuff, but the rabbit hole just keeps getting deeper and deeper....

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com