[Python-bugs-list] [ python-Bugs-520644 ] __slots__ are not pickled
noreply@sourceforge.net
noreply@sourceforge.net
Thu, 28 Feb 2002 11:36:36 -0800
Bugs item #520644, was opened at 2002-02-20 15:50
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=520644&group_id=5470
Category: Type/class unification
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Samuele Pedroni (pedronis)
Assigned to: Nobody/Anonymous (nobody)
Summary: __slots__ are not pickled
Initial Comment:
[Posted on behalf of Kevin Jacobs]
I have been hacking on ways to make lighter-weight
Python objects using the
__slots__ mechanism that came with Python 2.2 new-
style class. Everything
has gone swimmingly until I noticed that slots do not
get pickled/cPickled
at all!
Here is a simple test case:
import pickle,cPickle
class Test(object):
__slots__ = ['x']
def __init__(self):
self.x = 66666
test = Test()
pickle_str = pickle.dumps( test )
cpickle_str = cPickle.dumps( test )
untest = pickle.loads( pickle_str )
untestc = cPickle.loads( cpickle_str )
print untest.x # raises AttributeError
print untextc.x # raises AttributeError
...
see
http://aspn.activestate.com/ASPN/Mail/Message/python-
dev/1031499
----------------------------------------------------------------------
>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-02-28 14:36
Message:
Logged In: YES
user_id=6380
Good point. So maybe it should be up to the class to define
how to pickle slots.
An alternative idea could look at the type of descriptors;
slots use a different type than properties.
----------------------------------------------------------------------
Comment By: Samuele Pedroni (pedronis)
Date: 2002-02-28 14:29
Message:
Logged In: YES
user_id=61408
[Guido on python-dev]
In particular, the fact that instances of classes with
__slots__
appear picklable but lose all their slot values is a bug --
these
should either not be picklable unless you add a __reduce__
method, or
they should be pickled properly. ...
I haven't made up my mind on how to fix this -- it would
be
nice if __slots__ would automatically be pickled, but it's
tricky
(although I think it's doable -- without ever referencing
the
__slots__ variable :-).
[pedronis - my 2cts]
unless you plan some low-level (non-python-level) solution,
I think a main question is whether member and properties
are distinguishable and maybe whether among members
basic type members (file.softspace etc) and __slots__
members are distinguishable
It would be somehow strange and redundant if properties
value would be automatically pickled (I see them as
computed value)
In java (bean) properties are not pickled and even
fields (= slots) can be marked as transient to avoid
their serialization.
In your picture it seems that all those things are
not to be dinstinguished, so probably no automatic
serialization, if there are members and given that
actually files for example cannot be pickled, would
be a reasonable solution.
Otherwise (distinguishable case) other
automatic approaches can make sense too.
Just some - I hope valuable - input and my opinion.
----------------------------------------------------------------------
Comment By: Kevin Jacobs (jacobs99)
Date: 2002-02-22 10:03
Message:
Logged In: YES
user_id=459565
Oops. Please ignore the last paragraph of point #5.
Samuele's __allslots__ is fine with regard to the
example I presented.
----------------------------------------------------------------------
Comment By: Kevin Jacobs (jacobs99)
Date: 2002-02-22 09:52
Message:
Logged In: YES
user_id=459565
Samuele's sltattr.py is an interesting approach, though I
am not entirely
sure it is necessary or feasible sufficiently address the
significant
problems with slots via proxying __dict__ (see #5 below).
Here is a mostly
complete list of smaller changes that are somewhat
orthogonal to how we
address accesses to __dict__:
1) Flatten slot lists: Change obj.__class__.__slots__ to
return an
immutable list of all slot descriptors in the object
(including all
those of base classes). The motivation for this is
similar in spirit
to storing a flattened __mro__.
The advantages of this change are:
a) allows for fast and explicit object reflection that
correctly finds
all dict attributes, all slot attributes.
b) allows reflection implementations (like vars
(object) and pickle) to
treat dict and slot attrs differently if we choose
not to proxy
__dict__. This has several advantages, as
explained in change #2.
Also importantly, this way it is not possible
to "lose" descriptors
permanently by deleting them from
obj.__class__.__dict__.
2) Update reflection API even if we do not choose to
proxy __dict__: Alter
vars(object) to return a dictionary of all attributes,
including both
the contents of the non-proxied __dict__ and the valid
attributes that
result from iterating over __slots__ and evaluating
the descriptors.
The details of how this is best implemented depend on
how we wish to
define the behavior of modifying the resulting
dictionary. It could be
either:
a) explicitly immutable, which involves creating
proxy objects
b) mutable, which involves copying
c) undefined, which means implicitly immutable
Aside from the questions over the nature of the return
type, this
implementation (coupled with #1) has distinct
advantages. Specifically
the native object.__dict__ has a very natural internal
representation
that pairs attribute names directly with values. In
contrast, a fair
amount of additional work is needed to extract the
slots that store
values and create a dictionary of their names and
values. Other
implementations will require a great deal more work
since they would
have to traverse though base classes to collecting
slot descriptors.
3) Flatten slot inheritance: Update the new-style object
inheritance
mechanism to re-use slots of the same name, rather
than creating a new
slot and hiding the old. This makes the inheritance
semantics of slots
equivalent to those of normal instance attributes and
avoids
introducing an ad-hoc and obscure method of data
hiding.
4) Update standard library to use new reflection API (and
make them robust
to properies at the same time) if we choose not to
proxy __dict__.
Virtually all of the changes are simple and involve
updating these
constructs:
a) obj.__dict__
b) obj.__dict__[blah]
c) obj.__dict__[blah] = x
(What these will become depends on other factors,
including the context
and semantics of vars(obj).)
Here is a fairly complete list of Python 2.2 modules
that will need to
be updated:
copy, copy_reg, inspect, pickle, pydoc, cPickle,
Bastion, codeop,
dis, doctest, gettext, ihooks, imputil, knee, pdb,
profile, rexec,
rlcompleter, tempfile, unittest, xmllib, xmlrpclib
5) (NB: potentially controversial and not required) We
could alter the
descriptor protocol to make slots (and properties)
more transparent
when the values they reference do not exist. Here
is an example to
illustrate this:
class A(object):
foo = 1
class B(A):
__slots__ = ('foo',)
b = B()
print b.foo
> 1 or AttributeError?
Currently an AttributeError is raised. However, it
is a fairly easy
change to make AttributeErrors signal that
attribute resolution is
to continue until either a valid descriptor is
evaluated, an
instance-attribute is found, or until the
resolution fails after
search the meta-type, the type and the instance
dictionary.
The problem illustrated by the above code also
occurs when trying to
create proxies for __dict__, if the proxy worked on
the basis of the
collected slot descriptors (__allslots__ in
Samuele's example).
I am prepared to submit patches to address each of these
issues. However, I
do want feedback beforehand, so that I do not waste time
implementing
something that will never be accepted.
----------------------------------------------------------------------
Comment By: Samuele Pedroni (pedronis)
Date: 2002-02-21 20:33
Message:
Logged In: YES
user_id=61408
some slots more like attrs illustrative python code
----------------------------------------------------------------------
Comment By: Kevin Jacobs (jacobs99)
Date: 2002-02-21 12:51
Message:
Logged In: YES
user_id=459565
This bug raises questions about what a slot really is.
After a fair amount of discussion on Python-dev, we
have come up with basically two answers:
1) a slot is a struct-member that is part of the
private implementation of an object. Slots should
have their own semantics and not be expected to
act like Python instance attributes.
2) slots should be treated just like dict instance
attributes except they are allocated statically
within the object itself, and require slightly
different reflection methods.
Under (1), this bug isn't really a bug. The class should
implement a __reduce__ function or otherwise hook into
the copy_reg system.
Under (2), this bug is just the tip of the iceberg.
There are about 8 other problems with the current
slot implementation that need to be resolved before
slots act almost identically to normal instance attributes.
Thankfully, I am fairly confident that I can supply
patches that can achieve this, though I am waiting for
Guido to comment on this issue when he returns from his
trip.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=520644&group_id=5470