python2.7 infinite recursion when loading pickled object

Dear all, I discovered a problem using cPickle.loads from CPython 2.7.6. The last line in the following code raises an infinite recursion class T(object): def __init__(self): self.item = list() def __getattr__(self, name): return getattr(self.item, name) import cPickle t = T() l = cPickle.dumps(t) cPickle.loads(l) loads triggers T.__getattr__ using "getattr(inst, "__setstate__", None)" for looking up a "__setstate__" method, which is not implemented for T. As the item attribute is missing at this time, the ininfite recursion starts. The infinite recursion disappears if I attach a default implementation for __setstate__ to T: def __setstate__(self, dd): self.__dict__ = dd This could be fixed by using „hasattr“ in pickle before trying to call „getattr“. Is this a bug or did I miss something ? Kind Regards, Uwe

On 8/11/2014 5:10 AM, Schmitt Uwe (ID SIS) wrote: Python usage questions should be directed to python-list, for instance.
I discovered a problem using cPickle.loads from CPython 2.7.6.
The problem is your code having infinite recursion. You only discovered it with pickle.
The last line in the following code raises an infinite recursion
class T(object):
def __init__(self): self.item = list()
def __getattr__(self, name): return getattr(self.item, name)
This is a (common) bug in your program. __getattr__ should call self.__dict__(name) to avoid the recursion. -- Terry Jan Reedy

Terry Reedy wrote:
On 8/11/2014 5:10 AM, Schmitt Uwe (ID SIS) wrote:
Python usage questions should be directed to python-list, for instance.
I discovered a problem using cPickle.loads from CPython 2.7.6.
The problem is your code having infinite recursion. You only discovered it with pickle.
The last line in the following code raises an infinite recursion
class T(object):
def __init__(self): self.item = list()
def __getattr__(self, name): return getattr(self.item, name)
This is a (common) bug in your program. __getattr__ should call self.__dict__(name) to avoid the recursion.
Read again. The OP tries to delegate attribute lookup to an (existing) attribute. IMO the root cause of the problem is that pickle looks up __dunder__ methods in the instance rather than the class.

On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__peter__@web.de> wrote:
Read again. The OP tries to delegate attribute lookup to an (existing) attribute.
IMO the root cause of the problem is that pickle looks up __dunder__ methods in the instance rather than the class.
The recursion comes from the attempted lookup of self.item, when __init__ hasn't been called. ChrisA

On Mon, 11 Aug 2014 21:43:00 +1000, Chris Angelico <rosuav@gmail.com> wrote:
On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__peter__@web.de> wrote:
Read again. The OP tries to delegate attribute lookup to an (existing) attribute.
IMO the root cause of the problem is that pickle looks up __dunder__ methods in the instance rather than the class.
The recursion comes from the attempted lookup of self.item, when __init__ hasn't been called.
Indeed, and this is what the OP missed. With a class like this, it is necessary to *make* it pickleable, since the pickle protocol doesn't call __init__. --David

Chris Angelico wrote:
On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__peter__@web.de> wrote:
Read again. The OP tries to delegate attribute lookup to an (existing) attribute.
IMO the root cause of the problem is that pickle looks up __dunder__ methods in the instance rather than the class.
The recursion comes from the attempted lookup of self.item, when __init__ hasn't been called.
You are right. Sorry for the confusion.

"Schmitt Uwe (ID SIS)" <uwe.schmitt@id.ethz.ch> writes:
I discovered a problem using cPickle.loads from CPython 2.7.6.
The last line in the following code raises an infinite recursion
class T(object):
def __init__(self): self.item = list()
def __getattr__(self, name): return getattr(self.item, name)
import cPickle
t = T()
l = cPickle.dumps(t) cPickle.loads(l) ... Is this a bug or did I miss something ?
The issue is that your __getattr__ raises RuntimeError (due to infinite recursion) for non-existing attributes instead of AttributeError. To fix it, you could use object.__getattribute__: class C: def __init__(self): self.item = [] def __getattr__(self, name): return getattr(object.__getattribute__(self, 'item'), name) There were issues in the past due to {get,has}attr silencing non-AttributeError exceptions; therefore it is good that pickle breaks when it gets RuntimeError instead of AttributeError. -- Akira
participants (6)
-
Akira Li
-
Chris Angelico
-
Peter Otten
-
R. David Murray
-
Schmitt Uwe (ID SIS)
-
Terry Reedy