[ python-Bugs-931877 ] Segfault in object_reduce_ex

[SourceForge.net]

Bugs item #931877, was opened at 2004-04-08 19:46
Message generated for change (Comment added) made by zseil
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=931877&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Tim Peters (tim_one)
Assigned to: Nobody/Anonymous (nobody)
Summary: Segfault in object_reduce_ex

Initial Comment:
Shane Hathaway bumped into this, unbounded recursion 
in typeobject.c's object_reduce_ex().  This occurs in 
Python 2.3.3 and current CVS.

Assigned to Guido, to ponder whether object_reduce_ex 
is doing what it should; if it is (which seems likely to 
me), I suppose we need to inject a recursion counter to 
prevent the segfault.

The failing case is short, but I'll attach it (temp99.py) to 
avoid SF line mangling.  While the test uses pickle, same 
symptom if it's changed to use cPickle instead.

Jim Fulton's analysis:

"""
This is a very clever infinite loop.  The proxy doesn't
actually proxy, but it does manage to confuse reduce 
about what's going on.

reduce tries to figure out if it has been overridden by
asking whether the class's reduce is the same as
object.__reduce__. It doesn't expect to be lied to
about the class.  Things wouldn't have been so bad if
the proxy had proxied __reduce__ as well as __class__.
"""

The priority hasn't been bumped, because "the real 
code" from which this was whittled down wasn't doing 
what it needed to do anyway, and the recursion went 
away when the real code was repaired.

----------------------------------------------------------------------

Comment By: Žiga Seilnacht (zseil)
Date: 2006-04-01 03:43

Message:
Logged In: YES 
user_id=1326842


See patch #1462488. If that patch is accepted, this bug
should be closed as fixed.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2006-03-15 06:05

Message:
Logged In: YES 
user_id=6380

Unassigning. I need to concentrate on Python 3000.

----------------------------------------------------------------------

Comment By: Georg Brandl (birkenfeld)
Date: 2006-01-10 22:58

Message:
Logged In: YES 
user_id=1188172

Still crashing with 2.5...

----------------------------------------------------------------------

Comment By: Nathan Srebro (nati)
Date: 2004-11-30 02:26

Message:
Logged In: YES 
user_id=63133

This infinite recursion also occurs in another place, that
got me stumped for a couple of days when old code that
worked with Python 2.2 stopped working.  If __class__ is not
fidgeted with (as in original bug report), but a descriptor
returns a custom reduce for the class, but not for its
objects, reduce enters an infinite loop on the object:

"""
class descriptor_for_reduce(object):
    def __get__(self,obj,tp=None):
        if obj is not None: return
super(ASpecialClass,obj).__reduce__
        return self.reducer
    def reducer(self,proto=None):
        return "VerySpecial"

class ASpecialClass(object):
    __reduce__ = descriptor_for_reduce()

copy.copy(ASpecialClass())
"""

ASpecialClass().__reduce__ is object.__reduce__, which is
implemented by typeobject.c:object_reduce_ex.  This function
(that doesn't know if its called as the __reduce__ or the
__reduce_ex__ method) tries to detect if the object's
__reduce__ is overridden.  It does so by checking if the
object's class's __reduce__ is overridden, and in fact it
is.  It then assumes that the object's __reduce__ is
overridden, and calls it.  But the object's __reduce__ is
the same function, causing the infinite loop.

If __reduce_ex__ is used instead of __reduce__, the problem
goes away, ASpecialClass().__reduce_ex__() return the usual
tuple, and ASpecialClass.__reduce_ex__() return
"VerySpecial".  But when __reduce__ is overridden,
ASpecialClass().__reduce__() enters an infinite loop.

I believe this is a legitimate example that should behave
just as when __reduce_ex__ is overridden.  The example
doesn't lie about __class__, and it is certainly legitimate
for define a property that behaves differently for the class
and for its objects.

Where did this come up and why would I ever care about a
class's __reduce__?  The __reduce__ attribute of a class is
never used by (the standard) pickle or copy, since
save_global() is called instead. However, I have a custom
pickler, implemented as a subclass of pickle.Pickler, which
falls back on the class's __reduce__ when save_global()
fails.  This way, I can pickle certain classes that are
created at run-time (and can be easily recreated, e.g. from
their bases and dictionaries).


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2004-04-08 19:51

Message:
Logged In: YES 
user_id=31435

Hmm!  The temp99.py download link doesn't work for me.  
Here's the content in case it doesn't work for others either:

"""
import cPickle as pickle
from pickle import dumps

class SimpleItem:
    def __reduce__(self):
        return (self.__class__, None, {})

class Proxy(object):
    __class__ =  property(lambda self: self.__target.__class__)

    def __init__(self, target):
        self.__target = target

    def __getstate__(self):
        raise RuntimeError("No don't pickle me!  Aaarrgghh!")

p = Proxy(SimpleItem())
dumps(p)
"""

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=931877&group_id=5470