[issue9269] Cannot pickle self-referencing sets

[Alexander Belopolsky]

New submission from Alexander Belopolsky <belopolsky at users.sourceforge.net>:

Attached script, cycle.py demonstrates a simplification of the behavior reported by mike bayer in msg77200.

Essentially, the script attempts to pickle a set that contains a class instance that has an attribute referring back to the set:

class C:
    pass
c = C()
cycle = set([c])
c.foo = cycle

An attempt to pickle the *cycle* object triggers an assertion error in 2.7 or in 3.2 with disabled _pickle acceleration and produces a broken cycle in 3.2 or if cPickle is used instead of pickle in 2.7.

$ python3 cycle.py
FAIILURE
..

$ python2 cycle.py
Traceback (most recent call last):
  ..
  File ".../pickle.py", line 244, in memoize
    assert id(obj) not in self.memo
AssertionError

If you run cycle.py with an argument, it uses a dict instead of set to create the cycle and shows that the cycles with dict can be pickled correctly:

$ python3 cycle.py dict
SUCCESS
..

After reporting success or failure, cycle.py, prints a disassembly of the pickle stream which makes it clear what happens:

In case of dict, we see the following:

$ python3 cycle.py dict
SUCCESS
 ..
    2: }    EMPTY_DICT
    3: q    BINPUT     0
 ..
   26: X    BINUNICODE 'foo'
..
   36: h    BINGET     0
   38: s    SETITEM
..
   40: N    NONE
   41: s    SETITEM

An empty dict is created and saved in the memo.  Then a C object is built with foo attribute is set to the dict retrieved from the memo. Finally, the same dict is updated with (C object, None) key-value pair.  The result is the cycle identical to the one we built in python code.

The sets, however, are built differently.  There is no pickle opcode to add items to a set, so all set items must exist by the time set is built.  So here is what we see:  

$ python3 cycle.py
FAIILURE
    2: c    GLOBAL     'builtins set'
   16: q    BINPUT     0
.. instead of empty set the constructor is saved in memo
   42: X    BINUNICODE 'foo'
   52: h    BINGET     0
..
   63: R    REDUCE
.. a set object containing c is constructed 
   66: s    SETITEM  
.. and assigned to c.foo
   72: R    REDUCE
.. another set object is constructed containing c
 
As a result, we have

cycle = {c}
c.foo = {c}

Instead of 

c.foo = cycle

----------
assignee: belopolsky
components: Interpreter Core
files: cycle.py
messages: 110397
nosy: belopolsky
priority: normal
severity: normal
stage: needs patch
status: open
title: Cannot pickle self-referencing sets
type: behavior
versions: Python 2.7, Python 3.2
Added file: http://bugs.python.org/file18020/cycle.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9269>
_______________________________________