[New-bugs-announce] [issue9269] Cannot pickle self-referencing sets

Alexander Belopolsky report at bugs.python.org
Fri Jul 16 01:01:42 CEST 2010

New submission from Alexander Belopolsky <belopolsky at users.sourceforge.net>:

Attached script, cycle.py demonstrates a simplification of the behavior reported by mike bayer in msg77200.

Essentially, the script attempts to pickle a set that contains a class instance that has an attribute referring back to the set:

class C:
c = C()
cycle = set([c])
c.foo = cycle

An attempt to pickle the *cycle* object triggers an assertion error in 2.7 or in 3.2 with disabled _pickle acceleration and produces a broken cycle in 3.2 or if cPickle is used instead of pickle in 2.7.

$ python3 cycle.py

$ python2 cycle.py
Traceback (most recent call last):
  File ".../pickle.py", line 244, in memoize
    assert id(obj) not in self.memo

If you run cycle.py with an argument, it uses a dict instead of set to create the cycle and shows that the cycles with dict can be pickled correctly:

$ python3 cycle.py dict

After reporting success or failure, cycle.py, prints a disassembly of the pickle stream which makes it clear what happens:

In case of dict, we see the following:

$ python3 cycle.py dict
    2: }    EMPTY_DICT
    3: q    BINPUT     0
   26: X    BINUNICODE 'foo'
   36: h    BINGET     0
   38: s    SETITEM
   40: N    NONE
   41: s    SETITEM

An empty dict is created and saved in the memo.  Then a C object is built with foo attribute is set to the dict retrieved from the memo. Finally, the same dict is updated with (C object, None) key-value pair.  The result is the cycle identical to the one we built in python code.

The sets, however, are built differently.  There is no pickle opcode to add items to a set, so all set items must exist by the time set is built.  So here is what we see:  

$ python3 cycle.py
    2: c    GLOBAL     'builtins set'
   16: q    BINPUT     0
.. instead of empty set the constructor is saved in memo
   42: X    BINUNICODE 'foo'
   52: h    BINGET     0
   63: R    REDUCE
.. a set object containing c is constructed 
   66: s    SETITEM  
.. and assigned to c.foo
   72: R    REDUCE
.. another set object is constructed containing c
As a result, we have

cycle = {c}
c.foo = {c}

Instead of 

c.foo = cycle

assignee: belopolsky
components: Interpreter Core
files: cycle.py
messages: 110397
nosy: belopolsky
priority: normal
severity: normal
stage: needs patch
status: open
title: Cannot pickle self-referencing sets
type: behavior
versions: Python 2.7, Python 3.2
Added file: http://bugs.python.org/file18020/cycle.py

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list