[Python-Dev] Weak Dictionary Iteration Behavior in Python 3

[Adam Olsen]

On Sat, Sep 13, 2008 at 2:20 PM, Armin Ronacher
<armin.ronacher at active-4.com> wrote:
> Hi everybody,
>
> In Python 2.x when iterating over a weak key dictionary for example, the common
> idom for doing that was calling dictionary.keys() to ensure that a list of all
> objects is returned it was safe to iterate over as a weak reference could stop
> existing during dict iteration which of course raises a runtime error by the
> dict iterator.
>
> This was documented behavior and worked pretty well, with the small problem that
> suddenly all references in the dict wouldn't die until iteration is over because
> the list holds references to the object.
>
> This no longer works in Python 3 because .keys() on the weak key dictionary
> returns a generator over the key view of the internal dict which of course has
> the same problem as iterkeys in Python 2.x.
>
> The following code shows the problem::
>
>    from weakref import WeakKeyDictionary
>
>    f1 = Foo()
>    f2 = Foo()
>    d = WeakKeyDictionary()
>    d[f1] = 42
>    d[f2] = 23
>
>    i = iter(d.keys()) # or use d.keyrefs() here which has the same problem
>    print(next(i))
>    del f2
>    print(next(i))
>
> This example essentially dies with "RuntimeError: dictionary changed
> size during iteration" as soon as f2 is deleted.
>
> Iterating over weak key dictionaries might not be the most common task but I
> know some situations where this is necessary.  Unfortunately I can't see a
> way to achieve that in Python 3.

IMO, this is a deeper problem than suggested.  As far as I know,
python does not (and should not) make promises as to when it'll
collect object.  We should expect weakrefs to be cleared at random
points, and code defensively.

One approach I've used before is to enqueue all the cleared weakrefs,
then process that queue when the WeakKeyDictionary is modified.


-- 
Adam Olsen, aka Rhamphoryncus