"RuntimeError: dictionary changed size during iteration" ; Good atomic copy operations?

[Raymond Hettinger]

[robert]
> In very rare cases a program crashes (hard to reproduce) :
>
> * several threads work on an object tree with dict's etc. in it. Items
> are added, deleted, iteration over .keys() ... ). The threads are "good"
> in such terms, that this core data structure is changed only by atomic
> operations, so that the data structure is always consistent regarding
> the application. Only the change-operations on the dicts and lists
> itself seem to cause problems on a Python level ..
>
> * one thread periodically pickle-dumps the tree to a file:
>    >>> cPickle.dump(obj, f)
>
> "RuntimeError: dictionary changed size during iteration" is raised by
> .dump ( or a similar "..list changed ..." )
>
> What can I do about this to get a stable pickle-dump without risiking
> execution error or even worse - errors in the pickled file ?

See if this fixes the problem for you:

    try:
        sys.setcheckinterval(sys.maxint)
        cPickle.dump(obj, f)                      # now runs atomically
    finally:
        sys.setcheckinterval(100)


Be careful where you use this technique.  In addition to suspending
other threads, it has the side-effect of suspending control-break
checks.  IOW, you won't be able to break out of the dump().



Raymond