[Python-Dev] Issue 14417: consequences of new dict runtime error

[Guido van Rossum]

On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> Some of us have expressed uneasiness about the consequences of dict
> raising an error on lookup if the dict has been modified, the fix Victor
> made to solve one of the crashers.
>
> I don't know if I speak for the others, but (assuming that I understand
> the change correctly) my concern is that there is probably a significant
> amount of threading code out there that assumes that dict *lookup* is
> a thread-safe operation.  Much of that code will, if moved to Python
> 3.3, now be subject to random runtime errors for which it will not
> be prepared.  Further, code which appears safe can suddenly become
> unsafe if a refactoring of the code causes an object to be stored in
> the dictionary that has a Python equality method.

My original assessment was that this only affects dicts whose keys
have a user-implemented __hash__ or __eq__ implementation, and that
the number of apps that use this *and* assume the threadsafe property
would be pretty small. This is just intuition, I don't have hard
facts. But I do want to stress that not all dict lookups automatically
become thread-unsafe, only those that need to run user code as part of
the key lookup.

> Would it be possible to modify the fix so that the lookup is retried a
> non-trivial but finite number of times, so that normal code will work
> and only pathological code will break?

FWIW a similar approach was rejected as a fix for the hash DoS attack.

> I know that I really don't want to think about having to audit the
> (significantly threaded) application I'm currently working on to make sure
> it is "3.3 safe".  Dict lookup operations are *common*, and we've never
> had to think about whether or not they were thread-safe before (unless
> there were inter-thread synchronization issues involved, of course).
> Nor am I sure the locking dict type suggested by Jim on the issue would
> help, since a number of the dicts we are using are produced by library
> code.  So we'd have to wait for those libraries to be ported to 3.3....

Agreed that this is somewhat scary.

-- 
--Guido van Rossum (python.org/~guido)