[Python-Dev] Issue 14417: consequences of new dict runtime error

[Guido van Rossum]

Etienne, I have not understood either of your messages in this thread. They
just did not make sense to me. Do you actually understand the issue at hand?

--Guido

On Friday, March 30, 2012, Etienne Robillard wrote:

> On 03/29/2012 06:07 PM, R. David Murray wrote:
>
>> On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel<stefan_ml at behnel.de>
>>  wrote:
>>
>>> R. David Murray, 29.03.2012 22:31:
>>>
>>>> On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote:
>>>>
>>>>> On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote:
>>>>>
>>>>>> Some of us have expressed uneasiness about the consequences of dict
>>>>>> raising an error on lookup if the dict has been modified, the fix
>>>>>> Victor
>>>>>> made to solve one of the crashers.
>>>>>>
>>>>>> I don't know if I speak for the others, but (assuming that I
>>>>>> understand
>>>>>> the change correctly) my concern is that there is probably a
>>>>>> significant
>>>>>> amount of threading code out there that assumes that dict *lookup* is
>>>>>> a thread-safe operation.  Much of that code will, if moved to Python
>>>>>> 3.3, now be subject to random runtime errors for which it will not
>>>>>> be prepared.  Further, code which appears safe can suddenly become
>>>>>> unsafe if a refactoring of the code causes an object to be stored in
>>>>>> the dictionary that has a Python equality method.
>>>>>>
>>>>>
>>>>> My original assessment was that this only affects dicts whose keys
>>>>> have a user-implemented __hash__ or __eq__ implementation, and that
>>>>> the number of apps that use this *and* assume the threadsafe property
>>>>> would be pretty small. This is just intuition, I don't have hard
>>>>> facts. But I do want to stress that not all dict lookups automatically
>>>>> become thread-unsafe, only those that need to run user code as part of
>>>>> the key lookup.
>>>>>
>>>>
>>>> You are probably correct, but the thing is that one still has to do the
>>>> code audit to be sure...and then make sure that no one later introduces
>>>> such an object type as a dict key.
>>>>
>>>
>>> The thing is: the assumption that arbitrary dict lookups are GIL-atomic
>>> has
>>> *always* been false. Only those that do not involve Python code execution
>>> for the hash key calculation or the object comparison are. That includes
>>> the built-in strings and numbers (and tuples of them), which are by far
>>> the
>>> most common dict keys. Looking up arbitrary user provided objects is
>>> definitely not guaranteed to be atomic.
>>>
>>
>> Well, I'm afraid I was using the term 'thread safety' rather too loosely
>> there.  What I mean is that if you do a dict lookup, the lookup either
>> returns a value or a KeyError, and that if you get back an object that
>> object has internally consistent state.  The problem this fix introduces
>> is that the lookup may fail with a RuntimeError rather than a KeyError,
>> which it has never done before.
>>
>> I think that is what Guido means by code that uses objects with python
>> eq/hash *and* assumes threadsafe lookup.  If mutation of the objects
>> or dict during the lookup is a concern, then the code would use locks
>> and wouldn't have the problem.  But there are certainly situations
>> where it doesn't matter if the dictionary mutates during the lookup,
>> as long as you get either an object or a KeyError, and thus no locks are
>> (currently) needed.
>>
>> Maybe I'm being paranoid about breakage here, but as with most backward
>> compatibility concerns, there are probably more bits of code that will
>> be affected than our intuition indicates.
>>
>> --David
>> ______________________________**_________________
>>
>
> what this suppose to mean exactly? To "mutate" is a bit odd concept for a
> programming language I suppose. Also I suppose I must be missing something
> which makes you feel like this is an OT post when the problem seem most
> likely to be exclusively in python 3.3, another reason I guess to not
> upgrade yet all that massively using 2to3. :-)
>
> cheers,
> Etienne
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120330/70f3d136/attachment.html>