[Python-ideas] Fwd: Concurrent safety?

[Mike Meyer]

On Wed, Nov 2, 2011 at 4:01 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/1/2011 7:49 PM, Mike Meyer wrote:
>> On Mon, Oct 31, 2011 at 11:55 PM, Terry
>> Reedy<tjreedy at udel.edu>  wrote:
>>> This is one of the hard problems that keep getting swept under the
>>> rug while we do easier things. Well, we have overhauled unicode and
>>> packaging for 3.3, so maybe concurrency can get some attention.
>> Hey, it worked!
> Yes. Coroutines, which are a form of in-thread concurrency, are another
> 'under the rug' subject, which Greg has re-exposed. We need some possibly
> initially off-the-wall ideas for both.

I've been avoiding pointing out that connection, but that's one of the
reasons I kept saying "thread of execution" instead of just "thread" -
at least until recently. There are more ways to get concurrency than
threading and processes.

>>> Would it be helpful, for instance, to have truly immutable
>>> restricted tuples and frozensets, whose __new__ methods only
>>> allowed true immutables (None, booleans, numbers, strings, other
>>> restricted tuples and frozensets) as members?
>> Possibly. However, so long as the mutations they make don't change
>> the externally visible behavior, then for the purposes of this
>> discussion, they already are immutable. Or is it possible that
>> concurrent updates of that not-externally-visible state could cause
>> things to break?
> It would seem to me that a list within a tuple or frozenset is just as
> 'dangerous' as a list that is exposed directly. Correspondingly, a safe_list
> that locked all appropriate operations should be equally safe within or
> without.

Right. However, What needs to made safe for concurrency here is the
list, not the tuple. As you point out, that doesn't help for things
like dict keys.

This discussion was more about objects that compute and then cache a
value based on their contents - hash values being one of them.

>>> How about a metaclass, say 'immutable', that made the instances of
>>> a user class truly immutable? (I don't know how to do this, but
>>> lets assume it can be done -- perhaps with a new frozendict.) If
>>> such were possible, instances of instances of such a metaclass
>>> could be added to the list above.
>>
>> Well, on the basis that we're all adults, I'm willing to accept that
>> a programmer saying "I want instances of this class to be immutable"
>> means they'll only subvert whatever mechanism is used to do this
>> when it's safe to do so (i.e. - "not externally visible"), so
>> catching casual attempts - assignments to attributes - to do so will
>> do, then we can do this by providing a __setattr__ method that always
>> throws an exception.
> I know about that, but there is no way for a safe_tuple class to know what a
> __setattr__ method does. But my new idea here is just to call hash(). That
> is not completely safe, but perhaps within 'consenting adults' territory.

Presumably, there'd be a better way to ask that question. Something
like "isinstance(object, Immutable)" where Immutable adds the
appropriate __setattr__ method (though that other problems).

> (In other words, if someone adds a __hash__ method to a class with mutable
> instances, puts such instances into a safe_tuple shared with multiple
> threads, mutates from multiple threads, and has a problem, ... too bad. And
> I hope you would never have to debug such ;-).

Doesn't sound much worse than most concurrency bugs to me. They tend
to crop up in places where it's "obvious" things aren't concurrent.
Those have replaced stray pointer bugs as the nastiest bugs to deal
with, at least for me.

    <mike