[Python-ideas] Fwd: Concurrent safety?

[Paul Moore]

On 1 November 2011 21:09, Mike Meyer <mwm at mired.org> wrote:
> On Tue, Nov 1, 2011 at 9:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> I don't know if you've considered this already, but a for-loop in
>> Python creates an iterator and then mutates it (by calling next()) on
>> each run through the loop. I can't see any way this could be a
>> concurrency problem in itself, but you'd likely need to either
>> reimplement the for loop to avoid relying on mutable iterators, or
>> you'd need to add some sort of exclusion for iterators in for loops.
>
> How about a third option? Iterators have to be locked to do a next in
> general, as they can be bound and thus shared between execution threads. On
> the other hand, locking & unlocking should be the major performance hit, so
> you don't want to do that on something that's going to be happening a lot,
> so the caller should be allowed to do something to indicate that it's not
> required. Locking the iterator should do that. So the next method needs to
> add a test to see if self is locked, and if not lock and then unlock self.

I'm not sure what you mean here. Suppose I have

l = [1,2,3]
for i in l:
  print(i)

Here, the thing you need to lock is not l, as it's not being mutated,
but the temporary iterator generated by the for loop. That's not
exposed to the user, so you can't lock it manually. Should it be
locked? It can never be seen from another thread. But how do you code
that exception to the rule?

What about

l = iter([1,2,3])
for i in l:
  print(i)

Here the for loop gnerates iter(l) - which, simply because of the
implementation of __iter__ for iterators, returns l. So should I lock
l here? It *is* exposed to other threads, potentially. How does the
compiler detect the difference between this and the previous example?

This seems to me to be a recipe for having users scatter arbitrary
locks around their code "just to shut the  interpreter up". It's not
at all clear that it helps people think, in that there's no easy
mental model people can acquire to help them reason about what is
going on. Just a load of exceptions that need to be silenced somehow.

Paul.