[Python-ideas] Fwd: Concurrent safety?

[Mike Meyer]

On Wed, Nov 2, 2011 at 2:27 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 1 November 2011 21:09, Mike Meyer <mwm at mired.org> wrote:
>> On Tue, Nov 1, 2011 at 9:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> I don't know if you've considered this already, but a for-loop in
>>> Python creates an iterator and then mutates it (by calling next()) on
>>> each run through the loop. I can't see any way this could be a
>>> concurrency problem in itself, but you'd likely need to either
>>> reimplement the for loop to avoid relying on mutable iterators, or
>>> you'd need to add some sort of exclusion for iterators in for loops.
>>
>> How about a third option? Iterators have to be locked to do a next in
>> general, as they can be bound and thus shared between execution threads. On
>> the other hand, locking & unlocking should be the major performance hit, so
>> you don't want to do that on something that's going to be happening a lot,
>> so the caller should be allowed to do something to indicate that it's not
>> required. Locking the iterator should do that. So the next method needs to
>> add a test to see if self is locked, and if not lock and then unlock self.
>
> I'm not sure what you mean here. Suppose I have
>
> l = [1,2,3]
> for i in l:
>  print(i)
>
> Here, the thing you need to lock is not l, as it's not being mutated,
> but the temporary iterator generated by the for loop. That's not
> exposed to the user, so you can't lock it manually. Should it be
> locked? It can never be seen from another thread. But how do you code
> that exception to the rule?

You don't have to do anything. Iterators need to lock themselves to be
safe in concurrent use, so this will work fine, with the temporary
iterator doing whatever locking is needed.

> What about
>
> l = iter([1,2,3])
> for i in l:
>  print(i)

This will work the same as the above. However, since you have the
iterator in hand, you have the option to lock it before entering the
for loop, which will cause it to *not* do it's own internal locking.

This brings up an interesting point, though - look for mail with the
subject "Concurrency survey..."

    <mike