[New-bugs-announce] [issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

New submission from Brian <brian at merrells.org>:

Despite carefully matching my get() and task_done() statements I would 
often trigger "raise ValueError('task_done() called too many times')" in 
my multiprocessing.JoinableQueue (multiprocessing/queues.py)

Looking over the code (and a lot of debug logging), it appears that the 
issue arises from JoinableQueue.put() not being protected with a locking 
mechanism.  A preemption after the first line allows other processes to 
resume without releasing the _unfinished_tasks semaphore.

The simplest solution seems to be allowing task_done() to block while 
waiting to acquire the _unfinished_tasks semaphore.

Replacing:
if not self._unfinished_tasks.acquire(False):
  raise ValueError('task_done() called too many times')

With simply:
self._unfinished_tasks.acquire()

This would however remove the error checking provided (given the many 
far more subtler error that can be made, I might argue it is of limited 
value).  Alternately the JoinableQueue.put() method could be better 
protected.

----------
components: Library (Lib)
messages: 77806
nosy: merrellb
severity: normal
status: open
title: multiprocessing.JoinableQueue task_done() issue
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue4660>
_______________________________________