schlesin at cshl.edu
Wed Oct 7 04:15:25 CEST 2009
I keep running into a deadlock in a fairly simple parallel script
using Multiprocessing.Queue for sending tasks and receiving results.
>From the documentation I cannot figure out what is happening and none
of the examples seem to cover quite what I am doing. The main code is
results = mp.Queue()
tasks = mp.JoinableQueue()
tasks.put( (0,0) )
procs = [ mp.Process(target=work, args=(tasks, results)) for i in range
for p in procs:
p.daemon = True
for i in range(nprocs): tasks.put('STOP')
for p in procs: p.join()
except Empty: break
The function 'work' both consumes tasks adding the results to the
output queue and adds new tasks to the input queue based on its
def work(tasks, results):
for task in iter(tasks.get, 'STOP'):
res = calc(*task)
This program will hang while the main process joins the workers (after
all results are computed, i.e. after tasks.join() ). The workers have
finished function 'work', but have not terminated yet.
Calling results.cancel_join_thread as a last line in 'work' prevents
the deadlocks, as does terminating the workers directly. However I am
not sure why that would be needed and if it might not make me loose
It seems to be the workers cannot finish pusing buffered results into
the output queue when calling 'results.join_thread' while terminating,
but why is that? I tried calling 'results.close()' before joining the
workers in the main process, but it does not make a difference.
Is there something I am understanding wrong about the interface? Is
there a much better way to do what I am trying to do above?
More information about the Python-list