Multiprocessing.Queue deadlock

Felix schlesin at
Wed Oct 7 04:15:25 CEST 2009


I keep running into a deadlock in a fairly simple parallel script
using Multiprocessing.Queue for sending tasks and receiving results.
>From the documentation I cannot figure out what is happening and none
of the examples seem to cover quite what I am doing. The main code is

results = mp.Queue()
tasks = mp.JoinableQueue()
tasks.put( (0,0) )
procs = [ mp.Process(target=work, args=(tasks, results)) for i in range
for p in procs:
    p.daemon = True

for i in range(nprocs): tasks.put('STOP')
for p in procs: p.join()
while 1:
    except Empty: break

The function 'work' both consumes tasks adding the results to the
output queue and adds new tasks to the input queue based on its

def work(tasks, results):
    for task in iter(tasks.get, 'STOP'):
        res = calc(*task)
        if res:
	    tasks.put((task[0], res[1]))

This program will hang while the main process joins the workers (after
all results are computed, i.e. after tasks.join() ). The workers have
finished function 'work', but have not terminated yet.

Calling results.cancel_join_thread as a last line in 'work' prevents
the deadlocks, as does terminating the workers directly. However I am
not sure why that would be needed and if it might not make me loose

It seems to be the workers cannot finish pusing buffered results into
the output queue when calling 'results.join_thread' while terminating,
but why is that? I tried calling 'results.close()' before joining the
workers in the main process, but it does not make a difference.

Is there something I am understanding wrong about the interface? Is
there a much better way to do what I am trying to do above?


More information about the Python-list mailing list