multiprocessing deadlock

Brian Quinlan brian at
Fri Oct 23 04:18:32 CEST 2009

My test reduction:

import multiprocessing
import queue

def _process_worker(q):
     while True:
             something = q.get(block=True, timeout=0.1)
         except queue.Empty:
             print('Grabbed item from queue:', something)

def _make_some_processes(q):
     processes = []
     for _ in range(10):
         p = multiprocessing.Process(target=_process_worker, args=(q,))
     return processes

def _do(i):
     print('Run:', i)
     q = multiprocessing.Queue()
     for j in range(30):
     processes = _make_some_processes(q)

     while not q.empty():

#    The deadlock only occurs on Mac OS X and only when these lines
#    are commented out:
#    for p in processes:
#        p.join()

for i in range(100):


Output (on Mac OS X using the svn version of py3k):
% ~/bin/python3.2
Run: 0
Grabbed item from queue: 0
Grabbed item from queue: 1
Grabbed item from queue: 2
Grabbed item from queue: 29
Run: 1

At this point the script produces no additional output. If I uncomment 
the lines above then the script produces the expected output. I don't 
see any docs that would explain this problem and I don't know what the 
rule would be e.g. you just join every process that uses a queue before 
  the queue is garbage collected.

Any ideas why this is happening?


More information about the Python-list mailing list