matt at sidefx.com
Tue Mar 2 21:59:15 CET 2010
I'm having a problem with the multiprocessing package.
I'm trying to use a simple pattern where a supervisor object starts a
bunch of worker processes, instantiating them with two queues (a job
queue for tasks to complete and an results queue for the results). The
supervisor puts all the jobs in the "job" queue, then join()s the
workers, and then pulls all the completed results off the "results" queue.
(I don't think I can just use something like Pool.imap_unordered for
this because the workers need to be objects with state.)
Here's a simplified example:
The problem is that seemingly randomly, but almost always, the worker
processes will deadlock at some point and stop working before they
complete. This will leave the whole program stalled forever. This seems
more likely the more work each worker does (to the point where adding
the time.sleep(0.01) as seen in the example code above guarantees it).
The problem seems to occur on both Windows and Mac OS X.
I've tried many random variations of the code (e.g. using JoinableQueue,
calling cancel_join_thread() on one or both queues even though I have no
idea what it does, etc.) but keep having the problem.
Am I just using multiprocessing wrong? Is this a bug? Any advice?
More information about the Python-list