How to use threading and Queue modules
"Martin v. Löwis"
martin at v.loewis.de
Tue Mar 4 18:10:30 CET 2003
Paul Moore wrote:
> But now I hit a problem: How do I wait for the workers to all finish?
> If I just join all the workers in turn, I risk the queue filling up,
> resulting in deadlock. If I wait on the queue, I don't know when all
> the workers have finished.
Is it the case that you have one result per worker? The I recommend that
each worker puts a pair (thread, thread-result) into the queue. The main
thread waits on the queue, picks up the result, and joins the worker thread.
If there are multiple results per worker, add a flag indicating that the
result you got is the last result.
> Maybe a queue isn't the right data structure here. Is there another,
> more suitable, data structure which can collect results from worker
> threads? Maybe all I want is a locked list. After all, the queue would
> be fine if it couldn't fill up. (There isn't enough data here for it
> to be a problem to keep it all in memory).
Ah, so you are concerned that the queue may fill up? Don't be: If you
don't give a maximum size for queue construction, the queue is unbounded.
More information about the Python-list