How to use threading and Queue modules

[Alex Martelli]

Paul Moore wrote:

> I have a problem trying to use the threading and Queue modules. What
> I'm trying to do is to fire off a host of background threads, and let
> them do some work. In my main thread, I want to wait for the
> background workers to complete, and then pick up their results.
> 
> The best way to handle the results seems to be via a Queue.Queue, so
> that I don't have to worry about locking issues.

Oh yes, definitely!

> But now I hit a problem: How do I wait for the workers to all finish?
> If I just join all the workers in turn, I risk the queue filling up,
> resulting in deadlock. If I wait on the queue, I don't know when all
> the workers have finished.

Simplest: have each worker post to the results Queue a distinguished
message saying "I'm finished" -- the main thread collects and processes
those to count the number of worker threads still outstanding, and
when that "number still outstanding" is 0, well, there you are!

> Maybe a queue isn't the right data structure here. Is there another,
> more suitable, data structure which can collect results from worker
> threads? Maybe all I want is a locked list. After all, the queue would
> be fine if it couldn't fill up. (There isn't enough data here for it
> to be a problem to keep it all in memory).

Queue.Queue is unbounded by default, except for memory limitations,
so I may not be understanding exactly what you mean.

> I'm new to multi-threading, so I don't want something too hard :-)

I think that either of the solutions -- not worrying about "filling
up" if memory limitation isn't an issue, counting threads still
outstanding by use of distinguished messages signifying thread
termination -- is pretty simple.


> Basically, the only reason I'm using threads at all is to multiplex a
> lot of database conversations (50 or more) where IO and network times
> are the bottleneck - with threading, I can run all 50 in the time it
> takes the slowest to connect!

Sounds like a good reason to me.


Alex