threads or queue for this task

[Mark Hammond]

robin at execulink.com wrote:
> I am missing something I think with the difference between the
> threading and Queue modules. These could really do with some updated
> docs with examples.
> 
> I wish to have a server continually cycle (until interrupted by
> keyboard, signal, or whatever). It needs to perform several (say 6)
> different tasks, each of which may take significantly different
> lengths of time to perform.
> 
> My first implementation used threading.py. I wrote a dispatcher to
> start each of the threads, and then wait until the thread count was
> back down to 1. This I put in a while loop, and surrounded it with the
> necessary exception code.
> 
> Each cycle through the while has to wait for the slowest process to
> finish before completing. During this time, the other processes just
> sit around doing nothing. What I really want is for each process to
> start up again when they are done, so that there are no more than 6
> going at once. In other words, I want to be sure I don't start a new
> process 3 before the old process 3 is done.
> 
> I have looked at Queue but it doesn't seem to have enough control to
> make this happen.
> 
> I am sure I am overlooking something obvious. Thanks ahead of time for
> any help!

It is common for threading code to look somewhat like:

threads = []
for i in range(number_of_tasks):
   thread.append(thread_for_task)
# let the threads run - just wait for shutdown
for t in threads:
   t.join()

Each thread itself contains a loop which continually look for more work 
to do, or exit.  The queue module is useful for giving threads work to 
do - some other thread can queue new things, and the worker thread 
removes them, blocking while nothing is in the queue.

Look for Aahz's threading tutorial - URL not at hand, but it can't be 
too hard to find.  Indeed, I predict he will tell you exactly where it is ;)

Hope this helps,

Mark.