[Python-Dev] Another threading idea

[Dave Brueck]

Raymond Hettinger wrote:
> FWIW, I've been working on a way to simplify the use of queues with daemon 
> consumer threads
> 
> Sometimes, I launch one or more consumer threads that wait for a task to enter a 
> queue and then work on the task. A recurring problem is that I sometimes need to 
> know if all of the tasks have been completed so I can exit or do something with 
> the result.
> 
> If each thread only does a single task, I can use t.join() to wait until the 
> task is done.  However, if the thread stays alive and waits for more Queue 
> entries, then there doesn't seem to be a good way to tell when all the 
> processing is done.

The pattern I use is to use the None object to indicate that no more tasks are 
coming - just add it to your queue once for every worker thread that exists and 
have your worker threads exit when they get a task of None.

Your setup code to start the threads and add tasks to the queue wouldn't need to 
change; once all the tasks have been enqueued you'd wait for completion like this:

     # Tell all the workers to quit
     for t in worker_threads():
         q.put(None)

     # Wait for all the workers to quit
     for t in worker_threads():
         t.join()

     do_work_on_results()

The worker thread looks like this:

while 1:
     task = q.get()
     if task is None:
         break
     do_work(task)

-Dave