[Python-Dev] Another threading idea

[Raymond Hettinger]

FWIW, I've been working on a way to simplify the use of queues with daemon 
consumer threads

Sometimes, I launch one or more consumer threads that wait for a task to enter a 
queue and then work on the task. A recurring problem is that I sometimes need to 
know if all of the tasks have been completed so I can exit or do something with 
the result.

If each thread only does a single task, I can use t.join() to wait until the 
task is done.  However, if the thread stays alive and waits for more Queue 
entries, then there doesn't seem to be a good way to tell when all the 
processing is done.

So, the idea is to create a subclass of Queue that increments a counter when 
objects are enqueued, that provides a method for worker threads to decrement the 
counter when the work is done, and offers a blocking join() method that waits 
until the counter is zero

   # main thread
   q = TaskQueue()
   for t in worker_threads():
       t.start()
   for task in tasklist:
       q.put(task)                  # increments the counter and enqueues a task
   q.join()                         # all of the tasks are done (counter is 
zero)
   do_work_on_results()



   # worker thread
   while 1:
         task = q.get()             # task is popped but the counter is 
unchanged
         do_work(task)
         q.decrement()              # now the counter gets reduced


The idea is still in its infancy (no implementation and it hasn't been tried in 
real-world code) but I would like to get some feedback.  If it works out, I'll 
post a recipe to ASPN and see how it goes.


Raymond