[Python-Dev] Another threading idea

[Tim Peters]

[Raymond Hettinger]
> FWIW, I've been working on a way to simplify the use of queues with
> daemon consumer threads
>
> Sometimes, I launch one or more consumer threads that wait for a task
> to enter a queue and then work on the task. A recurring problem is that
> I sometimes need to know if all of the tasks have been completed so I
> can exit or do something with the result.

FWIW, instead of:

   # main thread
   q = TaskQueue()
   for t in worker_threads():
       t.start()
   for task in tasklist:
       q.put(task)                  # increments the counter and enqueues a task
   q.join()                         # all of the tasks are done
(counter is zero)
   do_work_on_results()

I've sometimes used a separate "work finished" queue, like so:

   # main thread
   q = TaskQueue()
   finished = Queue.Queue()
   for t in worker_threads():
       t.start()
   for task in tasklist:
       q.put(task)
   for task in tasklist:
       finished.get()
   do_work_on_results()

When a worker thread is done with a task, it simply does:

   finished.put(None)  # `None` can just as well be 42 or "done"

No explicit count is needed, although it's easy to add one if desired.
 The only trick is for the main thread to insist on doing
finished.get() exactly as many times as it does q.put().

This is easy and natural -- once you've seen it :-)  BTW, it's more
common for me to put something meaningful on the `finished` queue, as
my main thread often wants to accumulate some info about the outcomes
of tasks.