[Python-Dev] Another threading idea

[Guido van Rossum]

Isn't this a job for threading.BoundedSpemaphore()?

On 3/14/06, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
> FWIW, I've been working on a way to simplify the use of queues with daemon
> consumer threads
>
> Sometimes, I launch one or more consumer threads that wait for a task to enter a
> queue and then work on the task. A recurring problem is that I sometimes need to
> know if all of the tasks have been completed so I can exit or do something with
> the result.
>
> If each thread only does a single task, I can use t.join() to wait until the
> task is done.  However, if the thread stays alive and waits for more Queue
> entries, then there doesn't seem to be a good way to tell when all the
> processing is done.
>
> So, the idea is to create a subclass of Queue that increments a counter when
> objects are enqueued, that provides a method for worker threads to decrement the
> counter when the work is done, and offers a blocking join() method that waits
> until the counter is zero
>
>    # main thread
>    q = TaskQueue()
>    for t in worker_threads():
>        t.start()
>    for task in tasklist:
>        q.put(task)                  # increments the counter and enqueues a task
>    q.join()                         # all of the tasks are done (counter is
> zero)
>    do_work_on_results()
>
>
>
>    # worker thread
>    while 1:
>          task = q.get()             # task is popped but the counter is
> unchanged
>          do_work(task)
>          q.decrement()              # now the counter gets reduced
>
>
> The idea is still in its infancy (no implementation and it hasn't been tried in
> real-world code) but I would like to get some feedback.  If it works out, I'll
> post a recipe to ASPN and see how it goes.
>
>
> Raymond
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)