Accessing a shared generator from multiple threads.

Aahz aahz at
Thu Jan 22 02:01:11 CET 2004

In article <400EE6B5.27C6B694 at>,
Alan Kennedy  <alanmk at> wrote:
>Yes, you're right. Using a Queue in this situation does require the
>use of a dedicated thread for the producer. There is no way to "pull"
>values from a generator to multiple consumers through a Queue.Queue.
>The values have to be "pushed" onto the Queue.Queue by some producing
>thread of execution.


>The way I see it, the options are  
>Option 1. Spawn a separate thread to execute the producing generator.
>However, this has problems:-
>A: How do the threads recognise the end of the generated sequence?
>This is not a simple problem: the Queue simply being empty does not
>necessarily signify the end of the sequence (e.g., the producer thread
>might not be getting its fair share of CPU time).
>B: The Queue acts as a (potentially infinite) buffer for the generated
>values, thus eliminating one of the primary benefits of generators:
>their efficient "generate when required" nature. This can be helped
>somewhat by limiting the number of entries in the Queue, but it is
>still slightly unsatisfactory.
>C: A thread of execution has to be dedicated to the producer, thus
>consuming resources.

There are a number of ways of mitigating A and B; they mostly involve
using an extra Queue.Queue to send tokens to the generator thread when a
consumer wants data.  The generator thread then sends back a token that
(among other things) contains an attribute specifically for notifying
the consumer that the generator is exhausted.  See
for examples that show the technique, though they're not directly
relevant to this case.

My point is that I haven't (yet) seen many good use cases for sharing a
generator between threads, and I'm guessing that many people will try
using generators inappropriately for problems that really are better
suited to Queue.Queue.
Aahz (aahz at           <*>

A: No.
Q: Is top-posting okay?

More information about the Python-list mailing list