[Tutor] Using Queue

Kent Johnson kent37 at tds.net
Wed Apr 23 19:59:51 CEST 2008

On Wed, Apr 23, 2008 at 1:17 PM, bob gailer <bgailer at gmail.com> wrote:

> "The Queue module implements a multi-producer, multi-consumer FIFO queue.
> I understand producer, comsumer, FIFO.
> I don't understand multi-

More than one producer and/or consumer

> "It is especially useful in threads programming when information must be
> exchanged safely between multiple threads. "
> I understand threads. I've written some (to me fairly sophisticated)
> programs using Threading and conditions.
> I understand that threads might want to exchange information.
> I guess that queue supports the exchange by receiving and releasing items.
> Is that true?

Yes. Producers put things in the queue, consumers pull them out. Basically a
FIFO where the two ends are used by different threads.

> I don't know what "safely" means.
> "The Queue class in this module implements all the required locking
> semantics." I have no idea what that means nor does any of the ensuing
> documentation explain.

Ok, do you understand thread safety, or conversely, what kind of problems
can happen when two threads share data without any kind of locking?

For example, suppose you want a queue with a limited number of items allowed
in it, and producers should block if the queue is full. You need an atomic
"check for full and push" operation; otherwise you can have race conditions
between two threads. If the queue has one empty slot, you have to protect
- thread A checks for queue full - not full
- thread B checks for queue full - not full
- thread A pushes - queue is now full
- thread B pushes - overflow

queue.Queue uses locks to prevent this and other problems.

> Do you have a specific use in mind?

I have an application that uses Threading. It is not a producer, consumer
> application, just a bunch of threads that are started at the same time. And
> they do not communicate with each other, just with the main thread. But that
> seems to be true of Queue also.

Possibly you could use a Queue here. The main thread would be the only
consumer; the rest of the threads would be producers.

>  Queue is used to facilitate communication between threads, so any Queue
> > example includes multiple threads.
> >
> >
> >  Regarding the sample code:
> I must add (by trial and error) at least these 3 lines to the main
> program:
> from threading import *
> from Queue import Queue
> num_worker_threads = 3
> Then it bombs:
> File "J:\python\queue.py", line 17, in <module>
>   for item in source():
> NameError: name 'source' is not defined
> No idea what source is supposed to be. A callable object? And where in
> this example are the producer threads?

Yes, the example is really just a sketch. source is a callable object that
returns an iterable object whose items represent some kind of unit of work.

The threads created by calling Thread(target=worker) are the consumer
threads. The main thread is the producer - it  puts items into the queue in
the for loop.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20080423/c809cbd1/attachment.htm>

More information about the Tutor mailing list