Multiprocessing.Queue - I want to end.

Cameron Simpson cs at
Fri May 1 01:22:26 EDT 2009

On 01May2009 08:37, I wrote:
| On 30Apr2009 22:57, MRAB <google at> wrote:
| > The producer could send just one None to indicate that it has finished
| > producing.
| > Each consumer could get the data from the queue, but if it's None then
| > put it back in the queue for the other consumer, then clean up and
| > finish.
| > When all the consumers have finished, the queue will contain just the
| > single None.
| And as it happens I have an IterableQueue class right here which does
| _exact_ what was just described. You're welcome to it if you like.
| Added bonus is that, as the name suggests, you can use the class as
| an iterator:
|   for item in iterq:
|     ...
| The producer calls iterq.close() when it's done.

Someone asked, so code appended below.

Cameron Simpson <cs at> DoD#743

class IterableQueue(Queue): 
  ''' A Queue obeying the iterator protocol.
      Note: Iteration stops when a None comes off the Queue.

  def __init__(self, *args, **kw):
    ''' Initialise the queue.
    Queue.__init__(self, *args, **kw)

  def put(self, item, *args, **kw):
    ''' Put an item on the queue.
    assert not self.__closed, "put() on closed IterableQueue"
    assert item is not None, "put(None) on IterableQueue"
    return Queue.put(self, item, *args, **kw)

  def _closeAtExit(self):
    if not self.__closed:

  def close(self):
    if self.__closed:
      # this should be a real log message
      print >>sys.stderr, "close() on closed IterableQueue"

  def __iter__(self):
    ''' Iterable interface for the queue.
    return self

  def next(self):
    if item is None:
      Queue.put(self,None)      # for another iterator
      raise StopIteration
    return item

More information about the Python-list mailing list