[Python-ideas] `__iter__` for queues?

[Nick Coghlan]

MRAB wrote:
> 'Empty' could mean either a non-blocking .get and stopping when there
> are no more items in the queue, or a blocking .get and stopping when
> there are no more items in the queue _and_ the queue has been closed by
> the producer(s).

When there are multiple valid approaches to iterating over a queue, and
none of them are more obviously fundamental than any of the others, does
it make sense for us to bless one by implementing it as the __iter__ method?

Now, an iteration API (separate from __iter__) that helped developers
easily avoid some of the pitfalls of rolling your own iteration
techniques (such as the reinsertion of sentinel values) may provide some
value, but I suspect it would be difficult to produce an effective API
even with function parameters to tweak the behaviour.

I will note that the check-and-reinsert example given elsewhere in this
thread contains a race condition in the multiple producer use case or in
cases where a single producer may place additional items in the queue
after the shutdown sentinel:

def _iterqueue(queue):
  while 1:
    item = queue.get()
    # Queue is not locked here, producers may insert more items
    # and other consumers may process items that were in the queue
    # after the sentinel
    if item is StopIteration:
      # We put StopIteration into the queue again, but it may not be
      # at the front if a producer inserted something after the original
      # insertion of the sentinel value
      queue.put(StopIteration)
      break
    else:
      yield item

This approach also doesn't work for sentinel values that are intended to
mean "stop processing the queue, go do something else, then come back
and start processing this queue again" since the queue can no longer be
processed at all after the sentinel value has been inserted once.

Cheers,
Nick.

P.S. An easy way to iterate over a queue in the simple "single consumer
with sentinel value" use case:

  for obj in iter(q.get, sentinel):
    pass

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------