[Python-ideas] `__iter__` for queues?

[Raymond Hettinger]

On Jan 19, 2010, at 6:06 PM, Terry Reedy wrote:

> On 1/19/2010 4:20 PM, cool-RR wrote:
>> 
>> For me, iterating on the queue means just calling `get` repeatedly until
>> it's empty.
> 
  
If that is all you're looking for, then just do that.  Write a short generator to dump the queue.  You can do that without blocking your producer thread.

The problem with proposing an change to the API for Queue.Queue and MultiProcessing.Queue is that it presents non-obvious pitfalls for other common use cases.  It is better to leave it out of the API and let users explicitly craft solutions that fit their problem (perhaps using two queues:  Producer --->  DatabaseUpdater --> Consumer or some solution involving locking).  For many uses of Queue, it doesn't make sense to block a producer for the time it takes to iterate, nor does it make sense to have a non-blocking iterator (leading to a race condition and inconsistent results).

It is better to use the API as designed than to propose a new method than may be unsuitable for many use cases.


> 
> So iterating with an iterator is not quite the same as repeated one-item fetches.
> 


Right.  The two ideas should not be conflated.  A Queue object is all about buffering producers and consumers running in different threads.  Anything API that is at odds with that notion is probably not a good idea (i.e. iterating over a dynamically updating queue buffer).


Raymond