[Python-ideas] `__iter__` for queues?

[MRAB]

cool-RR wrote:
> On Wed, Jan 20, 2010 at 2:08 AM, Raymond Hettinger 
> <raymond.hettinger at gmail.com <mailto:raymond.hettinger at gmail.com>> wrote:
> 
> 
>      > Could it be made threadsafe?
> 
>     To me, this is exactly the right line of questioning.
> 
>     Is there a use case for iterating through a queue
>     that is being mutated by consumers and producers?
>     If so, should the consumers and producers be locked
>     out during iteration or should the iterator fail if it detects
>     a mutation (like we currently do for dictionaries that
>     change during iteration).
> 
> 
> In my program, I have one thread putting data into a queue, and the main 
> thread periodically iterates over the whole queue to integrate all the 
> data into a central data structure.
> 
> The general use case is when you want a thread to take items from a 
> queue and process them until the queue's empty. Like:
> 
> for work_item in queue:
>     process(work_item)
> 
>  
> 
>     The semantics should be dictated by use cases.
>     When someone writes "for t in q: action(q)" what
>     is the most useful thing to happen when another
>     thread does a get or put?
> 
> 
> (I think you did a typo in your code sample, I think you meant `action(t)`)
> 
> What I'm suggesting is that the iteration won't care about what other 
> threads do with the queue. It'll just `get` until it's empty.
> 
I agree. I don't see why __iter__ wouldn't work when .get does.

'Empty' could mean either a non-blocking .get and stopping when there
are no more items in the queue, or a blocking .get and stopping when 
there are no more items in the queue _and_ the queue has been closed by
the producer(s).
>  
> 
>     My personal opinion is that queues are better-off without
>     an iterator.  They serve mainly to buffer consumers and
>     producers and iteration does fit in well.  In other words,
>     adding __iter__ to queues in threaded environment
>     may do more harm than good.
> 
>     Raymond
> 
> 
> Okay. I don't have much experience in this, so unless someone else will 
> find this suggestion useful, I'll stick with my custom function.
>