[Python-ideas] Add closing and iteration to threading.Queue

[MRAB]

On 2018-10-23 06:13, Nathaniel Smith wrote:
> On Sun, Oct 21, 2018 at 8:31 PM, Guido van Rossum <guido at python.org> wrote:
>> On Sun, Oct 21, 2018 at 6:08 PM Nathaniel Smith <njs at pobox.com> wrote:
>>> I'm not sure if this is an issue the way Queue is used in practice, but in
>>> general you have to be careful with this kind of circular flow because if
>>> your queue communicates backpressure (which it should) then circular flows
>>> can deadlock.
>>
>> Nathaniel, would you be able to elaborate more on the issue of backpressure?
>> I think a lot of people here are not really familiar with the concepts and
>> its importance, and it changes how you have to think about queues and the
>> like.
> 
> Sure.
> 
> Suppose you have some kind of producer connected to some kind of
> consumer. If the producer consistently runs faster than the consumer,
> what should happen? By default with queue.Queue, there's no limit on
> its internal buffer, so if the producer puts, say, 10 items per
> second, and the consumer only gets, say, 1 item per second, then the
> internal buffer grows by 9 items per second. Basically you have a
> memory leak, which will eventually crash your program. And well before
> that, your latency will become terrible. How can we avoid this?
> 
[snip]
The purpose of the sentinel is to tell the consumer(s) that there are no 
more items, that the producer has finished producing. The sentinel is 
the only item in the queue, there will be no more items after it, and 
backpressure is not an issue.