[Python-ideas] Add closing and iteration to threading.Queue

Vladimir Filipović hemflit at gmail.com
Mon Oct 22 03:21:34 EDT 2018


Nathaniel, thank you for the pointer to Trio.
Its approach seems very robust. I'm relieved to see that a solution so
fundamentally rebuilt has also settled on very similar semantics for
its `.close_put()`.

I think your `.clone()` idiom is very clear when the communication
objects are treated as distinct endpoints. Something with similar
effects on closing (not necessarily similar in idiom) would probably
be a neat enhancement to the standard Queue, though if I was making
that, I'd do it in an external package.

------

Antoine, regarding multiprocessing.Queue:

The similarity of meaning behind closing that I was getting at is that
mp.Q.close() means "I am done writing to this queue, and I don't care
about the rest of you", whereas the proposed meaning of q.Q.close() is
"Listen up, we are all done writing to this queue". I don't know yet
that this difference necessarily creates a true incompatibility.

That the effects (in terms of eager OS-resource cleanup) are different
shouldn't be a problem in itself - every implementation does the right
thing for itself.

------

On Mon, Oct 22, 2018 at 2:03 AM Terry Reedy <tjreedy at udel.edu> wrote:
> The proposed close method would only half-close the queue: closed to
> puts, open to gets (but perhaps close completely when the last item is
> gotten.

In other words: in this proposal, there is no such thing as "closed
for retrieval". A closed queue means exactly that it's closed for
insertion.

Retrieval becomes permanently impossible once the queue is closed and
exhausted, and that's a condition that get() must treat correctly and
usefully, but calling that condition "closed / completely closed /
closed for retrieval" would muddle up the terminology.

In the proposed implementation I've called it "exhausted", a name I've
picked up god-knows-when and from god-knows-where, but it seemed
reasonable.

------

Regarding sentinels in general: They are a limited-purpose solution,
and this proposal should make them unnecessary in 99% of the cases.

Firstly, they only naturally apply to FIFO queues. You could hack your
use of LIFO and priority queues to also rely on sentinels, but it's
very kludgey in the general cases, not a natural fit, and not
generalizable to user-created children of Queue (which Queue otherwise
explicitly aspires to support).

Secondly, they only work when the producer is the one driving the flow
and notifying the consumer that "no more is forthcoming". They don't
work when the producer is the one who needs to be notified.

Thirdly, they're a potential cause of deadlocks when the same threads
act as both producers and consumers. (E.g. in a parallelized
breadth-first-search.) I'm sure this is the circular flow that
Nathaniel was referring to, but I'll let him detail it more or correct
me.

Fourthly, they don't make it easy to query the Queue about whether
it's closed. This probably isn't a big deal admittedly.

Sure, when sentinels are adequate, they're adequate. This proposal
aims to be more general-purpose than that.


More information about the Python-ideas mailing list