<div dir="auto">Hi Vladimir,<div dir="auto"><br></div><div dir="auto">It's great to see people revisiting these old stdlib tools. Closure tracking is definitely a big point of awkwardness for Queues. In Trio we started with a straight copy of threading.Queue, and this turned out to be a major friction point for users. We just deprecated our version of Queue and replaced it with a new design. Our new thing is probably more radical than you want to get in the stdlib (we ended up splitting the object into two pieces, a sender object and a receiver object), but you might find the discussions interesting:</div><div dir="auto"><br></div><div dir="auto">Manual:</div><div dir="auto"><a href="https://trio.readthedocs.io/en/latest/reference-core.html#using-channels-to-pass-values-between-tasks">https://trio.readthedocs.io/en/latest/reference-core.html#using-channels-to-pass-values-between-tasks</a><br></div><div dir="auto"><br></div><div dir="auto">A more minimal proposal to add closure tracking to trio.Queue:</div><div dir="auto"><a href="https://github.com/python-trio/trio/pull/573">https://github.com/python-trio/trio/pull/573</a><br></div><div dir="auto"><br></div><div dir="auto">Follow-up issue with design questions we're still thinking about (also links to earlier design discussions):</div><div dir="auto"><a href="https://github.com/python-trio/trio/issues/719">https://github.com/python-trio/trio/issues/719</a><br></div><div dir="auto"><br></div><div dir="auto">We only started shipping this last week, so we're still getting experience with it.</div><div dir="auto"><br></div><div dir="auto">-n</div></div><br><div class="gmail_quote"><div dir="ltr">On Sun, Oct 21, 2018, 10:59 Vladimir Filipović <<a href="mailto:hemflit@gmail.com">hemflit@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi!<br>
<br>
I originally submitted this as a pull request. Raymond Hettinger<br>
suggested it should be given a shakeout in python-ideas first.<br>
<br>
<a href="https://github.com/python/cpython/pull/10018" rel="noreferrer noreferrer" target="_blank">https://github.com/python/cpython/pull/10018</a><br>
<a href="https://bugs.python.org/issue35034" rel="noreferrer noreferrer" target="_blank">https://bugs.python.org/issue35034</a><br>
<br>
------<br>
<br>
Briefly:<br>
<br>
Add a close() method to Queue, which should simplify many common uses<br>
of the class and reduce the space for some easy-to-make errors.<br>
<br>
Also add an __iter__() method which in conjunction with close() would<br>
further simplify some common use patterns.<br>
<br>
------<br>
<br>
At eye-watering length:<br>
<br>
Apologies in advance for the length of this message. This isn't a PEP<br>
in disguise, it's a proposal for a very small, simple and I dare<br>
imagine uncontroversial feature. I'm new to contributing to Python and<br>
after the BPO/github submission I didn't manage to come up with a<br>
better way to present it than this.<br>
<br>
The issue<br>
<br>
Code using threading.Queue often needs to coordinate a "work is<br>
finished as far as I care" state between the producing and consuming<br>
side. Not "work" in the task_done() sense of completion of processing<br>
of queue items, "work" in the simpler sense of just passing data<br>
through the queue.<br>
<br>
For example, a producer can be driving the communication by enqueuing<br>
e.g. names of files that need to be processed, and once it's enqueued<br>
the last filename, it can be useful to inform the consumers that no<br>
further names will be coming, so after they've retrieved what's<br>
in-flight currently, they don't need to bother waiting for any more.<br>
Alternatively, a consumer can be driving the communication, and may<br>
need to let the producers know "I'm not interested in any more, so you<br>
can stop wasting resources on producing and enqueuing them".<br>
Also, a third, coordinating component may need to let both sides know<br>
that "Our collective work here is done. Start wrapping it up y'all,<br>
but don't drop any items that are still in-flight."<br>
<br>
In practice it's probably the exception, not the rule, when any piece<br>
of code interacting with a Queue _doesn't_ have to either inform<br>
another component that its interest in transferring the data has<br>
ended, or watch for such information.<br>
<br>
In the most common case of producer letting consumers know that it's<br>
done, this is usually implemented (over and over again) with sentinel<br>
objects, which is at best needlessly verbose and at worst error-prone.<br>
A recipe for multiple consumers making sure nobody misses the sentinel<br>
is not complicated, but neither is it obvious the first time one needs<br>
to do it.<br>
When a generic sentinel (None or similar) isn't adequate, some<br>
component needs to create the sentinel object and communicate it to<br>
the others, which complicates code, and especially complicates<br>
interfaces between components that are not being developed together<br>
(e.g. if one of them is part of a library and expects the library-user<br>
code to talk to it through a Queue).<br>
<br>
In the less common cases where the producers are the ones being<br>
notified, there isn't even a typical solution - everything needs to be<br>
cooked up from scratch using synchronization primitives.<br>
<br>
------<br>
<br>
A solution<br>
<br>
Adding a close() method to the Queue that simply prohibits all further<br>
put()'s (with other methods acting appropriately when the queue is<br>
closed) would simplify a lot of this in a clean and safe way - for the<br>
most obvious example, multi-consumer code would not have to juggle<br>
sentinel objects.<br>
<br>
Adding a further __iter__() method (that would block as necessary, and<br>
stop its iteration once the queue is closed and exhausted) would<br>
especially simplify many unsophisticated consumers.<br>
<br>
This is a current fairly ordinary pattern:<br>
<br>
# Producer:<br>
while some_condition:<br>
q.put(generate_item())<br>
q.put(sentinel)<br>
<br>
# Consumer:<br>
while True:<br>
item = q.get()<br>
if item == sentinel:<br>
q.put(sentinel)<br>
break<br>
process(item)<br>
<br>
(This consumer could be simplified a little with an assignment<br>
expression or an iter(q.get, sentinel), but one of those is super new<br>
and the other seems little-known in spite of being nearly old enough<br>
to vote.)<br>
<br>
With the close() and __iter__(), this would become:<br>
<br>
# Producer:<br>
with closing(q):<br>
while some_condition:<br>
q.put(generate_item())<br>
<br>
# Consumer:<br>
for item in q:<br>
process(item)<br>
<br>
Apart from it being shorter and less error-prone (e.g. if<br>
generate_item raises), the implied interface for initializing the two<br>
components is also simplified, because there's no sentinel to pass<br>
around.<br>
<br>
More complex consumers that couldn't take advantage of the __iter__()<br>
would still benefit from being able to explicitly and readably find<br>
out (via exception or querying) that the queue has been closed and<br>
exhausted, without juggling the sentinel.<br>
<br>
I honestly think this small addition would be an unqualified win. And<br>
it would not change anything for code that doesn't want to use it.<br>
<br>
------<br>
<br>
I've got a sample implementation ready for Queue and its children.<br>
(Link is at the start of this message. It includes documentation<br>
updates too, in case those clarify anything further at this stage.)<br>
<br>
If this is going in the right direction, I'm happy to do the same for<br>
SimpleQueue, but I haven't done it yet. I'm still getting my bearings<br>
around the C code base.<br>
<br>
------<br>
<br>
To immediately answer some of Raymond's initial comments at BPO:<br>
<br>
This is completely independent from Queue's existing task-tracking<br>
protocol. One is about controlling transport, and the other about<br>
tracking the completion of processing after transport. I hesitate to<br>
call them "orthogonal", but there is no functional overlap between<br>
them at all.<br>
<br>
I keep talking about "common cases" and such weaselly concepts above,<br>
and really it's just conjecture based on my own limited experience and<br>
informal conversations.<br>
BUT, I've also done a survey of the standard library itself. There are<br>
four packages that use threading.Queue: concurrent, idlelib, logging,<br>
multiprocessing. All except idlelib have at least one piece that would<br>
have benefited from a Queue.close() if it had been available when they<br>
were being written.<br>
<br>
I've now had a look at a handful of other languages too:<br>
- Java and C# have nothing like this. They basically start from a<br>
deque as a collection, and add synchronization features to it; C++ STL<br>
doesn't even go that far. None of them do the Python thing where a<br>
Queue is a dedicated communication tool that just uses a collection as<br>
part of its implementation.<br>
- Ruby (to my mild surprise) also does nothing like this.<br>
- Clojure does just about the same thing as this proposal, yay.<br>
- Go does approximately the same thing, just more finicky about<br>
practical usage. The producer can say `close(q)` and the consumers can<br>
say `for item := range q { process(item) }` which do exactly the same<br>
thing as the proposed Python equivalents, but it has some harsh<br>
limitations that are not applicable to Python. (Can't re-close a<br>
closed channel, can't query whether a channel is closed outside of<br>
retrieving an item).<br>
<br>
------<br>
<br>
To anticipate a couple more possible questions:<br>
<br>
- What would this proposal do about multiple producers/consumers<br>
needing to jointly decide _when_ to close the queue?<br>
<br>
Explicitly nothing.<br>
<br>
The queue's state is either closed or not, and it doesn't care who<br>
closed it. It needs to interact correctly with multiple consumers and<br>
multiple producers, but once any one piece of code closes it, the<br>
correct interaction is acting like a closed queue for everybody.<br>
<br>
When e.g. multiple producers need to arrange among themselves that the<br>
queue be closed only after all of them are done producing, then it's<br>
not the queue's job to coordinate _that_. They probably need a<br>
Semaphore or a more complex coordinator in charge of the closing. Yes,<br>
this too is a non-trivial-to-get-right situation, but trying to solve<br>
this one inside the Queue class seems like bloat.<br>
<br>
- Why not make the Queue a context manager while you're at it? Most<br>
closeable classes do it.<br>
<br>
I don't see that as a clear win in this case. Happy to add it if<br>
there's consensus in its favour, of course.<br>
<br>
I think `with closing(q):` is much more expressive than `with q:`<br>
while still brief. The Queue is more versatile in use than files,<br>
database cursors and many other resource-type classes, so the meaning<br>
of a `with q:` would not be as immediately suggestive as with them. It<br>
also often needs to be passed around between creation and initial use<br>
(the whole point is that more than one piece of code has access to it)<br>
so the common idiom `with Queue() as q:` would be a slightly less<br>
natural fit than with resource-like classes.<br>
<br>
- Should SimpleQueue do the same thing as Queue?<br>
<br>
Yes, I would propose so and volunteer to implement it.<br>
Some details would be adapted to SimpleQueue's existing promises<br>
(put() can't fail) but I think this feature is very much in the spirit<br>
of SimpleQueue.<br>
<br>
- Should multiprocessing.Queue do the same thing?<br>
<br>
I think so, though I'm not proposing it.<br>
<br>
It already has a close() method, whose meaning is very similar but not<br>
identical to (a subset of) the proposed threading.Queue.close's<br>
meaning (with resource-management effects not relevant to<br>
threading.Queue either way). I'd hope that this "not identical" part<br>
would not cause any problems in practice (lots of things aren't<br>
identical between those two Queues), but all hoping aside, maybe<br>
people more experienced than me can evaluate if that's really the<br>
case.<br>
<br>
I also have no clear idea if the feature would be as valuable, and if<br>
the implementation would be as easy and clean as they are with<br>
threading.Queue.<br>
<br>
- Should asyncio.Queue do the same thing?<br>
<br>
Probably? I'm too unfamiliar with asyncio for my opinion on this to be<br>
of value. So I'm not proposing it.<br>
<br>
------<br>
_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org" target="_blank" rel="noreferrer">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer noreferrer" target="_blank">http://python.org/psf/codeofconduct/</a><br>
</blockquote></div>