[Python-ideas] PEP 3156 feedback: wait_one vs par vs concurrent.futures.wait

[Nick Coghlan]

On Sun, Dec 23, 2012 at 1:54 AM, Guido van Rossum <guido at python.org> wrote:
> On Sat, Dec 22, 2012 at 12:04 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I deliberately chose to return coroutines. My rationale is to be able
>> to handle the case where multiple operations become ready without
>> having to make multiple trips around the event loop by having the
>> iterator switch between two modes: when the complete set is empty, it
>> yields a coroutine that calls wait and then returns the first complete
>> future, while when there are already complete futures available, it
>> yields a coroutine that just returns one of them immediately. It's
>> really the same rationale as that for having @coroutine not
>> automatically wrap things in Task - if we can avoid the event loop in
>> cases that don't actually need to wait for an event, that's a good
>> thing.
>
> I think I see it now. The first item yielded is the simplest thing
> that can be used with yield-from, i.e. a coroutine. Then if multiple
> futures are ready at once, you return an item of the same type, i.e. a
> coroutine. This is essentially wrapping a Future in a coroutine! If we
> could live with the items being alternatingly coroutines and Futures,
> we could just return the Future in this case. BTW, yield from <future>
> need not go to the scheduler if the Future is already done -- the
> Future,__iter__ method should be:
>
>     def __iter__(self):
>         if not self.done():
>             yield self  # This tells Task to wait for completion.
>         return self.result()  # May raise too.
>
> (I forgot this previously.)

And I'd missed it completely :)

In that case, yeah, yielding any already completed Futures directly
from as_completed() should work. The "no completed operations" case
will still need a coroutine, though, as it needs to update the
"complete" and "incomplete" sets inside the iterator. Since we know
we're certain to hit the scheduler in that case, we may as well wrap
it directly in a task so we're always returning some kind of future.
The impl might end up looking something like:

    def as_completed(fs):
        incomplete = fs
        while incomplete:
            # Phase 1 of the loop, we yield a Task that waits for operations
            @coroutine
            def _wait_for_some():
                nonlocal complete, incomplete
                complete, incomplete = yield from tulip.wait(fs,
return_when=FIRST_COMPLETED)
                return complete.pop().result()
            yield Task(_wait_for_some())
            # Phase 2 of the loop, we pass back the already complete operations
            while complete:
                yield complete.pop()

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia