[Python-ideas] PEP 3156 feedback: wait_one vs par vs concurrent.futures.wait

[Guido van Rossum]

On Sat, Dec 22, 2012 at 12:04 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Dec 22, 2012 at 4:20 PM, Guido van Rossum <guido at python.org> wrote:
>> I did update the PEP. There are some questions about details; e.g. I
>> think the 'fs' argument should allow a mixture of Futures and
>> coroutines (the latter will be wrapped Tasks) and the sets returned by
>> wait() should contain Futures and Tasks.
>
> Yes, I think I wrote my examples that way, even though I didn't say
> that in the text.

Good.

>> You propose that
>> as_completed() returns an iterator whose items are coroutines; why not
>> Futures? (They're more versatile even if slightly slower that
>> coroutines.) I can sort of see the reasoning but want to tease out
>> whether you meant it that way.
>
> I deliberately chose to return coroutines. My rationale is to be able
> to handle the case where multiple operations become ready without
> having to make multiple trips around the event loop by having the
> iterator switch between two modes: when the complete set is empty, it
> yields a coroutine that calls wait and then returns the first complete
> future, while when there are already complete futures available, it
> yields a coroutine that just returns one of them immediately. It's
> really the same rationale as that for having @coroutine not
> automatically wrap things in Task - if we can avoid the event loop in
> cases that don't actually need to wait for an event, that's a good
> thing.

I think I see it now. The first item yielded is the simplest thing
that can be used with yield-from, i.e. a coroutine. Then if multiple
futures are ready at once, you return an item of the same type, i.e. a
coroutine. This is essentially wrapping a Future in a coroutine! If we
could live with the items being alternatingly coroutines and Futures,
we could just return the Future in this case. BTW, yield from <future>
need not go to the scheduler if the Future is already done -- the
Future,__iter__ method should be:

    def __iter__(self):
        if not self.done():
            yield self  # This tells Task to wait for completion.
        return self.result()  # May raise too.

(I forgot this previously.)

>> Also, we can't have __next__() raise
>> TimeoutError, since it never blocks; it will have to be the coroutine
>> (or Future) returned by __next__().
>
> Yeah, any exceptions should happen at the yield from call inside the
> loop. I *think* my implementation achieves that (since the coroutine
> instances it creates are passed out to the for loop for further
> processing), but it's quite possible I missed something.

It'll come out in implementation (in two weeks, maybe).

-- 
--Guido van Rossum (python.org/~guido)