[Python-ideas] The async API of the future: yield-from

[Nick Coghlan]

On Mon, Oct 15, 2012 at 8:25 PM, Calvin Spealman <ironfroggy at gmail.com> wrote:
> The more I follow this thread the less I understand the point of
> introducing a new use for yield-from in this discussion.

+1. To me, "yield from" is just a tool that brings generators back to
parity with functions when it comes to breaking up a larger algorithm
into smaller pieces. Where you would break a function out into
subfunctions and call them normally, with a generator you can break
out subgenerators and invoke them with yield from.

Any meaningful use of "yield from" in the coroutine context *has* to
ultimate devolve to an operation that:
1. Asks the scheduler to schedule another operation
2. Waits for that operation to complete

Guido's approach to that problem is that step 1 is handled by calling
functions that in turn call methods on a thread-local scheduler. These
methods return Future objects, which can subsequently be yielded to
the scheduler to say "I'm waiting for this future to be set".

I *thought* Greg's way combined step 1 and step 2 into a single
operation: the objects you yield *not only* say what you want to wait
for, but also what you want to do. However, his example par()
implementation killed that idea, since it turned out to need to
schedule tasks explicitly rather than their being a "execute this in
parallel" option.

So now I'm back to think that Greg and Guido are talking about
different levels. *Any* scheduling option will be able to be collapsed
into an async task invoked by "yield from" by writing:

    def simple_async_task():
        return yield start_task()

The part that still needs to be figured out is how you turn that
suspend/resume communications channel between the lowest level of the
task stack and the scheduling loop into something usable, as well as
how you handle iteration in a sensible way (I described my preferred
approach when writing about the API I'd like to see for an async
version of as_completed). I haven't seen anything to suggest that
"yield from"'s role should change from what it is in 3.3: a way to
factor out generators into multiple pieces with out breaking send()
and throw().

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia