[Python-ideas] The async API of the future: yield-from

[Piet Delport]

On Tue, Oct 16, 2012 at 9:45 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> In my system, spawn() isn't a wrapper -- it *is* the
> primitive way to create an independent task. And I
> think it's the only one we need.

I think you will at minimum need a way to suspend and resume tasks, in
addition to spawn(), as illustrated by the example of par() waiting for
not CPU-bound tasks.

This could be done either as actual suspend and resume primitives, or by
building on a related set of synchronization primitives, such as queues,
channels, or condition variables: there are a number of sets of that are
mutually co-expressible.

Suspending and resuming, in particular, is highly related to the
question of how you reify a task as a conventional callback, when the
need for that arises.

Here's one possible way of approaching this with a combined
suspend/resume primitive that might look familiar to people with a FP
background:

    result = yield suspend(lambda resume: ...)

(Here, "suspend" could be a scheduler-agnostic instruction object, a la
tasklib.suspend(), or a method on a global scheduler.)

suspend() would instruct the scheduler to stop running the current task,
and call its argument (the lambda in the above example) with a
"resume(value)" callable that will arrange to resume the task again with
the given value. The body of the lambda (or whatever is passed to
suspend()) would be responsible for doing something useful with the
resume() callable: e.g. in par() example, it would arrange that the last
child task triggers it.

In particular, this suspend() could be used to integrate fairly directly
with callback-based APIs: for example, if you have a Twisted Deferred,
you could do:

    result = yield suspend(d.addCallback)

to suspend the current task and add a callback to d that will resume it
again, and receive the Deferred's result.

To add support for exceptions, a variation of suspend() could pass two
callables, mirroring pairs like send/throw, or callback/errback:

    result = yield suspend2(lambda resume, throw: ...)

    result = yield suspend2(d.addCallbacks)

-- 
Piet Delport