[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Steve Dower]

Greg Ewing wrote:
> This will depend to some extent on whether Futures are considered
> part of the tasks layer or part of the callbacks layer. If they're
> considered part of the callbacks layer, they shouldn't have any
> methods that must be called with yield-from.

I put Futures very firmly in the callbacks layer (I guess the easiest reasoning for this is the complete lack of threading/async code in their implementation). Every time someone suggests "yielding a sentinel value" it seems that a Future is ideal for this - it even provides the other thread/tasklet/coroutine with a way to reactivate the original one, whether the two functions were written with knowledge of each other or not.

> As I've said, I think it would be better to have only 'yield from'
> calls in the public API, because it gives the implementation the
> greatest freedom.

I agree with this, though I still feel that we should be aiming for only 'yield' in the public API and leaving 'yield from' as a generalisation of this. For example, the two following pieces of code are basically equivalent:

@async
def task1():
    yield do_something_async_returning_a_future()

@async
def task2():
    yield task1()
    yield task1()

@async
def task3():
    yield task2()

task3().result()

And doing the same thing with yield from:

def task1():
    yield do_something_async_returning_a_future()

def task2():
    yield from task1()
    yield from task1()

@async
def task3():
    yield from task2()

task3().result()

This is also equivalent to this code:

@async
def task3():
    yield do_something_async_returning_a_future()
    yield do_something_async_returning_a_future()

task3().result()

And this:

def task():
    f = Future()
    do_something_async_returning_a_future().add_done_callback(
        lambda _: do_something_async_returning_a_future().add_done_callback(
            lambda _: f.set_result(None)
        )
    )
    return f

My point is that once we are using yield, yield from automatically becomes an option for composing operations. Teaching and validating this style is also easier, because the rule can be 'always use @async/yield in public APIs and just yield from in private APIs', and the biggest problem with not using yield from is that more Future objects are created. (The upsides were in my essay, but include compatibility with other Future-based APIs and composability between code from different sources.)

Cheers,
Steve