[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Guido van Rossum]

On Oct 21, 2012 9:48 AM, "Steve Dower" <Steve.Dower at microsoft.com> wrote:
>
> Greg Ewing wrote:
> > This will depend to some extent on whether Futures are considered
> > part of the tasks layer or part of the callbacks layer. If they're
> > considered part of the callbacks layer, they shouldn't have any
> > methods that must be called with yield-from.
>
> I put Futures very firmly in the callbacks layer (I guess the easiest
reasoning for this is the complete lack of threading/async code in their
implementation).

Did you check the source? That's simply incorrect. It uses locks, of the
threading variety.

( However one could write an implementation with the same interface that
doesn't.)

> Every time someone suggests "yielding a sentinel value" it seems that a
Future is ideal for this - it even provides the other
thread/tasklet/coroutine with a way to reactivate the original one, whether
the two functions were written with knowledge of each other or not.

This I like.

> > As I've said, I think it would be better to have only 'yield from'
> > calls in the public API, because it gives the implementation the
> > greatest freedom.
>
> I agree with this, though I still feel that we should be aiming for only
'yield' in the public API and leaving 'yield from' as a generalisation of
this. For example, the two following pieces of code are basically
equivalent:
>
> @async
> def task1():
>     yield do_something_async_returning_a_future()
>
> @async
> def task2():
>     yield task1()
>     yield task1()
>
> @async
> def task3():
>     yield task2()
>
> task3().result()
>
> And doing the same thing with yield from:
>
> def task1():
>     yield do_something_async_returning_a_future()
>
> def task2():
>     yield from task1()
>     yield from task1()
>
> @async
> def task3():
>     yield from task2()
>
> task3().result()
>
> This is also equivalent to this code:
>
> @async
> def task3():
>     yield do_something_async_returning_a_future()
>     yield do_something_async_returning_a_future()
>
> task3().result()
>
> And this:
>
> def task():
>     f = Future()
>     do_something_async_returning_a_future().add_done_callback(
>         lambda _:
do_something_async_returning_a_future().add_done_callback(
>             lambda _: f.set_result(None)
>         )
>     )
>     return f
>
> My point is that once we are using yield, yield from automatically
becomes an option for composing operations. Teaching and validating this
style is also easier, because the rule can be 'always use @async/yield in
public APIs and just yield from in private APIs', and the biggest problem
with not using yield from is that more Future objects are created. (The
upsides were in my essay, but include compatibility with other Future-based
APIs and composability between code from different sources.)

Hm. I think it'll be confusing. And the Futures-only-in-public-APIs rule
seems to encourage less efficient solutions.

--Guido van Rossum (sent from Android phone)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121021/9669ad24/attachment.html>