[Python-ideas] Are there asynchronous generators?

[Nick Coghlan]

On 29 June 2015 at 16:44, Paul Sokolovsky <pmiscml at gmail.com> wrote:
> Hello,
>
> On Sun, 28 Jun 2015 18:14:20 -0400
> Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>
>>
>> On 2015-06-28 11:52 AM, Paul Sokolovsky wrote:
>> >> There is also the problem that one cannot easily feed a queue,
>> >> >asynchronous generator, or any asynchronous iterator to a simple
>> >> >synchronous consumer like sum() or list() or "".join(). It would
>> >> >be nice if there was a way to wrap them to asynchronous ones when
>> >> >needed – something like (async
>> >> >sum)(asynchronously_produced_numbers()).
>> > All that is easily achievable with classical Python coroutines, not
>> > with asyncio garden variety of coroutines, which lately were casted
>> > into a language level with async/await disablers:
>> >
>> > def coro1():
>> >      yield 1
>> >      yield 2
>> >      yield 3
>> >
>> > def coro2():
>> >      yield from coro1()
>> >      yield 4
>> >      yield 5
>> >
>> > print(sum(coro2()))
>>
>>
>> You have easily achieved combining two generators with 'yield from'
>> and feeding that to 'sum' builtin.
>
> Right, the point here was that PEP492, banning usage of "yield" in
> coroutines, doesn't help with such simple and basic usage of them. And
> then I again can say what I said during initial discussion of PEP492:
> I have dual feeling about it: promise of making coroutines easier and
> more user friendly is worth all support, but step of limiting basic
> language usage in them doesn't seem good. What me and other people can
> do then is just trust that you guys know what you do and PEP492 will
> be just first step. But bottom line is that I personally don't find
> async/await worthy to use for now - it's better to stick to old good
> yield from, until the promise of truly better coroutines is delivered.

The purpose of PEP 492 is to fundamentally split the asynchronous IO
use case away from traditional generators. If you're using native
coroutines, you MUST have an event loop, or at least be using
something like asyncio.run_until_complete() (which spins up a
scheduler for the duration). If you're using generators without
@types.coroutine or @asyncio.coroutine (or the equivalent for tulip,
Twisted, etc), then you're expecting a synchronous driver rather than
an asynchronous one.

This isn't an accident, or something that will change at some point in
the future, it's the entire point of the exercise: having it be
obvious both how you're meant to interact with something based on the
way it's defined, and how you factor outside subcomponents of the
algorithm. Asynchronous driver? Use a coroutine. Synchronous driver?
Use a generator.

What we *don't* have are consumption functions that have an implied
"async for" inside them - functions like sum(), any(), all(), etc are
all synchronous drivers.

The other key thing we don't have yet? Asynchronous comprehensions.

A peak at the various options for parallel execution described in
https://docs.python.org/3/library/concurrent.futures.html
documentation helps illustrate why: once we're talking about applying
reduction functions to asynchronous iterables we're getting into
full-blown language-level-support-for-MapReduce territory. Do the
substeps still need to be executed in series? Or can the substeps be
executed in parallel, and either accumulated in iteration order or as
they become available? Does it perhaps make sense to *require* that
the steps be executable in parallel, such that we could write the
following:

    result = sum(x*x for async x in coro)

Where the reduction step remains synchronous, but we can mark the
comprehension/map step as asynchronous, and have that change the
generated code to create an implied lambda for the "lambda x: x*x"
calculation, dispatch all of those to the scheduler at once, and then
produce the results one at a time?

The answer to that is "quite possibly, but we don't really know yet".
PEP 492 is enough to address some major comprehensibility challenges
that exist around generators-as-coroutines. It *doesn't* bring
language level support for parallel MapReduce to Python, but it *does*
bring some interesting new building blocks for folks to play around
with in that regard (in particular, figuring out what we want the
comprehension level semantics of "async for" to be).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia