[Python-ideas] Are there asynchronous generators?

[Guido van Rossum]

Not following this in detail, but want to note that async isn't a good
model for parallelization (except I/O) because the expectation of
coroutines is single threading. The event loop serializes callbacks.
Changing this would break expectations and code.
On Jun 29, 2015 10:33 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

> On 29 June 2015 at 16:44, Paul Sokolovsky <pmiscml at gmail.com> wrote:
> > Hello,
> >
> > On Sun, 28 Jun 2015 18:14:20 -0400
> > Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> >
> >>
> >> On 2015-06-28 11:52 AM, Paul Sokolovsky wrote:
> >> >> There is also the problem that one cannot easily feed a queue,
> >> >> >asynchronous generator, or any asynchronous iterator to a simple
> >> >> >synchronous consumer like sum() or list() or "".join(). It would
> >> >> >be nice if there was a way to wrap them to asynchronous ones when
> >> >> >needed – something like (async
> >> >> >sum)(asynchronously_produced_numbers()).
> >> > All that is easily achievable with classical Python coroutines, not
> >> > with asyncio garden variety of coroutines, which lately were casted
> >> > into a language level with async/await disablers:
> >> >
> >> > def coro1():
> >> >      yield 1
> >> >      yield 2
> >> >      yield 3
> >> >
> >> > def coro2():
> >> >      yield from coro1()
> >> >      yield 4
> >> >      yield 5
> >> >
> >> > print(sum(coro2()))
> >>
> >>
> >> You have easily achieved combining two generators with 'yield from'
> >> and feeding that to 'sum' builtin.
> >
> > Right, the point here was that PEP492, banning usage of "yield" in
> > coroutines, doesn't help with such simple and basic usage of them. And
> > then I again can say what I said during initial discussion of PEP492:
> > I have dual feeling about it: promise of making coroutines easier and
> > more user friendly is worth all support, but step of limiting basic
> > language usage in them doesn't seem good. What me and other people can
> > do then is just trust that you guys know what you do and PEP492 will
> > be just first step. But bottom line is that I personally don't find
> > async/await worthy to use for now - it's better to stick to old good
> > yield from, until the promise of truly better coroutines is delivered.
>
> The purpose of PEP 492 is to fundamentally split the asynchronous IO
> use case away from traditional generators. If you're using native
> coroutines, you MUST have an event loop, or at least be using
> something like asyncio.run_until_complete() (which spins up a
> scheduler for the duration). If you're using generators without
> @types.coroutine or @asyncio.coroutine (or the equivalent for tulip,
> Twisted, etc), then you're expecting a synchronous driver rather than
> an asynchronous one.
>
> This isn't an accident, or something that will change at some point in
> the future, it's the entire point of the exercise: having it be
> obvious both how you're meant to interact with something based on the
> way it's defined, and how you factor outside subcomponents of the
> algorithm. Asynchronous driver? Use a coroutine. Synchronous driver?
> Use a generator.
>
> What we *don't* have are consumption functions that have an implied
> "async for" inside them - functions like sum(), any(), all(), etc are
> all synchronous drivers.
>
> The other key thing we don't have yet? Asynchronous comprehensions.
>
> A peak at the various options for parallel execution described in
> https://docs.python.org/3/library/concurrent.futures.html
> documentation helps illustrate why: once we're talking about applying
> reduction functions to asynchronous iterables we're getting into
> full-blown language-level-support-for-MapReduce territory. Do the
> substeps still need to be executed in series? Or can the substeps be
> executed in parallel, and either accumulated in iteration order or as
> they become available? Does it perhaps make sense to *require* that
> the steps be executable in parallel, such that we could write the
> following:
>
>     result = sum(x*x for async x in coro)
>
> Where the reduction step remains synchronous, but we can mark the
> comprehension/map step as asynchronous, and have that change the
> generated code to create an implied lambda for the "lambda x: x*x"
> calculation, dispatch all of those to the scheduler at once, and then
> produce the results one at a time?
>
> The answer to that is "quite possibly, but we don't really know yet".
> PEP 492 is enough to address some major comprehensibility challenges
> that exist around generators-as-coroutines. It *doesn't* bring
> language level support for parallel MapReduce to Python, but it *does*
> bring some interesting new building blocks for folks to play around
> with in that regard (in particular, figuring out what we want the
> comprehension level semantics of "async for" to be).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150629/48bf8e31/attachment.html>