<p dir="ltr">Not following this in detail, but want to note that async isn't a good model for parallelization (except I/O) because the expectation of coroutines is single threading. The event loop serializes callbacks. Changing this would break expectations and code. </p>
<div class="gmail_quote">On Jun 29, 2015 10:33 AM, "Nick Coghlan" <<a href="mailto:ncoghlan@gmail.com">ncoghlan@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 29 June 2015 at 16:44, Paul Sokolovsky <<a href="mailto:pmiscml@gmail.com">pmiscml@gmail.com</a>> wrote:<br>
> Hello,<br>
><br>
> On Sun, 28 Jun 2015 18:14:20 -0400<br>
> Yury Selivanov <<a href="mailto:yselivanov.ml@gmail.com">yselivanov.ml@gmail.com</a>> wrote:<br>
><br>
>><br>
>> On 2015-06-28 11:52 AM, Paul Sokolovsky wrote:<br>
>> >> There is also the problem that one cannot easily feed a queue,<br>
>> >> >asynchronous generator, or any asynchronous iterator to a simple<br>
>> >> >synchronous consumer like sum() or list() or "".join(). It would<br>
>> >> >be nice if there was a way to wrap them to asynchronous ones when<br>
>> >> >needed – something like (async<br>
>> >> >sum)(asynchronously_produced_numbers()).<br>
>> > All that is easily achievable with classical Python coroutines, not<br>
>> > with asyncio garden variety of coroutines, which lately were casted<br>
>> > into a language level with async/await disablers:<br>
>> ><br>
>> > def coro1():<br>
>> > yield 1<br>
>> > yield 2<br>
>> > yield 3<br>
>> ><br>
>> > def coro2():<br>
>> > yield from coro1()<br>
>> > yield 4<br>
>> > yield 5<br>
>> ><br>
>> > print(sum(coro2()))<br>
>><br>
>><br>
>> You have easily achieved combining two generators with 'yield from'<br>
>> and feeding that to 'sum' builtin.<br>
><br>
> Right, the point here was that PEP492, banning usage of "yield" in<br>
> coroutines, doesn't help with such simple and basic usage of them. And<br>
> then I again can say what I said during initial discussion of PEP492:<br>
> I have dual feeling about it: promise of making coroutines easier and<br>
> more user friendly is worth all support, but step of limiting basic<br>
> language usage in them doesn't seem good. What me and other people can<br>
> do then is just trust that you guys know what you do and PEP492 will<br>
> be just first step. But bottom line is that I personally don't find<br>
> async/await worthy to use for now - it's better to stick to old good<br>
> yield from, until the promise of truly better coroutines is delivered.<br>
<br>
The purpose of PEP 492 is to fundamentally split the asynchronous IO<br>
use case away from traditional generators. If you're using native<br>
coroutines, you MUST have an event loop, or at least be using<br>
something like asyncio.run_until_complete() (which spins up a<br>
scheduler for the duration). If you're using generators without<br>
@types.coroutine or @asyncio.coroutine (or the equivalent for tulip,<br>
Twisted, etc), then you're expecting a synchronous driver rather than<br>
an asynchronous one.<br>
<br>
This isn't an accident, or something that will change at some point in<br>
the future, it's the entire point of the exercise: having it be<br>
obvious both how you're meant to interact with something based on the<br>
way it's defined, and how you factor outside subcomponents of the<br>
algorithm. Asynchronous driver? Use a coroutine. Synchronous driver?<br>
Use a generator.<br>
<br>
What we *don't* have are consumption functions that have an implied<br>
"async for" inside them - functions like sum(), any(), all(), etc are<br>
all synchronous drivers.<br>
<br>
The other key thing we don't have yet? Asynchronous comprehensions.<br>
<br>
A peak at the various options for parallel execution described in<br>
<a href="https://docs.python.org/3/library/concurrent.futures.html" rel="noreferrer" target="_blank">https://docs.python.org/3/library/concurrent.futures.html</a><br>
documentation helps illustrate why: once we're talking about applying<br>
reduction functions to asynchronous iterables we're getting into<br>
full-blown language-level-support-for-MapReduce territory. Do the<br>
substeps still need to be executed in series? Or can the substeps be<br>
executed in parallel, and either accumulated in iteration order or as<br>
they become available? Does it perhaps make sense to *require* that<br>
the steps be executable in parallel, such that we could write the<br>
following:<br>
<br>
result = sum(x*x for async x in coro)<br>
<br>
Where the reduction step remains synchronous, but we can mark the<br>
comprehension/map step as asynchronous, and have that change the<br>
generated code to create an implied lambda for the "lambda x: x*x"<br>
calculation, dispatch all of those to the scheduler at once, and then<br>
produce the results one at a time?<br>
<br>
The answer to that is "quite possibly, but we don't really know yet".<br>
PEP 492 is enough to address some major comprehensibility challenges<br>
that exist around generators-as-coroutines. It *doesn't* bring<br>
language level support for parallel MapReduce to Python, but it *does*<br>
bring some interesting new building blocks for folks to play around<br>
with in that regard (in particular, figuring out what we want the<br>
comprehension level semantics of "async for" to be).<br>
<br>
Cheers,<br>
Nick.<br>
<br>
--<br>
Nick Coghlan | <a href="mailto:ncoghlan@gmail.com">ncoghlan@gmail.com</a> | Brisbane, Australia<br>
_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">http://python.org/psf/codeofconduct/</a></blockquote></div>