On 29 June 2015 at 16:44, Paul Sokolovsky <pmiscml@gmail.com> wrote:
Hello,
On Sun, 28 Jun 2015 18:14:20 -0400 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On 2015-06-28 11:52 AM, Paul Sokolovsky wrote:
There is also the problem that one cannot easily feed a queue,
asynchronous generator, or any asynchronous iterator to a simple synchronous consumer like sum() or list() or "".join(). It would be nice if there was a way to wrap them to asynchronous ones when needed – something like (async sum)(asynchronously_produced_numbers()). All that is easily achievable with classical Python coroutines, not with asyncio garden variety of coroutines, which lately were casted into a language level with async/await disablers:
def coro1(): yield 1 yield 2 yield 3
def coro2(): yield from coro1() yield 4 yield 5
print(sum(coro2()))
You have easily achieved combining two generators with 'yield from' and feeding that to 'sum' builtin.
Right, the point here was that PEP492, banning usage of "yield" in coroutines, doesn't help with such simple and basic usage of them. And then I again can say what I said during initial discussion of PEP492: I have dual feeling about it: promise of making coroutines easier and more user friendly is worth all support, but step of limiting basic language usage in them doesn't seem good. What me and other people can do then is just trust that you guys know what you do and PEP492 will be just first step. But bottom line is that I personally don't find async/await worthy to use for now - it's better to stick to old good yield from, until the promise of truly better coroutines is delivered.
The purpose of PEP 492 is to fundamentally split the asynchronous IO use case away from traditional generators. If you're using native coroutines, you MUST have an event loop, or at least be using something like asyncio.run_until_complete() (which spins up a scheduler for the duration). If you're using generators without @types.coroutine or @asyncio.coroutine (or the equivalent for tulip, Twisted, etc), then you're expecting a synchronous driver rather than an asynchronous one. This isn't an accident, or something that will change at some point in the future, it's the entire point of the exercise: having it be obvious both how you're meant to interact with something based on the way it's defined, and how you factor outside subcomponents of the algorithm. Asynchronous driver? Use a coroutine. Synchronous driver? Use a generator. What we *don't* have are consumption functions that have an implied "async for" inside them - functions like sum(), any(), all(), etc are all synchronous drivers. The other key thing we don't have yet? Asynchronous comprehensions. A peak at the various options for parallel execution described in https://docs.python.org/3/library/concurrent.futures.html documentation helps illustrate why: once we're talking about applying reduction functions to asynchronous iterables we're getting into full-blown language-level-support-for-MapReduce territory. Do the substeps still need to be executed in series? Or can the substeps be executed in parallel, and either accumulated in iteration order or as they become available? Does it perhaps make sense to *require* that the steps be executable in parallel, such that we could write the following: result = sum(x*x for async x in coro) Where the reduction step remains synchronous, but we can mark the comprehension/map step as asynchronous, and have that change the generated code to create an implied lambda for the "lambda x: x*x" calculation, dispatch all of those to the scheduler at once, and then produce the results one at a time? The answer to that is "quite possibly, but we don't really know yet". PEP 492 is enough to address some major comprehensibility challenges that exist around generators-as-coroutines. It *doesn't* bring language level support for parallel MapReduce to Python, but it *does* bring some interesting new building blocks for folks to play around with in that regard (in particular, figuring out what we want the comprehension level semantics of "async for" to be). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia