PEP 530: Asynchronous Comprehensions
Hi, Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6. I have a half-working implementation of the proposal which fully implements all required grammar and AST changes. What's left is to update the compiler to emit correct opcodes for async comprehensions. I'm confident that we can have a fully working patch before the feature freeze. Thank you, Yury PEP: 530 Title: Asynchronous Comprehensions Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury@magic.io> Discussions-To: <python-dev@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2016 Python-Version: 3.6 Post-History: 03-Sep-2016 Abstract ======== PEP 492 and PEP 525 introduce support for native coroutines and asynchronous generators using ``async`` / ``await`` syntax. This PEP proposes to add asynchronous versions of list, set, dict comprehensions and generator expressions. Rationale and Goals =================== Python has extensive support for synchronous comprehensions, allowing to produce lists, dicts, and sets with a simple and concise syntax. We propose implementing similar syntactic constructions for the asynchronous code. To illustrate the readability improvement, consider the following example:: result = [] async for i in aiter(): if i % 2: result.append(i) With the proposed asynchronous comprehensions syntax, the above code becomes as short as:: result = [i async for i in aiter() if i % 2] The PEP also makes it possible to use the ``await`` expressions in all kinds of comprehensions:: result = [await fun() for fun in funcs] Specification ============= Asynchronous Comprehensions --------------------------- We propose to allow using ``async for`` inside list, set and dict comprehensions. Pending PEP 525 approval, we can also allow creation of asynchronous generator expressions. Examples: * set comprehension: ``{i async for i in agen()}``; * list comprehension: ``[i async for i in agen()]``; * dict comprehension: ``{i: i ** 2 async for i in agen()}``; * generator expression: ``(i ** 2 async for i in agen())``. It is allowed to use ``async for`` along with ``if`` and ``for`` clauses in asynchronous comprehensions and generator expressions:: dataset = {data for line in aiter() async for data in line if check(data)} Asynchronous comprehensions are only allowed inside an ``async def`` function. In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed. ``await`` in Comprehensions --------------------------- We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions:: result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs} result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs} This is only valid in ``async def`` function body. Grammar Updates --------------- The proposal requires one change on the grammar level: adding the optional "async" keyword to ``comp_for``:: comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter] The ``comprehension`` AST node will have the new ``is_async`` argument. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On Sun, Sep 4, 2016 at 9:31 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and basynchronous generator expressions in Python 3.6.
Looks good to me! No content comments, and +1 on the proposal. One copyedit suggestion:
In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed.
Does this want a comma after "3.7"? Otherwise, LGTM. Bring on the asynciness! ChrisA
Hi Chris, On 2016-09-03 5:34 PM, Chris Angelico wrote:
On Sun, Sep 4, 2016 at 9:31 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and basynchronous generator expressions in Python 3.6. Looks good to me! No content comments, and +1 on the proposal.
Thanks!
One copyedit suggestion:
In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed. Does this want a comma after "3.7"?
Fixed! Yury
In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed.
Would this mean that with this PEP I can write a for loop like this? for (await?) item in async_generator(): ... code Or am I misunderstanding something? footnote: I am not really up to date with the async PEPs, so please correct me if this has already been discussed somewhere else.
Hi Matthias, On 2016-09-03 5:55 PM, Matthias welp wrote:
In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed.
Would this mean that with this PEP I can write a for loop like this?
for (await?) item in async_generator(): ... code
Or am I misunderstanding something?
No, this is an illegal syntax and will stay that way. The PEP allows to do this: [await item for item in generator()] and this: [await item async for item in async_generator()] The idea is to remove any kind of restrictions that we currently have on async/await. A lot of people just assume that PEP 530 is already implemented in 3.5. Yury
On 4 September 2016 at 01:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6.
Interesting proposal. Would be nice to have this! I have one question:
``await`` in Comprehensions ---------------------------
We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions::
result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs}
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
This is only valid in ``async def`` function body.
Do I understand correctly that the limitation that they are allowed only in async def is because await binds to the enclosing coroutine? There is an old "bug" (some people call this a feature) http://bugs.python.org/issue10544 If one uses yield in a comprehension, then it leads to unexpected results:
def f(): ... yield ... res = [(yield) for i in range(3)] ... return res ... fg = f() next(fg) next(fg) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: <generator object f.<locals>.<listcomp> at 0x7fe05b37a498>
This function yields only once and then returns another generator. This is because the yield in comprehension "lives" in an auxiliary function scope used to make the comprehension. So that such comprehension are even allowed outside function body:
[(yield) for i in range(3)] <generator object <listcomp> at 0x7fe05b3d41f0> [(yield from range(3)) for i in range(3)] <generator object <listcomp> at 0x7fe05ade11f0>
Do I understand correctly that this is not the case with asynchronous comprehensions? If this is not the case, then I like this, but this will be inconsistent with normal comprehensions. -- Ivan
Hi Ivan, On 2016-09-05 3:57 PM, Ivan Levkivskyi wrote:
On 4 September 2016 at 01:31, Yury Selivanov <yselivanov.ml@gmail.com <mailto:yselivanov.ml@gmail.com>> wrote:
Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6.
Interesting proposal. Would be nice to have this!
I have one question:
``await`` in Comprehensions ---------------------------
We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions::
result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs}
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
This is only valid in ``async def`` function body.
Do I understand correctly that the limitation that they are allowed only in async def is because await binds to the enclosing coroutine?
Correct.
There is an old "bug" (some people call this a feature) http://bugs.python.org/issue10544 If one uses yield in a comprehension, then it leads to unexpected results:
def f(): ... yield ... res = [(yield) for i in range(3)] ... return res ... fg = f() next(fg) next(fg) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: <generator object f.<locals>.<listcomp> at 0x7fe05b37a498>
This function yields only once and then returns another generator. This is because the yield in comprehension "lives" in an auxiliary function scope used to make the comprehension. So that such comprehension are even allowed outside function body:
[(yield) for i in range(3)] <generator object <listcomp> at 0x7fe05b3d41f0> [(yield from range(3)) for i in range(3)] <generator object <listcomp> at 0x7fe05ade11f0>
Do I understand correctly that this is not the case with asynchronous comprehensions?
If this is not the case, then I like this, but this will be inconsistent with normal comprehensions.
I'm not sure what will happen here. Will have an update once a reference implementation is ready. Seems that the bug you're referring to is something that should be just fixed. Yury
On 6 September 2016 at 11:22, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On 2016-09-05 3:57 PM, Ivan Levkivskyi wrote:
There is an old "bug" (some people call this a feature) http://bugs.python.org/issue10544 If one uses yield in a comprehension, then it leads to unexpected results:
I'm not sure what will happen here. Will have an update once a reference implementation is ready. Seems that the bug you're referring to is something that should be just fixed.
I was going to say that the problem with that last sentence is the "just" part, as for a long time, even though the more desirable behaviour was relatively clear ("make it work the way it did in Python 2"), we didn't have a readily available means of doing that. However, the last time I thought seriously about this problem was before we added "yield from", and that may actually be enough to change the situation. Specifically, the problem is that comprehensions and generator expressions in 3.x create a full closure, so code like: def example(): L = [(yield) for i in range(2)] print(L) is implicitly doing: def example(): def _bad_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = _bad_gen(range(2)) print(L) Prior to yield from, the code generator had no easy way to fix that, since it couldn't delegate yield operations to the underlying generator. However, given PEP 380, the code generator could potentially detect these situations, and implicitly use "yield from" to hoist the generator behaviour up to the level of the containing function, exactly as happened in Python 2: def example(): def _nested_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = yield from _nested_gen(range(2)) print(L)
gen = example() next(gen) gen.send(1) gen.send(2) [1, 2] Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
I don't think there's anything we can do about generator expressions short of PEP 530 itself though - they already misbehave in Python 2, and misbehave in exactly the same way in Python 3, since there's no way for the interpreter to tell the difference at runtime between the implicit yields from the generator expression itself, and any explicit yields used in generator subexpressions. By contrast, PEP 530 can work, since it doesn't *want* to tell the difference between the implicit awaits and explicit awaits, and the await and yield channels are already distinguished at the language level. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Yury, just for my understanding: On 04.09.2016 01:31, Yury Selivanov wrote:
We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions::
result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs}
This will produce normal lists, sets and dicts, right? Whereas the following will produce some sort of async lists, sets, and dicts?
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
If so, how do I read values from an async list/set/dict? Sven
Hi Sven, On 2016-09-05 4:27 PM, Sven R. Kunze wrote:
Hi Yury,
just for my understanding:
On 04.09.2016 01:31, Yury Selivanov wrote:
We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions::
result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs}
This will produce normal lists, sets and dicts, right?
Right.
Whereas the following will produce some sort of async lists, sets, and dicts?
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
If so, how do I read values from an async list/set/dict?
Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict. I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable. There is no concept of asynchronous datastructures in Python. Thanks, Yury
Sven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Sep 6, 2016 at 6:46 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Hi Sven,
I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable. using the same object as iterator and async generator, may be not. but isn't it good to write code for sync iteration and async iteration in the same class? so that if I'm writing a library, I can say that the library supports both.
Junior (3rd yr) student at Indian School of Mines,(IIT Dhanbad) Computer Science and Engineering Department ph: +91 9491 383 249 telegram_id: @eightnoteight
Hi Srinivas, On 06.09.2016 05:46, srinivas devaki wrote:
On Tue, Sep 6, 2016 at 6:46 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Hi Sven,
I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable. using the same object as iterator and async generator, may be not. but isn't it good to write code for sync iteration and async iteration in the same class? so that if I'm writing a library, I can say that the library supports both.
oh, wrong territory here! Python async community wants you to write everything twice: for the sync and async case. And don't dare to mentioned code sharing here. They will rip you apart. ;) Just kidding. Of course would it be great to write code only once but Yury want to preserve well-paid Python dev jobs in the industry because everything here needs to be maintained twice then. ;) No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain. Cheers, Sven
On 7 September 2016 at 04:24, Sven R. Kunze <srkunze@mail.de> wrote:
Python async community wants you to write everything twice: for the sync and async case. And don't dare to mentioned code sharing here. They will rip you apart. ;)
Just kidding. Of course would it be great to write code only once but Yury want to preserve well-paid Python dev jobs in the industry because everything here needs to be maintained twice then. ;)
Sven, this is not productive, not funny, and not welcome. Vent your frustrations with the fundamental split between synchronous and explicitly asynchronous software design elsewhere.
No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain.
As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 06.09.2016 20:37, Nick Coghlan wrote:
On 7 September 2016 at 04:24, Sven R. Kunze <srkunze@mail.de> wrote:
Python async community wants you to write everything twice: for the sync and async case. And don't dare to mentioned code sharing here. They will rip you apart. ;)
Just kidding. Of course would it be great to write code only once but Yury want to preserve well-paid Python dev jobs in the industry because everything here needs to be maintained twice then. ;) Sven, this is not productive, not funny, and not welcome. Vent your frustrations with the fundamental split between synchronous and explicitly asynchronous software design elsewhere.
Don't make a mistake here, Nick. I take this with some humor as it does not concern me in production. It's interesting to see though that people new to the discussion detect this obvious issue very fast.
No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain. As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous.
Of course "async" switches to async mode. But that was not the question. I asked WHY that's necessary not what it does. I already noted that Python can detect when to make the switch without a marker. And you fail to explain where the issue with this point of view is. Sven PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc. Is there something wrong with my config or is this deliberate on your part?
On 9/6/2016 2:54 PM, Sven R. Kunze wrote:
PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc.
For Thunderbird, this is normal behavior. I suspect you would find the same if your tried 'replay all' to multiple messages.
Is there something wrong with my config
No
or is this deliberate on your part?
No -- Terry Jan Reedy
On 06.09.2016 21:45, Terry Reedy wrote:
On 9/6/2016 2:54 PM, Sven R. Kunze wrote:
PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc.
For Thunderbird, this is normal behavior. I suspect you would find the same if your tried 'replay all' to multiple messages.
Actually no. Replying to your message for example leads to "Reply to List". That's odd. Sven
On 7 September 2016 at 04:54, Sven R. Kunze <srkunze@mail.de> wrote:
On 06.09.2016 20:37, Nick Coghlan wrote:
As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous.
Of course "async" switches to async mode. But that was not the question. I asked WHY that's necessary not what it does. I already noted that Python can detect when to make the switch without a marker. And you fail to explain where the issue with this point of view is.
The Python *runtime* can tell whether the result of an expression is a normal iterator or an asynchronous iterator, but the Python *compiler* can't. So at compile time, it has to decide what code to emit for the four different scenarios Anthony listed: # Normal comprehension, 100% synchronous and blocking squares = [i**i for i in range(10)] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value results = [await result for result in submit_requests()] # Non-blocking/async iterator that suspends before producing "normal" values results = [result async for submit_requests_and_wait_for_results()] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value results = [await result async for submit_requests_asynchronously()] And hence needs the "async" keyword to detect when it has the latter two cases rather than the first two. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 September 2016 at 04:54, Sven R. Kunze <srkunze@mail.de> wrote:
On 06.09.2016 20:37, Nick Coghlan wrote:
As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous.
Of course "async" switches to async mode. But that was not the question. I asked WHY that's necessary not what it does. I already noted that Python can detect when to make the switch without a marker. And you fail to explain where the issue with this point of view is.
The Python *runtime* can tell whether the result of an expression is a normal iterator or an asynchronous iterator, but the Python *compiler* can't.
Yes, so to remove "async" from the syntax, this would be handled at runtime. But there's no way this PEP is going to do that, so I agree there's no point in discussing this here. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Tue, Sep 6, 2016 at 9:24 PM, Sven R. Kunze <srkunze@mail.de> wrote: [...]
No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain.
I'm sure he would explain, but it seems I was first ;) [last-minute edit: no, Nick was first, but this is a slightly different angle]. First, the "async" gets inherited from PEP 492, so this has actually already been decided on. While not strictly necessary for a syntax for "async for", it makes it more explicit what happens under the hood -- that __a*__ methods are called and awaited, instead of simply calling __iter__/__next__ etc. as in regular loops/comprehensions. Not a lot to debate, I guess. No surprises here, just implementation work. -- Koos
Cheers, Sven
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On 06.09.2016 20:45, Koos Zevenhoven wrote:
On Tue, Sep 6, 2016 at 9:24 PM, Sven R. Kunze <srkunze@mail.de> wrote: [...]
No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain. I'm sure he would explain, but it seems I was first ;) [last-minute edit: no, Nick was first, but this is a slightly different angle].
First, the "async" gets inherited from PEP 492, so this has actually already been decided on. While not strictly necessary for a syntax for "async for", it makes it more explicit what happens under the hood -- that __a*__ methods are called and awaited, instead of simply calling __iter__/__next__ etc. as in regular loops/comprehensions.
Not a lot to debate, I guess. No surprises here, just implementation work.
Of course, I would do the same. I value consistency a lot (but the issue here remains). :) Cheers, Sven
On 06.09.2016 03:16, Yury Selivanov wrote:
Whereas the following will produce some sort of async lists, sets, and dicts?
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
If so, how do I read values from an async list/set/dict?
Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict.
So, what's the "async" good for then?
I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable.
Why do you implement it then? :D Put it differently, why are you sceptic about it?
There is no concept of asynchronous datastructures in Python.
I thought so, that's why I asked. ;) "async def" gives me something async, so I assumed it to be the case here as well. Cheers, Sven
On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze <srkunze@mail.de> wrote:
On 06.09.2016 03:16, Yury Selivanov wrote:
Whereas the following will produce some sort of async lists, sets, and
dicts?
result = [await fun() async for fun in funcs]
result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
If so, how do I read values from an async list/set/dict?
Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict.
So, what's the "async" good for then?
Maybe I'm off base here, but my understanding is the `async for` version would allow for suspension during the actual iteration, ie. using the __a*__ protocol methods, and not just by awaiting on the produced item itself. IOW, `[await ... async for ...]` will suspend at least twice, once during iteration using the __a*__ protocols and then again awaiting on the produced item, whereas `[await ... for ...]` will synchronously produce items and then suspend on them. So to use the async + await version, your (async) iterator must return awaitables to satisfy the `async for` part with then produce another awaitable we explicitly `await` on. Can someone confirm this understanding? And also that all 4 combinations are possible, each with different meaning: # Normal comprehension, 100% synchronous and blocking [... for ...] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value [await ... for ...] # Non-blocking/async iterator that suspends before producing "normal" values [... async for ...] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value [await ... async for ...] Is this correct? -- C Anthony
On 2016-09-06 9:40 AM, C Anthony Risinger wrote:
On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze <srkunze@mail.de <mailto:srkunze@mail.de>> wrote:
On 06.09.2016 03:16, Yury Selivanov wrote:
Whereas the following will produce some sort of async lists, sets, and dicts?
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
If so, how do I read values from an async list/set/dict?
Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict.
So, what's the "async" good for then?
Maybe I'm off base here, but my understanding is the `async for` version would allow for suspension during the actual iteration, ie. using the __a*__ protocol methods, and not just by awaiting on the produced item itself.
IOW, `[await ... async for ...]` will suspend at least twice, once during iteration using the __a*__ protocols and then again awaiting on the produced item, whereas `[await ... for ...]` will synchronously produce items and then suspend on them. So to use the async + await version, your (async) iterator must return awaitables to satisfy the `async for` part with then produce another awaitable we explicitly `await` on.
Can someone confirm this understanding? And also that all 4 combinations are possible, each with different meaning:
# Normal comprehension, 100% synchronous and blocking [... for ...]
# Blocking/sync iterator producing awaitables which suspend before producing a "normal" value [await ... for ...]
# Non-blocking/async iterator that suspends before producing "normal" values [... async for ...]
# Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value [await ... async for ...]
Is this correct?
All correct. I'll update the PEP to better clarify the semantics. Yury
On 06.09.2016 19:38, Yury Selivanov wrote:
On 2016-09-06 9:40 AM, C Anthony Risinger wrote:
On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze <srkunze@mail.de <mailto:srkunze@mail.de>> wrote:
So, what's the "async" good for then?
Maybe I'm off base here, but my understanding is the `async for` version would allow for suspension during the actual iteration, ie. using the __a*__ protocol methods, and not just by awaiting on the produced item itself.
IOW, `[await ... async for ...]` will suspend at least twice, once during iteration using the __a*__ protocols and then again awaiting on the produced item, whereas `[await ... for ...]` will synchronously produce items and then suspend on them. So to use the async + await version, your (async) iterator must return awaitables to satisfy the `async for` part with then produce another awaitable we explicitly `await` on.
Can someone confirm this understanding? And also that all 4 combinations are possible, each with different meaning:
# Normal comprehension, 100% synchronous and blocking [... for ...]
# Blocking/sync iterator producing awaitables which suspend before producing a "normal" value [await ... for ...]
# Non-blocking/async iterator that suspends before producing "normal" values [... async for ...]
# Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value [await ... async for ...]
Is this correct?
All correct. I'll update the PEP to better clarify the semantics.
Still don't understand what the "async" is good for. Seems redundant to me. Sven
On Tue, Sep 6, 2016 at 9:27 AM, Sven R. Kunze <srkunze@mail.de> wrote:
Hi Yury,
just for my understanding:
On 04.09.2016 01:31, Yury Selivanov wrote:
We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions::
result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs}
This will produce normal lists, sets and dicts, right?
Whereas the following will produce some sort of async lists, sets, and dicts?
result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs}
If so, how do I read values from an async list/set/dict?
AIUI they won't return "async lists" etc; what they'll do is asynchronously return list/set/dict. Imagine an async database query that returns Customer objects. You could get their names thus: names = [cust.name async for cust in find_customers()] And you could enumerate their invoices (another database query) with a double-async comprehension: invoices = {cust.name: await cust.list_invoices() async for cust in find_customers()} As always, you can unroll a comprehension to the equivalent statement form. _tmp = {} async for cust in find_customers(): _tmp[cust.name] = await cust.list_invoices() invoices = _tmp or with the original example: _tmp = [] async for fun in funcs: _tmp.append(await fun()) result = _tmp Hope that helps. ChrisA
On 4 September 2016 at 09:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
With the proposed asynchronous comprehensions syntax, the above code becomes as short as::
result = [i async for i in aiter() if i % 2]
After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can: result = [i for i in async aiter() if i % 2] I think the readability gain from that approach becomes even clearer with nested loops: result = [x for aiterable in async outer() for x in async aiterable] vs the current: result = [x async for aiterable in outer() async for x in async aiterable] In the first form, "async" is clearly a pre-qualifier on "outer()" and "aiterable", indicating they need to be asynchronous iterators rather than synchronous ones. By contrast, in the current form, the first "async" reads like a post-qualifer on "x" (it isn't, it modifies how outer() is handled in the outer loop), while the second looks like a post-qualifier on "outer()" (it isn't, it modified how aiterable is handled in the inner loop) If that means having to postpone full async comprehensions until "async" becomes a proper keyword in 3.7 and only adding "await in comprehensions and generator expressions" support to 3.6, that seems reasonable to me Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:31 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 4 September 2016 at 09:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
With the proposed asynchronous comprehensions syntax, the above code becomes as short as::
result = [i async for i in aiter() if i % 2]
After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can:
result = [i for i in async aiter() if i % 2]
Please, no. It may be totally correct from English grammar POV but brings different syntax from regular async for statement, e.g. async for row in db.execute(...): pass Currently regular comprehensions are pretty similar to `for` loop.
Why async comprehensions should look different from `async for` counterpart?
I think the readability gain from that approach becomes even clearer with nested loops:
result = [x for aiterable in async outer() for x in async aiterable]
vs the current:
result = [x async for aiterable in outer() async for x in async aiterable]
In the first form, "async" is clearly a pre-qualifier on "outer()" and "aiterable", indicating they need to be asynchronous iterators rather than synchronous ones.
By contrast, in the current form, the first "async" reads like a post-qualifer on "x" (it isn't, it modifies how outer() is handled in the outer loop), while the second looks like a post-qualifier on "outer()" (it isn't, it modified how aiterable is handled in the inner loop)
If that means having to postpone full async comprehensions until "async" becomes a proper keyword in 3.7 and only adding "await in comprehensions and generator expressions" support to 3.6, that seems reasonable to me
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
On 7 September 2016 at 21:37, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
On Wed, Sep 7, 2016 at 2:31 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can:
result = [i for i in async aiter() if i % 2]
Please, no. It may be totally correct from English grammar POV but brings different syntax from regular async for statement, e.g. async for row in db.execute(...): pass
The new issue that's specific to comprehensions is that the statement form doesn't have the confounding factor of having an expression to the left of it. Thus, the prefix is unambiguous and can be read as modifying the entire statement rather than just the immediately following keyword: async for row in db.execute(...): process(row) It's also pragmatically necessary right now due to the sleight of hand that Yury used in the code generation pipeline to bring in "async def", "async with", and "async for" without a __future__ statement, but without making them full keywords either. However, when we convert it to the comprehension form, a parsing ambiguity (for humans) arises that creates an inherent readability problem: [process(row) async for row in db.execute(...)] When reading that, is "async" a postfix operator being used in a normal comprehension (wrong, but somewhat plausible)? Or is it part of a compound keyword with "for" that modifies the iteration behaviour of that part of the comprehension (the correct interpretation)? [(process(row) async) for row in db.execute(...)] [process(row) (async for) row in db.execute(...)] The postfix operator interpretation is just plain wrong, but even the correct interpretation as a compound keyword sits between two expressions *neither* of which is the one being modified (that would be "db.execute()") By contrast, if the addition of full async comprehensions is deferred to 3.7 (when async becomes a true keyword), then the infix spelling can be permitted in both the statement and comprehension forms: for row in async db.execute(...): process(row) [process(row) for row in async db.execute(...)] with the prefix spelling of the statement form retained solely for backwards compatibility purposes (just as we retain "from __future__ import feature" flags even after the feature has become the default behaviour). The beauty of the infix form is that it *doesn't matter* whether someone reads it as a compound keyword with "in" or as a prefix modifying the following expression: [process(row) for row (in async) db.execute(...)] [process(row) for row in (async db.execute(...))] In both cases, it clearly suggests something special about the way "db.execute()" is going to be handled, which is the correct interpretation.
Currently regular comprehensions are pretty similar to `for` loop. Why async comprehensions should look different from `async for` counterpart?
Regular "for" loops don't have the problem of their introductory keyword being written as two words, as that's the culprit that creates the ambiguity when you add an expression to the left of it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: [...]
The new issue that's specific to comprehensions is that the statement form doesn't have the confounding factor of having an expression to the left of it. Thus, the prefix is unambiguous and can be read as modifying the entire statement rather than just the immediately following keyword:
async for row in db.execute(...): process(row)
It's also pragmatically necessary right now due to the sleight of hand that Yury used in the code generation pipeline to bring in "async def", "async with", and "async for" without a __future__ statement, but without making them full keywords either.
I don't think this issue strictly has anything to do with where that "async" is put in the syntax, as long as it's used within the definition of an async function: async def function(): # in here, is effectively "async" a keyword. Of course, in French, this would be: def function async(): # dedans, "async" est effectivement un mot clé I'm sure someone will be able to correct my French, though. [and Nick writes:]
[process(row) async for row in db.execute(...)]
When reading that, is "async" a postfix operator being used in a normal comprehension (wrong, but somewhat plausible)? Or is it part of a compound keyword with "for" that modifies the iteration behaviour of that part of the comprehension (the correct interpretation)?
[(process(row) async) for row in db.execute(...)] [process(row) (async for) row in db.execute(...)]
The postfix operator interpretation is just plain wrong, but even the correct interpretation as a compound keyword sits between two expressions *neither* of which is the one being modified (that would be "db.execute()")
By contrast, if the addition of full async comprehensions is deferred to 3.7 (when async becomes a true keyword), then the infix spelling can be permitted in both the statement and comprehension forms:
That's an interesting suggestion. What exactly is the relation between deferring this PEP and permitting the infix spelling? This would make it more obvious at a first glance, whether something is a with statement or for loop. The word "async" there is still not very easy to miss, especially with highlighted syntax. I didn't realize (or had forgotten) that PEP 492 is provisional.
for row in async db.execute(...): process(row)
[process(row) for row in async db.execute(...)]
with the prefix spelling of the statement form retained solely for backwards compatibility purposes (just as we retain "from __future__ import feature" flags even after the feature has become the default behaviour).
The beauty of the infix form is that it *doesn't matter* whether someone reads it as a compound keyword with "in" or as a prefix modifying the following expression:
[process(row) for row (in async) db.execute(...)] [process(row) for row in (async db.execute(...))]
In both cases, it clearly suggests something special about the way "db.execute()" is going to be handled, which is the correct interpretation.
And db.execute is an async iterarable after all, so "async" is a suitable adjective for db.execute(...). -- Koos [...]
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On 8 September 2016 at 00:57, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: [...]
The new issue that's specific to comprehensions is that the statement form doesn't have the confounding factor of having an expression to the left of it. Thus, the prefix is unambiguous and can be read as modifying the entire statement rather than just the immediately following keyword:
async for row in db.execute(...): process(row)
It's also pragmatically necessary right now due to the sleight of hand that Yury used in the code generation pipeline to bring in "async def", "async with", and "async for" without a __future__ statement, but without making them full keywords either.
I don't think this issue strictly has anything to do with where that "async" is put in the syntax, as long as it's used within the definition of an async function:
async def function(): # in here, is effectively "async" a keyword.
Good point - I was thinking there was additional cleverness around "async for" and "async with" as well, but you're right that it's specifically "async def" that enables the pseudo-keyword behaviour.
[and Nick writes:]
[process(row) async for row in db.execute(...)]
When reading that, is "async" a postfix operator being used in a normal comprehension (wrong, but somewhat plausible)? Or is it part of a compound keyword with "for" that modifies the iteration behaviour of that part of the comprehension (the correct interpretation)?
[(process(row) async) for row in db.execute(...)] [process(row) (async for) row in db.execute(...)]
The postfix operator interpretation is just plain wrong, but even the correct interpretation as a compound keyword sits between two expressions *neither* of which is the one being modified (that would be "db.execute()")
By contrast, if the addition of full async comprehensions is deferred to 3.7 (when async becomes a true keyword), then the infix spelling can be permitted in both the statement and comprehension forms:
That's an interesting suggestion. What exactly is the relation between deferring this PEP and permitting the infix spelling?
Just a mistake on my part regarding how we were currently handling "async" within "async def" statements. With that mistake corrected, there may not be any need to defer the suggestion, since it already behaves as a keyword in the context where it matters. That said, we *are* doing some not-normal things in the code generation pipeline to enable the pseudo-keyword behaviour, so I also wouldn't be surprised if there was a practical limitation on allowing the "async" to appear after the "in" rather than before the "for" prior to 3.7. It's also worth reviewing the minimalist grammar changes in PEP 492 and the associated discussion about "async def" vs "def async": * https://www.python.org/dev/peps/pep-0492/#grammar-updates * https://www.python.org/dev/peps/pep-0492/#why-async-def-and-not-def-async Changing "for_stmt" to allow the "for TARGET in [ASYNC] expr" spelling isn't as tidy a modification as just allowing ASYNC in front of any of def_stmt, for_stmt and with_stmt.
This would make it more obvious at a first glance, whether something is a with statement or for loop. The word "async" there is still not very easy to miss, especially with highlighted syntax.
I didn't realize (or had forgotten) that PEP 492 is provisional.
Right, and one of the reasons for that was because we hadn't fully worked through the implications for comprehensions and generator expressions at the time. Now that I see the consequences of attempting to transfer the "async keyword is a statement qualifier " notion to the expression form, I think we may need to tweak things a bit :)
The beauty of the infix form is that it *doesn't matter* whether someone reads it as a compound keyword with "in" or as a prefix modifying the following expression:
[process(row) for row (in async) db.execute(...)] [process(row) for row in (async db.execute(...))]
In both cases, it clearly suggests something special about the way "db.execute()" is going to be handled, which is the correct interpretation.
And db.execute is an async iterarable after all, so "async" is a suitable adjective for db.execute(...).
Exactly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 4 September 2016 at 09:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
With the proposed asynchronous comprehensions syntax, the above code becomes as short as::
result = [i async for i in aiter() if i % 2]
After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can:
result = [i for i in async aiter() if i % 2]
I agree this would be better, but the difference compared to PEP-492 async for loops (and even async with statements) would be awful :S. -- Koos
I think the readability gain from that approach becomes even clearer with nested loops:
result = [x for aiterable in async outer() for x in async aiterable]
vs the current:
result = [x async for aiterable in outer() async for x in async aiterable]
In the first form, "async" is clearly a pre-qualifier on "outer()" and "aiterable", indicating they need to be asynchronous iterators rather than synchronous ones.
By contrast, in the current form, the first "async" reads like a post-qualifer on "x" (it isn't, it modifies how outer() is handled in the outer loop), while the second looks like a post-qualifier on "outer()" (it isn't, it modified how aiterable is handled in the inner loop)
If that means having to postpone full async comprehensions until "async" becomes a proper keyword in 3.7 and only adding "await in comprehensions and generator expressions" support to 3.6, that seems reasonable to me
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
participants (12)
-
Andrew Svetlov -
C Anthony Risinger -
Chris Angelico -
Greg Ewing -
Ivan Levkivskyi -
Koos Zevenhoven -
Matthias welp -
Nick Coghlan -
srinivas devaki -
Sven R. Kunze -
Terry Reedy -
Yury Selivanov