PEP 530: Asynchronous Comprehensions
Hi, Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6. I have a half-working implementation of the proposal which fully implements all required grammar and AST changes. What's left is to update the compiler to emit correct opcodes for async comprehensions. I'm confident that we can have a fully working patch before the feature freeze. Thank you, Yury PEP: 530 Title: Asynchronous Comprehensions Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury@magic.io> Discussions-To: <python-dev@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2016 Python-Version: 3.6 Post-History: 03-Sep-2016 Abstract ======== PEP 492 and PEP 525 introduce support for native coroutines and asynchronous generators using ``async`` / ``await`` syntax. This PEP proposes to add asynchronous versions of list, set, dict comprehensions and generator expressions. Rationale and Goals =================== Python has extensive support for synchronous comprehensions, allowing to produce lists, dicts, and sets with a simple and concise syntax. We propose implementing similar syntactic constructions for the asynchronous code. To illustrate the readability improvement, consider the following example:: result = [] async for i in aiter(): if i % 2: result.append(i) With the proposed asynchronous comprehensions syntax, the above code becomes as short as:: result = [i async for i in aiter() if i % 2] The PEP also makes it possible to use the ``await`` expressions in all kinds of comprehensions:: result = [await fun() for fun in funcs] Specification ============= Asynchronous Comprehensions --------------------------- We propose to allow using ``async for`` inside list, set and dict comprehensions. Pending PEP 525 approval, we can also allow creation of asynchronous generator expressions. Examples: * set comprehension: ``{i async for i in agen()}``; * list comprehension: ``[i async for i in agen()]``; * dict comprehension: ``{i: i ** 2 async for i in agen()}``; * generator expression: ``(i ** 2 async for i in agen())``. It is allowed to use ``async for`` along with ``if`` and ``for`` clauses in asynchronous comprehensions and generator expressions:: dataset = {data for line in aiter() async for data in line if check(data)} Asynchronous comprehensions are only allowed inside an ``async def`` function. In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed. ``await`` in Comprehensions --------------------------- We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions:: result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs} result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs} This is only valid in ``async def`` function body. Grammar Updates --------------- The proposal requires one change on the grammar level: adding the optional "async" keyword to ``comp_for``:: comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter] The ``comprehension`` AST node will have the new ``is_async`` argument. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On Sun, Sep 4, 2016 at 9:31 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and basynchronous generator expressions in Python 3.6.
Looks good to me! No content comments, and +1 on the proposal. One copyedit suggestion:
Does this want a comma after "3.7"? Otherwise, LGTM. Bring on the asynciness! ChrisA
Would this mean that with this PEP I can write a for loop like this? for (await?) item in async_generator(): ... code Or am I misunderstanding something? footnote: I am not really up to date with the async PEPs, so please correct me if this has already been discussed somewhere else.
Hi Matthias, On 2016-09-03 5:55 PM, Matthias welp wrote:
No, this is an illegal syntax and will stay that way. The PEP allows to do this: [await item for item in generator()] and this: [await item async for item in async_generator()] The idea is to remove any kind of restrictions that we currently have on async/await. A lot of people just assume that PEP 530 is already implemented in 3.5. Yury
On 4 September 2016 at 01:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6.
Interesting proposal. Would be nice to have this! I have one question:
Do I understand correctly that the limitation that they are allowed only in async def is because await binds to the enclosing coroutine? There is an old "bug" (some people call this a feature) http://bugs.python.org/issue10544 If one uses yield in a comprehension, then it leads to unexpected results:
This function yields only once and then returns another generator. This is because the yield in comprehension "lives" in an auxiliary function scope used to make the comprehension. So that such comprehension are even allowed outside function body:
Do I understand correctly that this is not the case with asynchronous comprehensions? If this is not the case, then I like this, but this will be inconsistent with normal comprehensions. -- Ivan
On 6 September 2016 at 11:22, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I was going to say that the problem with that last sentence is the "just" part, as for a long time, even though the more desirable behaviour was relatively clear ("make it work the way it did in Python 2"), we didn't have a readily available means of doing that. However, the last time I thought seriously about this problem was before we added "yield from", and that may actually be enough to change the situation. Specifically, the problem is that comprehensions and generator expressions in 3.x create a full closure, so code like: def example(): L = [(yield) for i in range(2)] print(L) is implicitly doing: def example(): def _bad_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = _bad_gen(range(2)) print(L) Prior to yield from, the code generator had no easy way to fix that, since it couldn't delegate yield operations to the underlying generator. However, given PEP 380, the code generator could potentially detect these situations, and implicitly use "yield from" to hoist the generator behaviour up to the level of the containing function, exactly as happened in Python 2: def example(): def _nested_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = yield from _nested_gen(range(2)) print(L)
I don't think there's anything we can do about generator expressions short of PEP 530 itself though - they already misbehave in Python 2, and misbehave in exactly the same way in Python 3, since there's no way for the interpreter to tell the difference at runtime between the implicit yields from the generator expression itself, and any explicit yields used in generator subexpressions. By contrast, PEP 530 can work, since it doesn't *want* to tell the difference between the implicit awaits and explicit awaits, and the await and yield channels are already distinguished at the language level. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Sven, On 2016-09-05 4:27 PM, Sven R. Kunze wrote:
Right.
Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict. I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable. There is no concept of asynchronous datastructures in Python. Thanks, Yury
Hi Srinivas, On 06.09.2016 05:46, srinivas devaki wrote:
oh, wrong territory here! Python async community wants you to write everything twice: for the sync and async case. And don't dare to mentioned code sharing here. They will rip you apart. ;) Just kidding. Of course would it be great to write code only once but Yury want to preserve well-paid Python dev jobs in the industry because everything here needs to be maintained twice then. ;) No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain. Cheers, Sven
On 7 September 2016 at 04:24, Sven R. Kunze <srkunze@mail.de> wrote:
Sven, this is not productive, not funny, and not welcome. Vent your frustrations with the fundamental split between synchronous and explicitly asynchronous software design elsewhere.
As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 06.09.2016 20:37, Nick Coghlan wrote:
Don't make a mistake here, Nick. I take this with some humor as it does not concern me in production. It's interesting to see though that people new to the discussion detect this obvious issue very fast.
Of course "async" switches to async mode. But that was not the question. I asked WHY that's necessary not what it does. I already noted that Python can detect when to make the switch without a marker. And you fail to explain where the issue with this point of view is. Sven PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc. Is there something wrong with my config or is this deliberate on your part?
On 9/6/2016 2:54 PM, Sven R. Kunze wrote:
PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc.
For Thunderbird, this is normal behavior. I suspect you would find the same if your tried 'replay all' to multiple messages.
Is there something wrong with my config
No
or is this deliberate on your part?
No -- Terry Jan Reedy
On 7 September 2016 at 04:54, Sven R. Kunze <srkunze@mail.de> wrote:
The Python *runtime* can tell whether the result of an expression is a normal iterator or an asynchronous iterator, but the Python *compiler* can't. So at compile time, it has to decide what code to emit for the four different scenarios Anthony listed: # Normal comprehension, 100% synchronous and blocking squares = [i**i for i in range(10)] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value results = [await result for result in submit_requests()] # Non-blocking/async iterator that suspends before producing "normal" values results = [result async for submit_requests_and_wait_for_results()] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value results = [await result async for submit_requests_asynchronously()] And hence needs the "async" keyword to detect when it has the latter two cases rather than the first two. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes, so to remove "async" from the syntax, this would be handled at runtime. But there's no way this PEP is going to do that, so I agree there's no point in discussing this here. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Tue, Sep 6, 2016 at 9:24 PM, Sven R. Kunze <srkunze@mail.de> wrote: [...]
I'm sure he would explain, but it seems I was first ;) [last-minute edit: no, Nick was first, but this is a slightly different angle]. First, the "async" gets inherited from PEP 492, so this has actually already been decided on. While not strictly necessary for a syntax for "async for", it makes it more explicit what happens under the hood -- that __a*__ methods are called and awaited, instead of simply calling __iter__/__next__ etc. as in regular loops/comprehensions. Not a lot to debate, I guess. No surprises here, just implementation work. -- Koos
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On 06.09.2016 03:16, Yury Selivanov wrote:
So, what's the "async" good for then?
I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable.
Why do you implement it then? :D Put it differently, why are you sceptic about it?
There is no concept of asynchronous datastructures in Python.
I thought so, that's why I asked. ;) "async def" gives me something async, so I assumed it to be the case here as well. Cheers, Sven
On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze <srkunze@mail.de> wrote:
Maybe I'm off base here, but my understanding is the `async for` version would allow for suspension during the actual iteration, ie. using the __a*__ protocol methods, and not just by awaiting on the produced item itself. IOW, `[await ... async for ...]` will suspend at least twice, once during iteration using the __a*__ protocols and then again awaiting on the produced item, whereas `[await ... for ...]` will synchronously produce items and then suspend on them. So to use the async + await version, your (async) iterator must return awaitables to satisfy the `async for` part with then produce another awaitable we explicitly `await` on. Can someone confirm this understanding? And also that all 4 combinations are possible, each with different meaning: # Normal comprehension, 100% synchronous and blocking [... for ...] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value [await ... for ...] # Non-blocking/async iterator that suspends before producing "normal" values [... async for ...] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value [await ... async for ...] Is this correct? -- C Anthony
On Tue, Sep 6, 2016 at 9:27 AM, Sven R. Kunze <srkunze@mail.de> wrote:
AIUI they won't return "async lists" etc; what they'll do is asynchronously return list/set/dict. Imagine an async database query that returns Customer objects. You could get their names thus: names = [cust.name async for cust in find_customers()] And you could enumerate their invoices (another database query) with a double-async comprehension: invoices = {cust.name: await cust.list_invoices() async for cust in find_customers()} As always, you can unroll a comprehension to the equivalent statement form. _tmp = {} async for cust in find_customers(): _tmp[cust.name] = await cust.list_invoices() invoices = _tmp or with the original example: _tmp = [] async for fun in funcs: _tmp.append(await fun()) result = _tmp Hope that helps. ChrisA
On 4 September 2016 at 09:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can: result = [i for i in async aiter() if i % 2] I think the readability gain from that approach becomes even clearer with nested loops: result = [x for aiterable in async outer() for x in async aiterable] vs the current: result = [x async for aiterable in outer() async for x in async aiterable] In the first form, "async" is clearly a pre-qualifier on "outer()" and "aiterable", indicating they need to be asynchronous iterators rather than synchronous ones. By contrast, in the current form, the first "async" reads like a post-qualifer on "x" (it isn't, it modifies how outer() is handled in the outer loop), while the second looks like a post-qualifier on "outer()" (it isn't, it modified how aiterable is handled in the inner loop) If that means having to postpone full async comprehensions until "async" becomes a proper keyword in 3.7 and only adding "await in comprehensions and generator expressions" support to 3.6, that seems reasonable to me Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7 September 2016 at 21:37, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
The new issue that's specific to comprehensions is that the statement form doesn't have the confounding factor of having an expression to the left of it. Thus, the prefix is unambiguous and can be read as modifying the entire statement rather than just the immediately following keyword: async for row in db.execute(...): process(row) It's also pragmatically necessary right now due to the sleight of hand that Yury used in the code generation pipeline to bring in "async def", "async with", and "async for" without a __future__ statement, but without making them full keywords either. However, when we convert it to the comprehension form, a parsing ambiguity (for humans) arises that creates an inherent readability problem: [process(row) async for row in db.execute(...)] When reading that, is "async" a postfix operator being used in a normal comprehension (wrong, but somewhat plausible)? Or is it part of a compound keyword with "for" that modifies the iteration behaviour of that part of the comprehension (the correct interpretation)? [(process(row) async) for row in db.execute(...)] [process(row) (async for) row in db.execute(...)] The postfix operator interpretation is just plain wrong, but even the correct interpretation as a compound keyword sits between two expressions *neither* of which is the one being modified (that would be "db.execute()") By contrast, if the addition of full async comprehensions is deferred to 3.7 (when async becomes a true keyword), then the infix spelling can be permitted in both the statement and comprehension forms: for row in async db.execute(...): process(row) [process(row) for row in async db.execute(...)] with the prefix spelling of the statement form retained solely for backwards compatibility purposes (just as we retain "from __future__ import feature" flags even after the feature has become the default behaviour). The beauty of the infix form is that it *doesn't matter* whether someone reads it as a compound keyword with "in" or as a prefix modifying the following expression: [process(row) for row (in async) db.execute(...)] [process(row) for row in (async db.execute(...))] In both cases, it clearly suggests something special about the way "db.execute()" is going to be handled, which is the correct interpretation.
Currently regular comprehensions are pretty similar to `for` loop. Why async comprehensions should look different from `async for` counterpart?
Regular "for" loops don't have the problem of their introductory keyword being written as two words, as that's the culprit that creates the ambiguity when you add an expression to the left of it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: [...]
I don't think this issue strictly has anything to do with where that "async" is put in the syntax, as long as it's used within the definition of an async function: async def function(): # in here, is effectively "async" a keyword. Of course, in French, this would be: def function async(): # dedans, "async" est effectivement un mot clé I'm sure someone will be able to correct my French, though. [and Nick writes:]
That's an interesting suggestion. What exactly is the relation between deferring this PEP and permitting the infix spelling? This would make it more obvious at a first glance, whether something is a with statement or for loop. The word "async" there is still not very easy to miss, especially with highlighted syntax. I didn't realize (or had forgotten) that PEP 492 is provisional.
And db.execute is an async iterarable after all, so "async" is a suitable adjective for db.execute(...). -- Koos [...]
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On 8 September 2016 at 00:57, Koos Zevenhoven <k7hoven@gmail.com> wrote:
Good point - I was thinking there was additional cleverness around "async for" and "async with" as well, but you're right that it's specifically "async def" that enables the pseudo-keyword behaviour.
Just a mistake on my part regarding how we were currently handling "async" within "async def" statements. With that mistake corrected, there may not be any need to defer the suggestion, since it already behaves as a keyword in the context where it matters. That said, we *are* doing some not-normal things in the code generation pipeline to enable the pseudo-keyword behaviour, so I also wouldn't be surprised if there was a practical limitation on allowing the "async" to appear after the "in" rather than before the "for" prior to 3.7. It's also worth reviewing the minimalist grammar changes in PEP 492 and the associated discussion about "async def" vs "def async": * https://www.python.org/dev/peps/pep-0492/#grammar-updates * https://www.python.org/dev/peps/pep-0492/#why-async-def-and-not-def-async Changing "for_stmt" to allow the "for TARGET in [ASYNC] expr" spelling isn't as tidy a modification as just allowing ASYNC in front of any of def_stmt, for_stmt and with_stmt.
Right, and one of the reasons for that was because we hadn't fully worked through the implications for comprehensions and generator expressions at the time. Now that I see the consequences of attempting to transfer the "async keyword is a statement qualifier " notion to the expression form, I think we may need to tweak things a bit :)
Exactly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree this would be better, but the difference compared to PEP-492 async for loops (and even async with statements) would be awful :S. -- Koos
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Sun, Sep 4, 2016 at 9:31 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and basynchronous generator expressions in Python 3.6.
Looks good to me! No content comments, and +1 on the proposal. One copyedit suggestion:
Does this want a comma after "3.7"? Otherwise, LGTM. Bring on the asynciness! ChrisA
Would this mean that with this PEP I can write a for loop like this? for (await?) item in async_generator(): ... code Or am I misunderstanding something? footnote: I am not really up to date with the async PEPs, so please correct me if this has already been discussed somewhere else.
Hi Matthias, On 2016-09-03 5:55 PM, Matthias welp wrote:
No, this is an illegal syntax and will stay that way. The PEP allows to do this: [await item for item in generator()] and this: [await item async for item in async_generator()] The idea is to remove any kind of restrictions that we currently have on async/await. A lot of people just assume that PEP 530 is already implemented in 3.5. Yury
On 4 September 2016 at 01:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6.
Interesting proposal. Would be nice to have this! I have one question:
Do I understand correctly that the limitation that they are allowed only in async def is because await binds to the enclosing coroutine? There is an old "bug" (some people call this a feature) http://bugs.python.org/issue10544 If one uses yield in a comprehension, then it leads to unexpected results:
This function yields only once and then returns another generator. This is because the yield in comprehension "lives" in an auxiliary function scope used to make the comprehension. So that such comprehension are even allowed outside function body:
Do I understand correctly that this is not the case with asynchronous comprehensions? If this is not the case, then I like this, but this will be inconsistent with normal comprehensions. -- Ivan
On 6 September 2016 at 11:22, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I was going to say that the problem with that last sentence is the "just" part, as for a long time, even though the more desirable behaviour was relatively clear ("make it work the way it did in Python 2"), we didn't have a readily available means of doing that. However, the last time I thought seriously about this problem was before we added "yield from", and that may actually be enough to change the situation. Specifically, the problem is that comprehensions and generator expressions in 3.x create a full closure, so code like: def example(): L = [(yield) for i in range(2)] print(L) is implicitly doing: def example(): def _bad_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = _bad_gen(range(2)) print(L) Prior to yield from, the code generator had no easy way to fix that, since it couldn't delegate yield operations to the underlying generator. However, given PEP 380, the code generator could potentially detect these situations, and implicitly use "yield from" to hoist the generator behaviour up to the level of the containing function, exactly as happened in Python 2: def example(): def _nested_gen(_arg): result = [] for i in _arg: result.append((yield)) return result L = yield from _nested_gen(range(2)) print(L)
I don't think there's anything we can do about generator expressions short of PEP 530 itself though - they already misbehave in Python 2, and misbehave in exactly the same way in Python 3, since there's no way for the interpreter to tell the difference at runtime between the implicit yields from the generator expression itself, and any explicit yields used in generator subexpressions. By contrast, PEP 530 can work, since it doesn't *want* to tell the difference between the implicit awaits and explicit awaits, and the await and yield channels are already distinguished at the language level. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Sven, On 2016-09-05 4:27 PM, Sven R. Kunze wrote:
Right.
Consider "funcs" to be an asynchronous generator/iterable that produces a sequence of awaitables. The above comprehensions will await on each awaitable in funcs, producing regular list, set, and dict. I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable. There is no concept of asynchronous datastructures in Python. Thanks, Yury
Hi Srinivas, On 06.09.2016 05:46, srinivas devaki wrote:
oh, wrong territory here! Python async community wants you to write everything twice: for the sync and async case. And don't dare to mentioned code sharing here. They will rip you apart. ;) Just kidding. Of course would it be great to write code only once but Yury want to preserve well-paid Python dev jobs in the industry because everything here needs to be maintained twice then. ;) No really, I have absolutely no idea why you need to put that "async" in all places where Python can detect automatically if it needs to perform an async iteration or not. Maybe, Yury can explain. Cheers, Sven
On 7 September 2016 at 04:24, Sven R. Kunze <srkunze@mail.de> wrote:
Sven, this is not productive, not funny, and not welcome. Vent your frustrations with the fundamental split between synchronous and explicitly asynchronous software design elsewhere.
As Anthony already noted, the "async" keyword switches to the asynchronous version of the iterator protocol - you use this when your *iterator* needs to interact with the event loop, just as you do when deciding whether or not to mark a for loop as asynchronous. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 06.09.2016 20:37, Nick Coghlan wrote:
Don't make a mistake here, Nick. I take this with some humor as it does not concern me in production. It's interesting to see though that people new to the discussion detect this obvious issue very fast.
Of course "async" switches to async mode. But that was not the question. I asked WHY that's necessary not what it does. I already noted that Python can detect when to make the switch without a marker. And you fail to explain where the issue with this point of view is. Sven PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc. Is there something wrong with my config or is this deliberate on your part?
On 9/6/2016 2:54 PM, Sven R. Kunze wrote:
PS: Nick, I noted that while replying, my mail client made me responding to you and the list as cc.
For Thunderbird, this is normal behavior. I suspect you would find the same if your tried 'replay all' to multiple messages.
Is there something wrong with my config
No
or is this deliberate on your part?
No -- Terry Jan Reedy
On 7 September 2016 at 04:54, Sven R. Kunze <srkunze@mail.de> wrote:
The Python *runtime* can tell whether the result of an expression is a normal iterator or an asynchronous iterator, but the Python *compiler* can't. So at compile time, it has to decide what code to emit for the four different scenarios Anthony listed: # Normal comprehension, 100% synchronous and blocking squares = [i**i for i in range(10)] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value results = [await result for result in submit_requests()] # Non-blocking/async iterator that suspends before producing "normal" values results = [result async for submit_requests_and_wait_for_results()] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value results = [await result async for submit_requests_asynchronously()] And hence needs the "async" keyword to detect when it has the latter two cases rather than the first two. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes, so to remove "async" from the syntax, this would be handled at runtime. But there's no way this PEP is going to do that, so I agree there's no point in discussing this here. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Tue, Sep 6, 2016 at 9:24 PM, Sven R. Kunze <srkunze@mail.de> wrote: [...]
I'm sure he would explain, but it seems I was first ;) [last-minute edit: no, Nick was first, but this is a slightly different angle]. First, the "async" gets inherited from PEP 492, so this has actually already been decided on. While not strictly necessary for a syntax for "async for", it makes it more explicit what happens under the hood -- that __a*__ methods are called and awaited, instead of simply calling __iter__/__next__ etc. as in regular loops/comprehensions. Not a lot to debate, I guess. No surprises here, just implementation work. -- Koos
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On 06.09.2016 03:16, Yury Selivanov wrote:
So, what's the "async" good for then?
I doubt that anybody ever would write something like that; this is just examples of what the PEP will enable.
Why do you implement it then? :D Put it differently, why are you sceptic about it?
There is no concept of asynchronous datastructures in Python.
I thought so, that's why I asked. ;) "async def" gives me something async, so I assumed it to be the case here as well. Cheers, Sven
On Tue, Sep 6, 2016 at 10:20 AM, Sven R. Kunze <srkunze@mail.de> wrote:
Maybe I'm off base here, but my understanding is the `async for` version would allow for suspension during the actual iteration, ie. using the __a*__ protocol methods, and not just by awaiting on the produced item itself. IOW, `[await ... async for ...]` will suspend at least twice, once during iteration using the __a*__ protocols and then again awaiting on the produced item, whereas `[await ... for ...]` will synchronously produce items and then suspend on them. So to use the async + await version, your (async) iterator must return awaitables to satisfy the `async for` part with then produce another awaitable we explicitly `await` on. Can someone confirm this understanding? And also that all 4 combinations are possible, each with different meaning: # Normal comprehension, 100% synchronous and blocking [... for ...] # Blocking/sync iterator producing awaitables which suspend before producing a "normal" value [await ... for ...] # Non-blocking/async iterator that suspends before producing "normal" values [... async for ...] # Non-blocking/async iterator that suspends before producing awaitables which suspend again before producing a "normal" value [await ... async for ...] Is this correct? -- C Anthony
On Tue, Sep 6, 2016 at 9:27 AM, Sven R. Kunze <srkunze@mail.de> wrote:
AIUI they won't return "async lists" etc; what they'll do is asynchronously return list/set/dict. Imagine an async database query that returns Customer objects. You could get their names thus: names = [cust.name async for cust in find_customers()] And you could enumerate their invoices (another database query) with a double-async comprehension: invoices = {cust.name: await cust.list_invoices() async for cust in find_customers()} As always, you can unroll a comprehension to the equivalent statement form. _tmp = {} async for cust in find_customers(): _tmp[cust.name] = await cust.list_invoices() invoices = _tmp or with the original example: _tmp = [] async for fun in funcs: _tmp.append(await fun()) result = _tmp Hope that helps. ChrisA
On 4 September 2016 at 09:31, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
After using it a few times in examples, while I'm prepared to accept the agrammatical nature of "async for" in the statement form (where the adjective-noun phrase can be read as a kind of compound noun introducing the whole statement), I think for the comprehension form, we should aim to put the words in the right grammatical order if we can: result = [i for i in async aiter() if i % 2] I think the readability gain from that approach becomes even clearer with nested loops: result = [x for aiterable in async outer() for x in async aiterable] vs the current: result = [x async for aiterable in outer() async for x in async aiterable] In the first form, "async" is clearly a pre-qualifier on "outer()" and "aiterable", indicating they need to be asynchronous iterators rather than synchronous ones. By contrast, in the current form, the first "async" reads like a post-qualifer on "x" (it isn't, it modifies how outer() is handled in the outer loop), while the second looks like a post-qualifier on "outer()" (it isn't, it modified how aiterable is handled in the inner loop) If that means having to postpone full async comprehensions until "async" becomes a proper keyword in 3.7 and only adding "await in comprehensions and generator expressions" support to 3.6, that seems reasonable to me Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7 September 2016 at 21:37, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
The new issue that's specific to comprehensions is that the statement form doesn't have the confounding factor of having an expression to the left of it. Thus, the prefix is unambiguous and can be read as modifying the entire statement rather than just the immediately following keyword: async for row in db.execute(...): process(row) It's also pragmatically necessary right now due to the sleight of hand that Yury used in the code generation pipeline to bring in "async def", "async with", and "async for" without a __future__ statement, but without making them full keywords either. However, when we convert it to the comprehension form, a parsing ambiguity (for humans) arises that creates an inherent readability problem: [process(row) async for row in db.execute(...)] When reading that, is "async" a postfix operator being used in a normal comprehension (wrong, but somewhat plausible)? Or is it part of a compound keyword with "for" that modifies the iteration behaviour of that part of the comprehension (the correct interpretation)? [(process(row) async) for row in db.execute(...)] [process(row) (async for) row in db.execute(...)] The postfix operator interpretation is just plain wrong, but even the correct interpretation as a compound keyword sits between two expressions *neither* of which is the one being modified (that would be "db.execute()") By contrast, if the addition of full async comprehensions is deferred to 3.7 (when async becomes a true keyword), then the infix spelling can be permitted in both the statement and comprehension forms: for row in async db.execute(...): process(row) [process(row) for row in async db.execute(...)] with the prefix spelling of the statement form retained solely for backwards compatibility purposes (just as we retain "from __future__ import feature" flags even after the feature has become the default behaviour). The beauty of the infix form is that it *doesn't matter* whether someone reads it as a compound keyword with "in" or as a prefix modifying the following expression: [process(row) for row (in async) db.execute(...)] [process(row) for row in (async db.execute(...))] In both cases, it clearly suggests something special about the way "db.execute()" is going to be handled, which is the correct interpretation.
Currently regular comprehensions are pretty similar to `for` loop. Why async comprehensions should look different from `async for` counterpart?
Regular "for" loops don't have the problem of their introductory keyword being written as two words, as that's the culprit that creates the ambiguity when you add an expression to the left of it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: [...]
I don't think this issue strictly has anything to do with where that "async" is put in the syntax, as long as it's used within the definition of an async function: async def function(): # in here, is effectively "async" a keyword. Of course, in French, this would be: def function async(): # dedans, "async" est effectivement un mot clé I'm sure someone will be able to correct my French, though. [and Nick writes:]
That's an interesting suggestion. What exactly is the relation between deferring this PEP and permitting the infix spelling? This would make it more obvious at a first glance, whether something is a with statement or for loop. The word "async" there is still not very easy to miss, especially with highlighted syntax. I didn't realize (or had forgotten) that PEP 492 is provisional.
And db.execute is an async iterarable after all, so "async" is a suitable adjective for db.execute(...). -- Koos [...]
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On 8 September 2016 at 00:57, Koos Zevenhoven <k7hoven@gmail.com> wrote:
Good point - I was thinking there was additional cleverness around "async for" and "async with" as well, but you're right that it's specifically "async def" that enables the pseudo-keyword behaviour.
Just a mistake on my part regarding how we were currently handling "async" within "async def" statements. With that mistake corrected, there may not be any need to defer the suggestion, since it already behaves as a keyword in the context where it matters. That said, we *are* doing some not-normal things in the code generation pipeline to enable the pseudo-keyword behaviour, so I also wouldn't be surprised if there was a practical limitation on allowing the "async" to appear after the "in" rather than before the "for" prior to 3.7. It's also worth reviewing the minimalist grammar changes in PEP 492 and the associated discussion about "async def" vs "def async": * https://www.python.org/dev/peps/pep-0492/#grammar-updates * https://www.python.org/dev/peps/pep-0492/#why-async-def-and-not-def-async Changing "for_stmt" to allow the "for TARGET in [ASYNC] expr" spelling isn't as tidy a modification as just allowing ASYNC in front of any of def_stmt, for_stmt and with_stmt.
Right, and one of the reasons for that was because we hadn't fully worked through the implications for comprehensions and generator expressions at the time. Now that I see the consequences of attempting to transfer the "async keyword is a statement qualifier " notion to the expression form, I think we may need to tweak things a bit :)
Exactly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 7, 2016 at 2:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree this would be better, but the difference compared to PEP-492 async for loops (and even async with statements) would be awful :S. -- Koos
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
participants (12)
-
Andrew Svetlov -
C Anthony Risinger -
Chris Angelico -
Greg Ewing -
Ivan Levkivskyi -
Koos Zevenhoven -
Matthias welp -
Nick Coghlan -
srinivas devaki -
Sven R. Kunze -
Terry Reedy -
Yury Selivanov