[Python-ideas] PEP 530: Asynchronous Comprehensions

[Nick Coghlan]

On 8 September 2016 at 00:57, Koos Zevenhoven <k7hoven at gmail.com> wrote:
> On Wed, Sep 7, 2016 at 3:27 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> [...]
>>
>> The new issue that's specific to comprehensions is that the statement
>> form doesn't have the confounding factor of having an expression to
>> the left of it. Thus, the prefix is unambiguous and can be read as
>> modifying the entire statement rather than just the immediately
>> following keyword:
>>
>>     async for row in db.execute(...):
>>         process(row)
>>
>> It's also pragmatically necessary right now due to the sleight of hand
>> that Yury used in the code generation pipeline to bring in "async
>> def", "async with", and "async for" without a __future__ statement,
>> but without making them full keywords either.
>
> I don't think this issue strictly has anything to do with where that
> "async" is put in the syntax, as long as it's used within the
> definition of an async function:
>
> async def function():
>     # in here, is effectively "async" a keyword.

Good point - I was thinking there was additional cleverness around
"async for" and "async with" as well, but you're right that it's
specifically "async def" that enables the pseudo-keyword behaviour.

> [and Nick writes:]
>>     [process(row) async for row in db.execute(...)]
>>
>> When reading that, is "async" a postfix operator being used in a
>> normal comprehension (wrong, but somewhat plausible)? Or is it part of
>> a compound keyword with "for" that modifies the iteration behaviour of
>> that part of the comprehension (the correct interpretation)?
>>
>>     [(process(row) async) for row in db.execute(...)]
>>     [process(row) (async for) row in db.execute(...)]
>>
>> The postfix operator interpretation is just plain wrong, but even the
>> correct interpretation as a compound keyword sits between two
>> expressions *neither* of which is the one being modified (that would
>> be "db.execute()")
>>
>> By contrast, if the addition of full async comprehensions is deferred
>> to 3.7 (when async becomes a true keyword), then the infix spelling
>> can be permitted in both the statement and comprehension forms:
>
> That's an interesting suggestion. What exactly is the relation between
> deferring this PEP and permitting the infix spelling?

Just a mistake on my part regarding how we were currently handling
"async" within "async def" statements. With that mistake corrected,
there may not be any need to defer the suggestion, since it already
behaves as a keyword in the context where it matters.

That said, we *are* doing some not-normal things in the code
generation pipeline to enable the pseudo-keyword behaviour, so I also
wouldn't be surprised if there was a practical limitation on allowing
the "async" to appear after the "in" rather than before the "for"
prior to 3.7.

It's also worth reviewing the minimalist grammar changes in PEP 492
and the associated discussion about "async def" vs "def async":

* https://www.python.org/dev/peps/pep-0492/#grammar-updates
* https://www.python.org/dev/peps/pep-0492/#why-async-def-and-not-def-async

Changing "for_stmt" to allow the "for TARGET in [ASYNC] expr" spelling
isn't as tidy a modification as just allowing ASYNC in front of any of
def_stmt, for_stmt and with_stmt.

> This would make it more obvious at a first glance, whether something
> is a with statement or for loop. The word "async" there is still not
> very easy to miss, especially with highlighted syntax.
>
> I didn't realize (or had forgotten) that PEP 492 is provisional.

Right, and one of the reasons for that was because we hadn't fully
worked through the implications for comprehensions and generator
expressions at the time.

Now that I see the consequences of attempting to transfer the "async
keyword is a statement qualifier " notion to the expression form, I
think we may need to tweak things a bit :)

>> The beauty of the infix form is that it *doesn't matter* whether
>> someone reads it as a compound keyword with "in" or as a prefix
>> modifying the following expression:
>>
>>     [process(row) for row (in async) db.execute(...)]
>>     [process(row) for row in (async db.execute(...))]
>>
>> In both cases, it clearly suggests something special about the way
>> "db.execute()" is going to be handled, which is the correct
>> interpretation.
>
> And db.execute is an async iterarable after all, so "async" is a
> suitable adjective for db.execute(...).

Exactly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia