[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> On 22 November 2017 at 15:47, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> I generally don't understand "await" in any context, so I deferred
>> judgement on that :-) Based on your comment that they are equally
>> tricky, I'd suggest we prohibit them both ;-)
>>
>> Less facetiously, comprehensions are defined in the language reference
>> in terms of a source translation to nested loops. That description
>> isn't 100% precise, but nevertheless, if yield/async in a
>> comprehension doesn't behave like that, I'd consider it a bug. So
>> current behaviour (for both yield and await) is a bug, and your
>> proposed semantics for yield is correct.
>
>
> I think there may be a small misunderstanding here. The situation is
> different for comprehensions and generator expressions,
> let me summarize the current state:
>
> - yield in comprehensions works "wrong" (a shorthand for not according to
> the docs/naive expectations, i.e. not equivalent to for loop)
> - await in comprehensions works "right"
> - yield in generator expressions works "wrong"
> - await in generator expressions works "wrong"

"await" in generator expressions works *correct*. I explained it here:
https://bugs.python.org/issue32113

To clarify a bit more:

   r = (V for I in X)

is equivalent to:

   def _():
      for I in X:
        yield V
   r = _()

For synchronous generator expression:

   r = (f(i) for i in range(3))

is really:

   def _():
      for i in range(3):
        yield f(i)
   r = _()

For an asynchronous generator expression:

   r = (await f(i) for i in range(3))

is equivalent to:

   def _():
      for i in range(3):
        yield (await f(i))
   r = _()

I'll update PEP 530 to clarify these transformations better.

>
> After some thinking, both `yield` and `await` look quite mind bending in
> _generator expressions_, so maybe the right compromise strategy is:
>
> - fix yield in comprehensions
> - await in comprehensions already works
> - make both `yield` and `await` a SyntaxError in generator expressions.

I'm all for prohibiting using 'yield' expression in generator
expressions or comprehensions.  The semantics is way to hard to
understand and hence be of any value.

Making 'await' a SyntaxError is absolutely not an option.  Async
generator expressions are a shorthand syntax for defining asynchronous
generators (PEP 525), and it's already being used in the wild.

Yury