On 25 November 2017 at 11:04, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

On 25 November 2017 at 01:22, Guido van Rossum <guido@python.org> wrote:
The more I hear about this topic, the more I think that `await`, `yield` and `yield from` should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing `return` or `break`.

IIUC this would essentially mean rejecting PEP 530.
What do you think about banning `await`, `yield` and `yield from` only from generator expressions?
Comprehensions can be fixed (just make them equivalent to for-loops without leaks as discussed).

I'm leaning towards this as well - the immediate evaluation on comprehensions means that injecting an automatic "yield from" when will do the right thing due to this component of PEP 380: https://www.python.org/dev/peps/pep-0380/#the-refactoring-principle

Restating how folks intuitively think Py3 comprehensions containing "yield" and "await" should work (''__var" indicates hidden variables only accessible to the interpreter):

result = [(yield x) for x in iterable]

# Readers expect this to work much like

__listcomp1 = []

for __x1 in iterable:

__listcomp1.append(yield __x1)

result = __listcomp1

PEP 380's refactoring principle then means the above is required to be equivalent to:

def __listcomp1(__outermost_iterable):

__result = []

for x in __outermost_iterable:

__result.append(yield x)

return __result

result = yield from __listcomp1(iterable)

Which means that only missing piece in the current implementation is inserting the implicit "yield from" into the function containing the comprehension. No need for new lexical scoping conventions, so need for syntactical special cases, just teaching this part of the compiler to delegate to a subgenerator correctly. The compiler should just need a small local adjustment to check whether the implicit nested scope is a generator scope or not (and any Python source compiler, including CPython's, already needs to track that information in order to ensure it emits a generator function instead of a regular synchronous function).

From a language evolution perspective, such a change is relatively straightforward to explain on the basis of "Comprehensions became implicitly nested scopes in 3.0, but we didn't implement PEP 380's 'yield from' construct until Python 3.3, and it then took us until 3.7 to realise we could combine them to make yield expressions inside comprehensions behave more intuitively".

For "await", I'm pretty sure the current semantics are actually OK, but I confess I haven't been in a position to have to explain them to anyone either.

I honestly don't see a good way to salvage yield or yield from inside generator expressions though. If we dropped the implicit "yield" when an explicit one is present, that would break the symmetry between "[(yield x) for x in iterable]" and "list((yield x) for x in iterable)", while inserting an implicit "yield from" (as I'm now suggesting we do for comprehensions) wouldn't work at all (since a genexp returns a generator *fuction*, not a generator-iterator object). I'm also pretty sure it will be difficult for the compiler to tell the difference between explicit and implicit yield expressions in the subfunction (as there's nothing else anywhere in the language that needs to make that distinction).

Cheers,

Nick.

Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia