[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 25 November 2017 at 11:04, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

> On 25 November 2017 at 01:22, Guido van Rossum <guido at python.org> wrote:
>
>> The more I hear about this topic, the more I think that `await`, `yield`
>> and `yield from` should all be banned from occurring in all comprehensions
>> and generator expressions. That's not much different from disallowing
>> `return` or `break`.
>>
>>
> IIUC this would essentially mean rejecting PEP 530.
> What do you think about banning `await`, `yield` and `yield from` only
> from generator expressions?
> Comprehensions can be fixed (just make them equivalent to for-loops
> without leaks as discussed).
>

I'm leaning towards this as well - the immediate evaluation on
comprehensions means that injecting an automatic "yield from" when will do
the right thing due to this component of PEP 380:
https://www.python.org/dev/peps/pep-0380/#the-refactoring-principle

Restating how folks intuitively think Py3 comprehensions containing "yield"
and "await" should work (''__var" indicates hidden variables only
accessible to the interpreter):

    result = [(yield x) for x in iterable]

   # Readers expect this to work much like
    __listcomp1 = []
    for __x1 in iterable:
        __listcomp1.append(yield __x1)
    result = __listcomp1

PEP 380's refactoring principle then means the above is required to be
equivalent to:

    def __listcomp1(__outermost_iterable):
        __result = []
        for  x in __outermost_iterable:
            __result.append(yield x)
        return __result

    result = yield from __listcomp1(iterable)

Which means that only missing piece in the current implementation is
inserting the implicit "yield from" into the function containing the
comprehension. No need for new lexical scoping conventions, so need for
syntactical special cases, just teaching this part of the compiler to
delegate to a subgenerator correctly. The compiler should just need a small
local adjustment to check whether the implicit nested scope is a generator
scope or not (and any Python source compiler, including CPython's, already
needs to track that information in order to ensure it emits a generator
function instead of a regular synchronous function).

>From a language evolution perspective, such a change is relatively
straightforward to explain on the basis of "Comprehensions became
implicitly nested scopes in 3.0, but we didn't implement PEP 380's 'yield
from' construct until Python 3.3, and it then took us until 3.7 to realise
we could combine them to make yield expressions inside comprehensions
behave more intuitively".

For "await", I'm pretty sure the current semantics are actually OK, but I
confess I haven't been in a position to have to explain them to anyone
either.

I honestly don't see a good way to salvage yield or yield from inside
generator expressions though. If we dropped the implicit "yield" when an
explicit one is present, that would break the symmetry between "[(yield x)
for x in iterable]" and "list((yield x) for x in iterable)", while
inserting an implicit "yield from" (as I'm now suggesting we do for
comprehensions) wouldn't work at all (since a genexp returns a generator
*fuction*, not a generator-iterator object). I'm also pretty sure it will
be difficult for the compiler to tell the difference between explicit and
implicit yield expressions in the subfunction (as there's nothing else
anywhere in the language that needs to make that distinction).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171125/e99da295/attachment.html>