[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 25 November 2017 at 04:30, Guido van Rossum <guido at python.org> wrote:
>
>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> The more I hear about this topic, the more I think that `await`, `yield`
>>> and `yield from` should all be banned from occurring in all comprehensions
>>> and generator expressions. That's not much different from disallowing
>>> `return` or `break`.
>>>
>>
>> From the responses it seems that I tried to simplify things too far.
>> Let's say that `await` in comprehensions is fine, as long as that
>> comprehension is contained in an `async def`. While we *could* save `yield
>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>> and the fact that the presence of `yield [from]` *implicitly* makes the
>> surrounding `def` a generator makes things worse. It just requires too many
>> mental contortions to figure out what it does.
>>
>
> There were some arguments that `await` is like a function call, while
> `yield` is like `return`.
> TBH, I don't really like these arguments since to me they are to vague.
> Continuing this logic one can say that
> `return` is just a fancy function call (calling continuation with the
> result). To me there is one clear distinction:
> `return` and `break` are statements, while `yield`, `yield from`, and
> `await` are expressions.
>

Indeed. However, `yield from` as an expression is mostly a deprecated way
to write `await`, and `yield` as an expression is mostly an alternative way
of writing coroutines (it's the only way that exists in Python 2). Another
big difference is that the use of `yield [from]` affects the surrounding
function, making it a generator.

Continuing the topic of the ban, what exactly should be banned? For example
> will this still be valid?
>
>     def pack_two():
>         return [(yield), (yield)]  # Just a list display
>

It's not a comprehension so it's still valid.


> I don't see how this is controversial. It is clear that `pack_two` is a
> generator.
> If this is going to be prohibited, then one may be surprised by lack of
> referential transparency, since this will be valid:
>
>     def pack_two():
>         first = (yield)
>         second = (yield)
>         return [first, second]
>
> If the first example will be allowed, then one will be surprised why it
> can't be rewritten as
>
>     def pack_two():
>         return [(yield) for _ in range(2)]
>

And yet Nick's example shows that that is not equivalent!

def example():
    comp1 = yield from [(yield x) for x in ('1st', '2nd')]
    comp2 = yield from [(yield x) for x in ('3rd', '4th')]
    return comp1, comp2

In this example each thing that looks syntactically like a list
comprehension becomes actually a generator expression at at runtime! And so
does your example, so instead of a list of two items, it returns a
generator that will produce two values when iterated over.

That's not referential transparency to me, it feels more like a bug in the
code generator.

I want to ban this because apparently nobody besides Nick knows about this
behavior (I certainly didn't, and from the above it seems you don't either).


> I have found several other examples where it is not clear whether they
> should be prohibited with `yield` or not.
>

Such as?


> I still propose to rule out all of the above from generator expressions,
>> because those can escape from the surrounding scope.
>>
>
> Here I agree. Also note that the above problem does not apply to generator
> expressions since (x, x) and (x for _ in range(2)) are
> two very different expressions.
>

PS. A more radical proposal (not for 3.7) would be to deprecate yield as an
expression. It once was only a statement, but PEP 342 introduced yield as
an expression in order to do coroutines. We now have `async def` and
`await` as a superior coroutine mechanism. But we must continue to support
yield expressions because there is a lot of Python 2/3 compatible code that
depends on it. (E.g. Tornado.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171125/26118b52/attachment-0001.html>