[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Stephen J. Turnbull]

Paul Moore writes:

 > 1. List comprehensions expand into nested for/if statements in the
 > "obvious" way - with an empty list created to start and append used to
 > add items to it.
 >    1a. Variables introduced in the comprehension don't "leak" (see below).
 > 2. Generator expressions expand into generators with the same "nested
 > loop" behaviour, and a yield of the generated value.
 > 3. List comprehensions are the same as list(the equivalent generator
 > expression).

I'm a little late to this discussion, but I don't see how 3 can be
true if 1 and 2 are.

Because I believe my model of generator expressions is pretty simple,
I'll present it here:

- the generator expression in 2 and 3 implicitly creates a generator
  function containing the usual loop
  - the point being that it will "capture" all of the yields (implicit
    *and* explicit) in the generator expression
- then implicitly invokes it, 
- passing the (iterable) generator object returned to the containing
  expression.

"Look Ma, no yields (or generator functions) left!"

However, my model of comprehensions is exactly a for loop that appends
to an empty list repeatedly, but doesn't leak iteration variables.  So
a yield in a comprehension turns the function *containing* the
comprehension (which may or may not exist) into a generator function.
In other words, I don't agree with Ethan that a yield inside a list
comprehension should not affect the generator-ness of the containing
function.  What would this mean under that condition:

    [f(x) for x in (yield iterable)]

then?  IOW,

    x = ((yield i) for i in iterable)

IS valid syntax at top level, while

    x = [(yield i) for i in iterable]

IS NOT valid syntax at top level given those semantics.  The latter
"works" in Python 3.6;

    >>> for i in [(yield i) for i in (1, 2, 3)]:
    ...  i
    ... 
    1
    2
    3

though I think it should be a syntax error, and "bare" yield does not
"work":

    >>> i = 1
    >>> yield i
      File "<stdin>", line 1
    SyntaxError: 'yield' outside function
    >>> (yield i)
      File "<stdin>", line 1
    SyntaxError: 'yield' outside function

FWIW, that's the way I'd want it, and the way I've always understood
comprehensions and generator expressions.  I think this is consistent
with Yuri's, Serhiy's, and Ivan's claims, but I admit I'm not sure.
Of course the compiler need not create a generator function and invoke
it, but the resulting bytecode should be the same as if it did.

This means the semantics of [FOR-EXPRESSION] and [(FOR-EXPRESSION)]
should differ in the same way.

Steve