List, set, and dict comprehensions compile like:

# input

result = [expression for lhs in iterator_expression]

# output

def comprehension(iterator):

    out = []

    for lhs in iterator:

        out.append(expression)

    return out

result = comprehension(iter(iterator_expression))

When you make `expression` a `yield` the compiler thinks that `comprehension` is a generator function instead of a normal function.

We can manually translate the following comprehension:

result = [(yield n) for n in (0, 1)]

def comprehension(iterator):

    out = []

    for n in iterator:

        # (yield n) as an expression is the value sent into this generator

        out.append((yield n))

    return out

result = comprehension(iter((0, 1)))

We can see this in the behavior of `send` on the resulting generator:

In [1]: g = [(yield n) for n in (0, 1)]

In [2]: next(g)
Out[2]: 0

In [3]: g.send('hello')
Out[3]: 1

In [4]: g.send('world')
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-4-82c589e0c27d> in <module>()
----> 1 g.send('world')

StopIteration: ['hello', 'world']

The `return out` gets translated into `raise StopIteration(out)` because the code is a generator.

The bytecode for this looks like:

In [5]: %%dis
   ...: [(yield n) for n in (0, 1)]
   ...:
<module>
--------
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x7f4bae68eed0, file "<show>", line 1>)
              3 LOAD_CONST               1 ('<listcomp>')
              6 MAKE_FUNCTION            0
              9 LOAD_CONST               5 ((0, 1))
             12 GET_ITER
             13 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             16 POP_TOP
             17 LOAD_CONST               4 (None)
             20 RETURN_VALUE

<module>.<listcomp>
-------------------
  1           0 BUILD_LIST               0
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                13 (to 22)
              9 STORE_FAST               1 (n)
             12 LOAD_FAST                1 (n)
             15 YIELD_VALUE
             16 LIST_APPEND              2
             19 JUMP_ABSOLUTE            6
        >>   22 RETURN_VALUE

In `<module>` you can see us create the <listcomp> function, call `iter` on `(0, 1)`, and then call `<listcomp>(iter(0, 1))`. In `<listcomp>` you can see the loop like we had above, but unlike a normal comprehension we have a `YIELD_VALUE` (from the `(yield n)`) in the comprehension.

The reason that this is different in Python 2 is that list comprehension, and only list comprehensions, compile inline. Instead of creating a new function for the loop, it is done in the scope of the comprehension. For example:

# input

result = [expression for lhs in iterator_expression]

# output

result = []

for lhs in iterator_expression:

    result.append(lhs)

This is why `lhs` bleeds out of the comprehension. In Python 2, adding making `expression` a `yield` causes the _calling_ function to become a generator:

def f():

    [(yield n) for n in range(3)]

    # no return

# py2

>>> type(f())

generator

# py3

>> type(f())

NoneType

In Python 2 the following will even raise a syntax error:

In [5]: def f():
   ...:     return [(yield n) for n in range(3)]
   ...:    
   ...:    
  File "<ipython-input-5-3602a9999f46>", line 2
    return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator

Generator expressions are a little different because the compilation already includes a `yield`.

# input

(expression for lhs in iterator_expression)

# output

def genexpr(iterator):

    for lhs in iterator:

        yield expression

You can actually abuse this to write a cute `flatten` function:

`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`

because it translates to:

def genexpr(seq):

    for sub in seq:

        # (yield from sub) as an expression returns the last sent in value

        if (yield from sub) and False:

            # we never enter this branch

            yield None

That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error.