[Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory

[Nick Coghlan]

Stefan Behnel wrote:
> This is also an important issue for other Python implementations. Cython
> simply transforms comprehensions into the equivalent for-loop, so when we
> implement PEP 342 in Cython, we will have to find a way to emulate
> CPython's behaviour here (unless we decide to stick with Py2.x sematics,
> which would not be my preferred solution).

How do you do that without leaking the iteration variable into the
current namespace?

Avoiding that leakage is where the semantic change between 2.x and 3.x
came from here: 2.x just creates the for loop inline (thus leaking the
iteration variable into the current scope), while 3.x creates an inner
function that does the iteration so that the iteration variables exist
in their own scope without polluting the namespace of the containing
function.

The translation of your example isn't quite as Alexandre describes it -
we do at least avoid the overhead of creating a generator function in
the list comprehension case. It's more like:

    while True:
        def f():
            result = []
            for i in range(chunk_size):
                result.append((yield))
            return result
        target.send(f())

So what you end up with is a generator that has managed to bypass the
syntactic restriction that disallows returning non-None values from
generators. In CPython it appears that happens to end up being executed
as if the return was just another yield expression (most likely due to a
quirk in the implementation of RETURN_VALUE inside generators):

    while True:
        def f():
            result = []
            for i in range(chunk_size):
                result.append((yield))
            yield result
        target.send(f())

It seems to me that CPython should be raising a SyntaxError for yield
expressions inside comprehensions (in line with the "no returning values
other than None from generator functions" rule), and probably for
generator expressions as well.

Cheers,
Nick.

P.S. Experimentation at a 3.x interpreter prompt:

>>> def f():
...   return [(yield) for i in range(10)]
...
>>> x = f()
>>> next(x)
>>> for i in range(8):
...   x.send(i)
...
>>> x.send(8)
>>> next(x)
[0, 1, 2, 3, 4, 5, 6, 7, 8, None]
>>> x = f()
>>> next(x)
>>> for i in range(10): # A statement with a return value!
...   x.send(i)
...
[0, 1, 2, 3, 4, 5, 6, 7, 8, None]
>>> dis(f)
  2           0 LOAD_CONST               1 (<code object <listcomp> at
0xb7c53bf0, file "<stdin>", line 2>)
              3 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (range)
              9 LOAD_CONST               2 (10)
             12 CALL_FUNCTION            1
             15 GET_ITER
             16 CALL_FUNCTION            1
             19 RETURN_VALUE
>>> dis(f.__code__.co_consts[1])
  2           0 BUILD_LIST               0
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                13 (to 22)
              9 STORE_FAST               1 (i)
             12 LOAD_CONST               0 (None)
             15 YIELD_VALUE
             16 LIST_APPEND              2
             19 JUMP_ABSOLUTE            6
        >>   22 RETURN_VALUE


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------