[Cython] Scoped expressions and generators problem

[Stefan Behnel]

Vitja Makarov, 19.05.2011 22:34:
> I don't know how to handle scopedexprs in generators, here are some examples:
>
> [(yield i) for i in l] or [i for i in yield]
>
> In Python3 language mode scoped expression is created for list
> comprehension and loop variable is moved there.

Not only that. In Python 3 (and also for set/dict comprehensions in Py2.7), 
the above are basically generator expressions and the behaviour of yield 
inside of a generator expression is weird at best.

   >>> a = [(yield i) for i in (1,2,3)]
   >>> a
   <generator object <listcomp> at 0x1be2a00>
   >>> next(a)
   1
   >>> a.send(5)
   2
   >>> next(a)
   3
   >>> next(a)
   [5, None, None]

The second case is simpler and more obvious as the yield expression only 
determines the iterable, which happens before entering the loop.

Note, however, that the iterable is currently being evaluated inside of the 
wrong scope.

http://trac.cython.org/cython_trac/ticket/600


> So now it isn't stored inside generator closure and is lost between yields.

Right, it's a separate scope. I guess this means that expression scopes 
must behave differently when one of the surrounding scopes is a generator 
scope. They either have to somehow declare their local names in that outer 
scope, or the closure class generator would have to descend into these 
inner scopes as well to inject additional names into the closure, or we 
could let the yield expression node explicitly copy over local names from 
surrounding non-generator scopes into temps.

The current workings of declaring a block local C variable would match best 
with the third way IMHO, even if that's not the cleanest solution. 
Otherwise, we'd also have to change the way scoped expressions work.


> Btw there is one more problem I hope that's easy to solve:
> yield expression inside scoped expression is counted twice.

That means that the YieldNodeCollector must stop at scoped expression 
boundaries. However, as ticket #600 shows, this may not be completely 
trivial to fix.

It might work to let the scoped expression nodes own the iterable node, and 
to use a CloneNode in the place where the iterable is used inside of the 
loop. That way, the scope node can directly decide which scope to evaluate 
the iterable node in.

Stefan