[Python-ideas] A comprehension scope issue in PEP 572

[Guido van Rossum]

(Note: this is an off-topic side thread, unrelated to assignment
expressions. Inline comment below.)

On Fri, May 11, 2018 at 9:08 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Fri, May 11, 2018 at 9:15 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > * *maybe* discover that even the above expansion isn't quite accurate,
> and
> > that the underlying semantic equivalent is actually this (one way to
> > discover this by accident is to have a name error in the outermost
> iterable
> > expression):
> >
> >     def _genexp(_outermost_iter):
> >         for x in _outermost_iter:
> >             yield x
> >
> >     _result = _genexp(_outermost_iter)
> >
> > * and then realise that the optimised list comprehension form is
> essentially
> > this:
> >
> >     def _listcomp(_outermost_iter):
> >         result = []
> >         for x in _outermost_iter:
> >             result.append(x)
> >         return result
> >
> >     _result = _listcomp(data)
> >
> > Now that "yield" in comprehensions has been prohibited, you've learned
> all
> > the edge cases at that point
>
> Not quite! You missed one, just because comprehensions aren't weird
> enough yet. AFAIK you can't tell with the list comp, but with the
> genexp you can (by not iterating over it).
>
> >     def _genexp(_outermost_iter):
> >         for x in _outermost_iter:
> >             yield x
> >
> >     _result = _genexp(data)
>
> It's actually this:
>
>      def _genexp(_outermost_iter):
>          for x in _outermost_iter:
>              yield x
>
>      _result = _genexp(iter(_outermost_iter))
>
> I don't think there's anything in the main documentation that actually
> says this, although PEP 289 mentions it in the detaily bits. [1]
>
> ChrisA
>
> [1] https://www.python.org/dev/peps/pep-0289/#the-details
>

I'm not sure this is the whole story. I tried to figure out how often
__iter__ is called in a genexpr. I found that indeed I see iter() is called
as soon as the generator is brought to life, but it is *not* called a
second time the first time you call next(). However the translation you
show has a 'for' loop which is supposed to call iter() again. So how is
this done? It seems the generated bytecode isn't equivalent to a for-loop,
it's equivalent to s while loop that just calls next().

Disassembly of a regular generator:

def foo(a):
  for x in a: yield x

*  2           0 SETUP_LOOP              18 (to 20)*
              2 LOAD_FAST                0 (a)
*              4 GET_ITER*
        >>    6 FOR_ITER                10 (to 18)
              8 STORE_FAST               1 (x)
             10 LOAD_FAST                1 (x)
             12 YIELD_VALUE
             14 POP_TOP
             16 JUMP_ABSOLUTE            6
        >>   18 POP_BLOCK
        >>   20 LOAD_CONST               0 (None)
             22 RETURN_VALUE


But for a generator:

g = (x for x in C())

  1           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                10 (to 14)
              4 STORE_FAST               1 (x)
              6 LOAD_FAST                1 (x)
              8 YIELD_VALUE
             10 POP_TOP
             12 JUMP_ABSOLUTE            2
        >>   14 LOAD_CONST               0 (None)
             16 RETURN_VALUE

Note the lack of SETUP_LOOP and GET_ITER (but otherwise they are identical).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180511/bd05666e/attachment-0001.html>