[Python-ideas] PEP 572 version 2: Statement-Local Name Bindings

[Christoph Groth]

Nick Coghlan wrote:
> On 25 March 2018 at 22:44, Christoph Groth <christoph at grothesque.org> wrote:
> > I think that it's a helpful guideline to imagine what the ideal
> > behavior should be if we were not constrained by backwards
> > compatibility, and then try to follow it.  In the case at hand, we
> > all seem to agree that the fact that the outermost iterator of a
> > comprehension is evaluated in the surrounding scope is somewhat of a
> > wart, although one that is rarely visible.
>
> There's no such agreement, since generator expressions have worked
> that way since they were first introduced almost 15 years ago:
> https://www.python.org/dev/peps/pep-0289/#the-details

I was referring to [1], but I see now that things are more subtle.
Guido didn't express a dislike for the special treatment of the
outermost iterator, just (if I understand correctly) for the
consequences of this in class scope.

[1] https://mail.python.org/pipermail/python-ideas/2018-March/049461.html

> It's much easier to see the difference with generator expressions,
> since the evaluation of the loop body is delayed until the generator
> is iterated over, while the evaluation of the outermost iterator is
> immediate (so that any exceptions are more easily traced to the code
> responsible for them, and so that they don't need to create a closure
> in the typical case).

Thanks a lot for the explaination.  I know many of the more obscure
aspects of Python, but I wasn't aware of this.

I'm still not sure whether I agree that it is a good idea to treat the
evaluation of the outermost iterator differently from the rest.  It can
be especially confusing that the outermost iterator appears in the
middle of the generator expression, surrounded by code that is evaluated
in the inner scope.  But I don't want to start a futile discussion about
backwards-incompatible changes here.

> With comprehensions, the entire nested scope gets evaluated eagerly,
> so it's harder to detect that there's a difference in the evaluation
> scope of the outermost iterator in normal use. That difference *does*
> exist though, and we're not going to tie ourselves into knots to try
> to hide it (since the exact same discrepancy will necessarily exist
> for generator expressions, and semantic consistency between genexps
> and the corresponding comprehensions is highly desirable).

With tying into knots you mean my suggestion to bind both 'a' and 'b' at
the internal scope in

gen = (i+j for i in x[a := aa] for j in (b := bb))

even if x[a := aa] is evaluated in the outer scope?  I tend to agree
that this is indeed too arcane.  But then how about making ":=" work
always at the outer scope (and thus reserve the inner scope of
generators only for the index variables), i.e. make the above equivalent
to:

def g(it):
    nonlocal b    
    for i in it:
        for j in (b := bb):
            yield i+j
gen = g(iter(x[a := aa]))

That would be less confusing and indeed it could even turn out useful to
be able to access names that were bound inside the generator outside of
it.  Note that the use of ":=" inside generator expressions would be
optional, so it can be considered fair to treat such assignments
differently from the implicit assignments to the loop variables.