[Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Chris Angelico <rosuav at gmail.com>]
> ...
> You're correct. The genexp is approximately equivalent to:
>
> def genexp():
>     for p in small_primes:
>         thisp = p
>         yield n % thisp == 0
> while any(genexp()):
>     n //= thisp
>
> With generator expressions, since they won't necessarily be iterated
> over immediately, I think it's correct to create an actual nested
> function; you need the effects of closures. With comprehensions, it's
> less obvious, and what you're asking for might be more plausible. The
> question is, how important is the parallel between list(x for x in
> iter) and [x for x in iter] ? Guido likes it, and to create that
> parallel, list comps MUST be in their own functions too.

I don't care how they're implemented here; I only care here about the
visible semantics.


>> I have a long history of arguing that magically created lexically
>> nested anonymous functions try too hard to behave exactly like
>> explicitly typed lexically nested functions, but that's the trendy
>> thing to do so I always lose ;-)  The problem:  in a magically created
>> nested function, you have no possibility to say _anything_ about
>> scope; at least when you type it by hand, you can add `global` and/or
>> `nonlocal` declarations to more-or-less say what you want.

> That's a fair point. But there is another equally valid use-case for
> assignment expressions inside list comps:
>
> values = [y + 2 for x in iter if (y := f(x)) > 0]
>
> In this case, it's just as obvious that the name 'y' should be local
> to the comprehension, as 'x' is.

There's a difference, though:  if `y` "leaks", BFD.  Who cares? ;-)
If `y` remains inaccessible, there's no way around that.


> Since there's no way to declare "nonlocal y" inside the
> comprehension, you're left with a small handful of options:

i leapt straight to #3:


> 1) All names inside list comprehensions are common with their
> surrounding scope. The comprehension isn't inside a function, the
> iteration variable leaks, you can share names easily. Or if it *is*
> inside a function, all its names are implicitly "nonlocal" (in which
> case there's not much point having the function).

DOA.  Breaks old code.

> 2) All names are local to their own scope. No names leak, and that
> includes names made with ":=".

Saying "local to their own scope" _assumes_ what you're trying to
argue _for_ - it's circular.  In fact it's impossible to know what the
user intends the scope to be.


> 3) Some sort of rule like "iteration variables don't leak, but those
> used with := are implicitly nonlocal".

Explicitly, because "LHS inherits scope from its context" (whether
global, nonlocal, or local)  is part of what ":=" is defined to _mean_
then.


> Would create odd edge cases eg [x for x in iter if x := x] and that would
> probably result in x leaking.

Don't care.


> 4) A special adornment on local names if you don't want them to leak
>
> 5) A special adornment on local names if you DO want them to leak

Probably also DOA.


> 6) A combination of #3 and #4: "x := expr" will be nonlocal, ".x :=
> expr" will be local, "for x in iter" will be local. Backward
> compatible but a pain to explain.

Definitely DOA.

...

>> Since there's no way to explicitly identify the desired scope, I
>> suggest that ":=" inside magically created nested functions do the
>> more-useful-more-often thing:  treat the name being bound as if the
>> binding had been spelled in its enclosing context instead.  So, in the
>> above, if `thisp` was declared `global`, also `global` in the genexp;
>> if `nonlocal`, also `nonlocal`; else (almost always the case in real
>> life) local to the containing code (meaning it would be local to the
>> containing code, but nonlocal in the generated function).

> Is it really more useful more often?

I found no comprehensions of any kind in my code where binding
expressions would actually be of use unless the name "leaked".  Other
code bases may, of course, yield different guesses.  I'm not a "cram a
lot of stuff on each line" kind of coder.

But the point above remains:  if they don't leak, contexts that want
them to leak have no recourse.  If they do leak, then the other uses
would still work fine, but they'd possibly be annoyed by a leak they
didn't want.


>> No, I didn't have much use for `for` target names becoming magically
>> local to invisible nested functions either, but I appreciate that it's
>> less surprising overall.  Using ":=" is much more strongly screaming
>> "I'm going way out of my way to give a name to this thing, so please
>> don't fight me by assuming I need to be protected from the
>> consequences of what I explicitly asked for".

> Personally, I'd still like to go back to := creating a statement-local
> name, one that won't leak out of ANY statement. But the tide was
> against that one, so I gave up on it.

Part of that is because - as the existence of this thread attests to -
we can't even control all the scopes gimmicks Python already has.  So
people are understandably terrified of adding even more ;-)