[Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

In a different thread I noted that I sometimes want to write code like this:

    while any(n % p == 0 for p in small_primes):
        # divide p out - but what's p?

But generator expressions hide the value of `p` that succeeded, so I
can't.  `any()` and `all()` can't address this themselves - they
merely work with an iterable of objects to evaluate for truthiness,
and know nothing about how they're computed.  If you want to identify
a witness (for `any()` succeeding) or a counterexample (for `all()`
failing), you need to write a workalike loop by hand.

So would this spelling work using binding expressions?

    while any(n % (thisp := p) == 0 for p in small_primes):
        n //= thisp

I'm not entirely clear from what the PEP says, but best guess is "no",
from this part of the discussion[1]:

"""
It would be convenient to use this feature to create rolling or
self-effecting data streams:

    progressive_sums = [total := total + value for value in data]

This will fail with UnboundLocalError due to total not being
initalized. Simply initializing it outside of the comprehension is
insufficient - unless the comprehension is in class scope: ...
"""

So, in my example above, I expect that `thisp` is viewed as being
local to the created-by-magic lexically nested function implementing
the generator expression.  `thisp` would be bound on each iteration,
but would vanish when `any()` finished and the anonymous function
vanished with it.  I'd get a NameError on "n //= thisp" (or pick up
whatever object it was bound to before the loop).

I have a long history of arguing that magically created lexically
nested anonymous functions try too hard to behave exactly like
explicitly typed lexically nested functions, but that's the trendy
thing to do so I always lose ;-)  The problem:  in a magically created
nested function, you have no possibility to say _anything_ about
scope; at least when you type it by hand, you can add `global` and/or
`nonlocal` declarations to more-or-less say what you want.

Since there's no way to explicitly identify the desired scope, I
suggest that ":=" inside magically created nested functions do the
more-useful-more-often thing:  treat the name being bound as if the
binding had been spelled in its enclosing context instead.  So, in the
above, if `thisp` was declared `global`, also `global` in the genexp;
if `nonlocal`, also `nonlocal`; else (almost always the case in real
life) local to the containing code (meaning it would be local to the
containing code, but nonlocal in the generated function).

Then my example would work fine, and the PEP's would too just by adding

    total = 0

before it.

Funny:  before `nonlocal` was added, one of the (many) alternative
suggestions was that binding a name in an enclosing scope use ":="
instead of "=".

No, I didn't have much use for `for` target names becoming magically
local to invisible nested functions either, but I appreciate that it's
less surprising overall.  Using ":=" is much more strongly screaming
"I'm going way out of my way to give a name to this thing, so please
don't fight me by assuming I need to be protected from the
consequences of what I explicitly asked for".


[1] https://www.python.org/dev/peps/pep-0572/