[Python-ideas] A comprehension scope issue in PEP 572

[Tim Peters]

[Tim]
>> I have a long history of arguing that magically created lexically
>> nested anonymous functions try too hard to behave exactly like
>> explicitly typed lexically nested functions, but that's the trendy
>> thing to do so I always lose ;-)

[Nick Coghlan <ncoghlan at gmail.com>]
> You have the reasoning there backwards:

That's easy to believe - I also had a long history of resisting nested
scopes at all ;-)


> implicitly nested scopes behave like explicitly nested scopes because
> that was the *easy* way for me to implement them in Python 3.0 (since
> I got to re-use all the pre-existing compile time and runtime machinery
> that was built to handle explicit lexical scopes).
> Everything else I tried (including any suggestions made by others on the
> py3k mailing list when I discussed the problems I was encountering) ran into
> weird corner cases at either compile time or run time, so I eventually gave
> up and proposed that the implicit scope using to hide the iteration variable
> name binding be a full nested closure, and we'd just live with the
> consequences of that.

It's unfortunate that there are "consequences" at all.  That kind of
thing is done all the time in Lisp-ish languages, but they require
explicitly declaring names' scopes.  Python's "infer scope instead by
looking for bindings" worked great when it had 3 scopes total, but
keeps forcing "consequences" that may or may not be desired in a
generally-nested-scopes world.


> The sublocal scoping proposal in the earlier drafts of PEP 572 was our first
> serious attempt at defining a different way of doing things that would allow
> names to be hidden from surrounding code while still being visible in nested
> suites, and it broke people's brains to the point where Guido explicitly
> asked Chris to take it out of the PEP :)

To which removal I was sympathetic, BTW.


> However, something I *have* been wondering is whether or not it might make
> sense to allow inline scoping declarations in comprehension name bindings.
> Then your example could be written:
>
>     def ...:
>         p = None
>         while any(n % p for nonlocal p in small_primes):
>             # p was declared as nonlocal in the nested scope, so our p
> points to the last bound value

Which more directly addresses the underlying problem:  not really
"binding expressions" per se, but the lack of control over scope
decisions in comprehensions period.  It's not at all that nested
scopes are a poor model, it's that we have no control over what's
local _to_ nested scopes the language creates.  I'd say that's worth
addressing in its own right, regardless of PEP 572's fate.

BTW, the "p = None" there is annoying too ;-)


> Needing to switch from "nonlocal p" to "global p" at module level would
> likely be slightly annoying, but also a reminder that the bound name is now
> visible as a module attribute.

Or `nonlocal` could be taught that its use one level below `global`
has an obvious meaning:  global.


> If any other form of comprehension level name binding does eventually get
> accepted, then inline scope declarations could similarly be used to hoist
> values out into the surrounding scope:
>
>         rem = None
>         while any((nonlocal rem := n % p) for nonlocal p in small_primes):
>             # p and rem were declared as nonlocal in the nested scope, so
> our rem and p point to the last bound value

Right - as above, inline scope declarations would be applicable to all
forms of comprehension-generated code.  And to any other future
construct that creates lexically nested functions.