On 23 June 2018 at 13:48, Steven D'Aprano email@example.com wrote:
On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote: [...]
plenty of functional-language-inspired documentation to instead encourage folks to view comprehensions as tightly encapsulated declarative container construction syntax.
I can't say I've done a broad survey, but the third-party documentation I've read on comprehensions typically glosses over the scoping issues without mentioning them. To the extent that scoping is even hinted at, comprehensions are treated as expressions which are exactly equivalent to re-writing them as a for-loop in the current scope.
This is a typical example, found as the top result on googling for "python comprehensions":
Nothing is mentioned about scope, and it repeats the inaccurate but simple equivalency:
for item in list: if conditional: expression
But perhaps that tutorial is too old. Okay this recent one is only a little more than a year old:
Again, no mention of scoping issues, comprehensions are simply expressions which presumably run in the same scope as any other expression.
I think you over-estimate how many newcomers to Python are even aware that the scope of comprehensions is something to consider.
I put quite a bit of work into making it possible for folks to gloss over the distinction and still come to mostly-correct conclusions about how particular code snippets would behave.
I was only able to achieve it because the folks that designed lexical scoping before me had made *read* access to lexical scopes almost entirely transparent, and because generator expressions were designed to fail fast if there was a bug in the expression defining the outermost iterable (which meant that even at class scope, the outermost iterable expression still had access to class level variables, because it was evaluated *outside* the nested scope).
*Write* access to lexically nested scopes, by contrast, was omitted entirely from the original lexical scoping design, and when it was later added by https://www.python.org/dev/peps/pep-3104/, it was done using an explicit "nonlocal" declaration statement (expressly akin to "global"), and PEP 3099 explicitly ruled out the use of ":=" to implicitly declare the target name as being non-local.
PEP 572 is thus taking the position that:
- we now want to make write access to outer scopes implicit (despite PEP 3099 explicitly ruling that out as desired design feature) - but only in comprehensions and generator expressions (not lambda expressions, and not full nested functions) - and only for assignment expressions, not for loop iteration variables - and we want it to implicitly choose between a "global NAME" declaration and a "nonlocal NAME" declaration based on where the comprehension is defined - and this is OK because "nobody" actually understands how comprehensions hide the iteration variable in practice, and "everybody" thinks they're still a simple for loop like they were in Python 2 - the fact that the language reference, the behaviour at class scopes, the disassembly output, and the behaviour in a debugger all indicate that comprehensions are full nested scopes isn't important
This level of additional complication and complexity in the scoping semantics simply isn't warranted for such a minor readability enhancement as assignment expressions.
P.S. "You did such a good job of minimising the backwards compatibility breakage when we changed the semantics of scoping in comprehensions that we now consider your opinion on reasonable scoping semantics for comprehensions to be irrelevant, because everybody else still thinks they work the same way as they did in Python 2" is such a surreal position for folks to be taking that I'm having trouble working out how to effectively respond to it.
Guido has complained that "I keep changing my mind about what I want", but that's not what's actually going on: what I want is to keep folks from taking our already complicated scoping semantics and making it close to impossible for anyone to ever infer how they work from experimentation at the interactive prompt. That goal has pretty much stayed consistent since the parent local scoping proposal was first put forward.
What keeps changing is my tactics in pursuing that goal, based on my current perception of what the folks pushing that proposal are actually trying to achieve (which seems to be some combination of "We want to pretend that the Python 3 scoping changes in comprehensions never happened, but we still want to avoid leaking the iteration variables somehow" and "We want to enable new clever tricks with state export from comprehensions and generator expressions"), as well as repeatedly asking myself what *actually* bothers me about the proposal (which I've now distilled down to just the comprehension scoping issue, and the reliance on an arbitrary syntactic restriction against top level usage to avoid competing with traditional assignment statements).