So IIUC you are okay with the behavior described by the PEP but you want an explicit language feature to specify it?
I don't particularly like adding a `parentlocal` statement to the language, because I don't think it'll be generally useful. (We don't have `goto` in the language even though it could be used in the formal specification of `if`, for example. :-)
But as a descriptive mechanism to make the PEP's spec clearer I'm fine with it. Let's call it `__parentlocal` for now. It would work a bit like `nonlocal` but also different, since in the normal case (when there's no matching `nonlocal` in the parent scope) it would make the target a local in that scope rather than trying to look for a definition of the target name in surrounding (non-class, non-global) scopes. Also if there's a matching `global` in the parent scope, `__parentlocal` itself changes its meaning to `global`. If you want to push a target through several level of target scopes you can do that by having a `__parentlocal` in each scope that it should push through (this is needed for nested comprehensions, see below).
Given that definition of `__parentlocal`, in first approximation the scoping rule proposed by PEP 572 would then be: In comprehensions (which in my use in the PEP 572 discussion includes generator expressions) the targets of inline assignments are automatically endowed with a `__parentlocal` declaration, except inside the "outermost iterable" (since that already runs in the parent scope).
There would have to be additional words when comprehensions themselves are nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's intention is that inline assignments anywhere there end up targeting the scope containing the outermost comprehension. But this can all be expressed by adding `__parentlocal` for various variables in various places (including in the "outermost iterable" of inner comprehensions).
I'd also like to keep the rule prohibiting use of the same name as a comprehension loop control variable and as an inline assignment target; this rule would also prohibit shenanigans with nested comprehensions (for any set of nested comprehensions, any name that's a loop control variable in any of them cannot be an inline assignment target in any of them). This would also apply to the "outermost iterable".
Does this help at all, or did I miss something?
On Wed, Jun 27, 2018 at 5:27 AM Nick Coghlan email@example.com wrote:
On 26 June 2018 at 02:27, Guido van Rossum firstname.lastname@example.org wrote:
[This is my one reply in this thread today. I am trying to limit the
of time I spend to avoid another overheated escalation.]
Aye, I'm trying to do the same, and deliberately spending some evenings entirely offline is helping with that :)
On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan email@example.com wrote:
Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or 'NAME := EXPR'?" bothers me a bit, but it's the implementation implications of parent local scoping that I fear will create a semantic tar pit we can't get out of later.
Others have remarked this too, but it really bother me that you are
so much on the implementation of parent local scoping rather than on the "intuitive" behavior which is super easy to explain -- especially to
who isn't all that familiar (or interested) with the implicit scope
for the loop control variable(s). According to Steven (who noticed that
is barely mentioned in most tutorials about comprehensions) that is most people, however very few of them read python-dev.
It's not that much work for the compiler, since it just needs to do a
bit of (new) static analysis and then it can generate the bytecode to manipulate closure(s). The runtime proper doesn't need any new implementation effort. The fact that sometimes a closure must be
where no explicit initialization exists is irrelevant to the runtime --
only affects the static analysis, at runtime it's no different than if
explicit initialization was inside `if 0`.
One of the things I prize about Python's current code generator is how many of the constructs can be formulated as simple content-and-context independent boilerplate removal, which is why parent local scoping (as currently defined in PEP 572) bothers me: rather than being a new primitive in its own right, the PEP instead makes the notion of "an assignment expression in a comprehension or generator expression" a construct that can't readily decomposed into lower level building blocks the way that both assignment expressions on their own and comprehensions and generator expressions on their own can be. Instead, completely new language semantics arise from the interaction between two otherwise independent features.
Even changes as complicated as PEP 343's with statement, PEP 380's yield from, and PEP 492's native coroutines all include examples of how they could be written *without* the benefit of the new syntax.
By contrast, PEP 572's parent local scoping can't currently be defined that way. Instead, to explain how the code generator is going to be expected to handle comprehensions, you have to take the current comprehension semantics and add two new loops to link up the bound names correctly::
[item := x for x in items]
# Each bound name gets declared as local in the parent scope if 0: for item in (): pass def _list_comp(_outermost_iter): # Each bound name gets declared as: # - nonlocal if outer scope is a function scope # - global item if outer scope is a module scope # - an error, otherwise _result =  for x in _outermost_iter: _result.append(x) return _result _expr_result = _list_comp(items)
This is why my objections would be reduced significantly if the PEP explicitly admitted that it was defining a new kind of scoping semantics, and actually made those semantics available as an explicit "parentlocal NAME" declaration (behind a "from __future__ import parent_locals" guard), such that the translation of the above example to an explicitly nested scope could just be the visually straightforward::
def _list_comp(_outermost_iter): parentlocal item _result =  for x in _outermost_iter: item = x _result.append(x) return _result _expr_result = _list_comp(items)
That splits up the learning process for anyone trying to really understand how this particular aspect of Python's code generation works into two distinct pieces:
- "assignment expressions inside comprehensions and generator
expressions use parent local scoping"
- "parent local scoping works <the way that PEP 572 defines it>"
If the PEP did that, we could likely even make parent locals work sensibly for classes by saying that "parent local" for a method definition in a class body refers to the closure namespace where we already stash __class__ references for the benefit of zero-arg super (this would also be a far more robust way of defining private class variables than name mangling is able to offer).
Having parent locals available as a language level concept (rather than solely as an interaction between assignment expressions and implicitly nested scopes) also gets us to a point where context-independent code thunks that work both at module level and inside another function can be built as nested functions which declare all their working variables as parentlocal (you still need to define the thunks inline in the scope you want them to affect, since this isn't dynamic scoping, but when describing the code, you don't need to say "as a module level function define it this way, as a nested function define it that way").
An explicit "parentlocal NAME" concept at the PEP 572 layer would also change the nature of the draft "given" proposal from competing with PEP 572, to instead being a follow-up proposal that focused on providing control of target name declarations in lambda expressions, comprehensions, and generator expressions such that:
- (lambda arg: value := arg given parentlocal value) # Exports "value"
to parent scope
- any(x for x in items given parentlocal x) # Exports "x" to parent scope
- [y for x in data if (y := f(x)) given y] # *Avoids* exporting "y" to
With parent local scoping in the mix the proposed "given" syntax could also dispense with initialiser and type hinting support entirely and instead only allow:
- "... given NAME" (always local, no matter the default scoping)
- "... given parentlocal NAME" (always parent local, declaring if
- "... given nonlocal NAME" (always nonlocal, error if not declared in
- "... given global NAME" (always global, no matter how nested the
current scope is)
- "... given (TARGET1, TARGET2, ...)" (declaring multiple assignment
If you want an initialiser or a type hint, then you'd use parentlocal semantics. If you want to keep names local (e.g. to avoid exporting them as part of a module's public API) then you can do that, too.
Unfortunately, I think the key rationale for (b) is that if you *don't* do something along those lines, then there's a different strange scoping discrepancy that arises between the non-comprehension forms of container displays and the comprehension forms:
(NAME := EXPR,) # Binds a local tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local
[...] Those scoping inconsistencies aren't *new*, but provoking them currently involves either class scopes, or messing about with locals().
In what sense are they not new? This syntax doesn't exist yet.
The simplest way to illustrate the scope distinction today is with "len(locals())":
>>> [len(locals()) for i in range(1)]  >>> [len(locals())] 
But essentially nobody ever does that, so the distinction doesn't currently matter.
By contrast, where assignment expressions bind their targets matters a *lot*, so PEP 572 makes the existing scoping oddities a lot more significant.
You left out another discrepancy, which is more likely to hit people in
face: according to your doctrine, := used in the "outermost iterable"
create a local in the containing scope, since that's where the outermost iterable is evaluated. So in this example
a = [x := i+1 for i in range(y := 2)]
the scope of x would be the implicit function (i.e. it wouldn't leak)
the scope of y would be the same as that of a. (And there's an even more cryptic example, where the same name is assigned in both places.)
Yeah, the fact it deals with this problem nicely is one aspect of the parent local scoping that I find genuinely attractive.
Parent local scoping tries to mitigate the surface inconsistency by changing how write semantics are defined for implicitly nested scopes, but that comes at the cost of making those semantics inconsistent with explicitly nested scopes and with the read semantics of implicitly nested scopes.
Nobody thinks about write semantics though -- it's simply not the right abstraction to use here, you've introduced it because that's how *you*
The truth of the last part of that paragraph means that the only way for the first part of it to be true is to decide that my way of thinking is *so* unusual that nobody else in the 10 years that Python 3 has worked the way it does now has used the language reference, the source code, the disassembler, or the debugger to formulate a similar mental model of how they expect comprehensions and generator expressions to behave.
I'll grant that I may be unusual in thinking about comprehensions and generator expressions the way I do, and I definitely accept that most folks simply don't think about the subtleties of how they handle scopes in the first place, but I *don't* accept the assertion that I'm unique in thinking about them that way. There are simply too many edge cases in their current runtime behaviour where the "Aha!" moment at the end of a debugging effort is going to be the realisation that they're implemented as an implicitly nested scope, and we've had a decade of Python 3 use where folks prone towards writing overly clever comprehensions have been in a position to independently make that discovery.
The early iterations of PEP 572 tried to duck this whole realm of potential semantic inconsistencies by introducing sublocal scoping
There was also another variant in some iteration or PEP 572, after
scopes were already eliminated -- a change to comprehensions that would evaluate the innermost iterable in the implicit function. This would make the explanation of inline assignment in comprehensions consistent again (they were always local to the comprehension in that iteration of the
at the cost of a backward incompatibility that was ultimately withdrawn.
Yeah, the current "given" draft has an open question around the idea of having the presence of a "given" clause pull the outermost iterable evaluation inside the nested scope. It still doesn't really solve the problem, though, so I think I'd actually consider PEP-572-with-explicit-parent-local-scoping-support the version of assignment expressions that most cleanly handles the interaction with comprehension scopes without making that interaction rely on opaque magic (instead, it would be relying on an implicit target scope declaration, the same as any other name binding - the only unusual aspect is that the implicit declaration would be "parentlocal NAME" rather than the more typical local variable declaration).
-- Nick Coghlan | firstname.lastname@example.org | Brisbane, Australia