<div dir="ltr"><div>So IIUC you are okay with the behavior described by the PEP but you want an explicit language feature to specify it?</div><div><br></div><div>I don't particularly like adding a `parentlocal` statement to the language, because I don't think it'll be generally useful. (We don't have `goto` in the language even though it could be used in the formal specification of `if`, for example. :-)</div><div><br></div><div>But as a descriptive mechanism to make the PEP's spec clearer I'm fine with it. Let's call it `__parentlocal` for now. It would work a bit like `nonlocal` but also different, since in the normal case (when there's no matching `nonlocal` in the parent scope) it would make the target a local in that scope rather than trying to look for a definition of the target name in surrounding (non-class, non-global) scopes. Also if there's a matching `global` in the parent scope, `__parentlocal` itself changes its meaning to `global`. If you want to push a target through several level of target scopes you can do that by having a `__parentlocal` in each scope that it should push through (this is needed for nested comprehensions, see below).<br></div><div><br></div><div>Given that definition of `__parentlocal`, in first approximation the scoping rule proposed by PEP 572 would then be: In comprehensions (which in my use in the PEP 572 discussion includes generator expressions) the targets of inline assignments are automatically endowed with a `__parentlocal` declaration, except inside the "outermost iterable" (since that already runs in the parent scope).</div><div><br></div><div>There would have to be additional words when comprehensions themselves are nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's intention is that inline assignments anywhere there end up targeting the scope containing the outermost comprehension. But this can all be expressed by adding `__parentlocal` for various variables in various places (including in the "outermost iterable" of inner comprehensions).<br></div><div><br></div><div>I'd also like to keep the rule prohibiting use of the same name as a comprehension loop control variable and as an inline assignment target; this rule would also prohibit shenanigans with nested comprehensions (for any set of nested comprehensions, any name that's a loop control variable in any of them cannot be an inline assignment target in any of them). This would also apply to the "outermost iterable".</div><div><br></div><div>Does this help at all, or did I miss something?</div><div><br></div><div>--Guido<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jun 27, 2018 at 5:27 AM Nick Coghlan <<a href="mailto:ncoghlan@gmail.com">ncoghlan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 26 June 2018 at 02:27, Guido van Rossum <<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>> wrote:<br>
> [This is my one reply in this thread today. I am trying to limit the amount<br>
> of time I spend to avoid another overheated escalation.]<br>
<br>
Aye, I'm trying to do the same, and deliberately spending some<br>
evenings entirely offline is helping with that :)<br>
<br>
> On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan <<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>> wrote:<br>
>><br>
>> Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or<br>
>> 'NAME := EXPR'?" bothers me a bit, but it's the implementation<br>
>> implications of parent local scoping that I fear will create a<br>
>> semantic tar pit we can't get out of later.<br>
><br>
> Others have remarked this too, but it really bother me that you are focusing<br>
> so much on the implementation of parent local scoping rather than on the<br>
> "intuitive" behavior which is super easy to explain -- especially to someone<br>
> who isn't all that familiar (or interested) with the implicit scope created<br>
> for the loop control variable(s). According to Steven (who noticed that this<br>
> is barely mentioned in most tutorials about comprehensions) that is most<br>
> people, however very few of them read python-dev.<br>
><br>
> It's not that much work for the compiler, since it just needs to do a little<br>
> bit of (new) static analysis and then it can generate the bytecode to<br>
> manipulate closure(s). The runtime proper doesn't need any new<br>
> implementation effort. The fact that sometimes a closure must be introduced<br>
> where no explicit initialization exists is irrelevant to the runtime -- this<br>
> only affects the static analysis, at runtime it's no different than if the<br>
> explicit initialization was inside `if 0`.<br>
<br>
One of the things I prize about Python's current code generator is how<br>
many of the constructs can be formulated as simple content-and-context<br>
independent boilerplate removal, which is why parent local scoping (as<br>
currently defined in PEP 572) bothers me: rather than being a new<br>
primitive in its own right, the PEP instead makes the notion of "an<br>
assignment expression in a comprehension or generator expression" a<br>
construct that can't readily decomposed into lower level building<br>
blocks the way that both assignment expressions on their own and<br>
comprehensions and generator expressions on their own can be. Instead,<br>
completely new language semantics arise from the interaction between<br>
two otherwise independent features.<br>
<br>
Even changes as complicated as PEP 343's with statement, PEP 380's<br>
yield from, and PEP 492's native coroutines all include examples of<br>
how they could be written *without* the benefit of the new syntax.<br>
<br>
By contrast, PEP 572's parent local scoping can't currently be defined<br>
that way. Instead, to explain how the code generator is going to be<br>
expected to handle comprehensions, you have to take the current<br>
comprehension semantics and add two new loops to link up the bound<br>
names correctly::<br>
<br>
[item := x for x in items]<br>
<br>
becomes:<br>
<br>
# Each bound name gets declared as local in the parent scope<br>
if 0:<br>
for item in (): pass<br>
def _list_comp(_outermost_iter):<br>
# Each bound name gets declared as:<br>
# - nonlocal if outer scope is a function scope<br>
# - global item if outer scope is a module scope<br>
# - an error, otherwise<br>
_result = []<br>
for x in _outermost_iter:<br>
_result.append(x)<br>
return _result<br>
<br>
_expr_result = _list_comp(items)<br>
<br>
This is why my objections would be reduced significantly if the PEP<br>
explicitly admitted that it was defining a new kind of scoping<br>
semantics, and actually made those semantics available as an explicit<br>
"parentlocal NAME" declaration (behind a "from __future__ import<br>
parent_locals" guard), such that the translation of the above example<br>
to an explicitly nested scope could just be the visually<br>
straightforward::<br>
<br>
def _list_comp(_outermost_iter):<br>
parentlocal item<br>
_result = []<br>
for x in _outermost_iter:<br>
item = x<br>
_result.append(x)<br>
return _result<br>
<br>
_expr_result = _list_comp(items)<br>
<br>
That splits up the learning process for anyone trying to really<br>
understand how this particular aspect of Python's code generation<br>
works into two distinct pieces:<br>
<br>
- "assignment expressions inside comprehensions and generator<br>
expressions use parent local scoping"<br>
- "parent local scoping works <the way that PEP 572 defines it>"<br>
<br>
If the PEP did that, we could likely even make parent locals work<br>
sensibly for classes by saying that "parent local" for a method<br>
definition in a class body refers to the closure namespace where we<br>
already stash __class__ references for the benefit of zero-arg super<br>
(this would also be a far more robust way of defining private class<br>
variables than name mangling is able to offer).<br>
<br>
Having parent locals available as a language level concept (rather<br>
than solely as an interaction between assignment expressions and<br>
implicitly nested scopes) also gets us to a point where<br>
context-independent code thunks that work both at module level and<br>
inside another function can be built as nested functions which declare<br>
all their working variables as parentlocal (you still need to define<br>
the thunks inline in the scope you want them to affect, since this<br>
isn't dynamic scoping, but when describing the code, you don't need to<br>
say "as a module level function define it this way, as a nested<br>
function define it that way").<br>
<br>
An explicit "parentlocal NAME" concept at the PEP 572 layer would also<br>
change the nature of the draft "given" proposal from competing with<br>
PEP 572, to instead being a follow-up proposal that focused on<br>
providing control of target name declarations in lambda expressions,<br>
comprehensions, and generator expressions such that:<br>
<br>
- (lambda arg: value := arg given parentlocal value) # Exports "value"<br>
to parent scope<br>
- any(x for x in items given parentlocal x) # Exports "x" to parent scope<br>
- [y for x in data if (y := f(x)) given y] # *Avoids* exporting "y" to<br>
parent scope<br>
<br>
With parent local scoping in the mix the proposed "given" syntax could<br>
also dispense with initialiser and type hinting support entirely and<br>
instead only allow:<br>
<br>
- "... given NAME" (always local, no matter the default scoping)<br>
- "... given parentlocal NAME" (always parent local, declaring if necessary)<br>
- "... given nonlocal NAME" (always nonlocal, error if not declared in<br>
outer scope)<br>
- "... given global NAME" (always global, no matter how nested the<br>
current scope is)<br>
- "... given (TARGET1, TARGET2, ...)" (declaring multiple assignment targets)<br>
<br>
If you want an initialiser or a type hint, then you'd use parentlocal<br>
semantics. If you want to keep names local (e.g. to avoid exporting<br>
them as part of a module's public API) then you can do that, too.<br>
<br>
>> Unfortunately, I think the key rationale for (b) is that if you<br>
>> *don't* do something along those lines, then there's a different<br>
>> strange scoping discrepancy that arises between the non-comprehension<br>
>> forms of container displays and the comprehension forms:<br>
>><br>
>> (NAME := EXPR,) # Binds a local<br>
>> tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local<br>
>> [...]<br>
>> Those scoping inconsistencies aren't *new*, but provoking them<br>
>> currently involves either class scopes, or messing about with<br>
>> locals().<br>
><br>
> In what sense are they not new? This syntax doesn't exist yet.<br>
<br>
The simplest way to illustrate the scope distinction today is with<br>
"len(locals())":<br>
<br>
>>> [len(locals()) for i in range(1)]<br>
[2]<br>
>>> [len(locals())]<br>
[7]<br>
<br>
But essentially nobody ever does that, so the distinction doesn't<br>
currently matter.<br>
<br>
By contrast, where assignment expressions bind their targets matters a<br>
*lot*, so PEP 572 makes the existing scoping oddities a lot more<br>
significant.<br>
<br>
> You left out another discrepancy, which is more likely to hit people in the<br>
> face: according to your doctrine, := used in the "outermost iterable" would<br>
> create a local in the containing scope, since that's where the outermost<br>
> iterable is evaluated. So in this example<br>
><br>
> a = [x := i+1 for i in range(y := 2)]<br>
><br>
> the scope of x would be the implicit function (i.e. it wouldn't leak) while<br>
> the scope of y would be the same as that of a. (And there's an even more<br>
> cryptic example, where the same name is assigned in both places.)<br>
<br>
Yeah, the fact it deals with this problem nicely is one aspect of the<br>
parent local scoping that I find genuinely attractive.<br>
<br>
>> Parent local scoping tries to mitigate the surface inconsistency by<br>
>> changing how write semantics are defined for implicitly nested scopes,<br>
>> but that comes at the cost of making those semantics inconsistent with<br>
>> explicitly nested scopes and with the read semantics of implicitly<br>
>> nested scopes.<br>
><br>
><br>
> Nobody thinks about write semantics though -- it's simply not the right<br>
> abstraction to use here, you've introduced it because that's how *you* think<br>
> about this.<br>
<br>
The truth of the last part of that paragraph means that the only way<br>
for the first part of it to be true is to decide that my way of<br>
thinking is *so* unusual that nobody else in the 10 years that Python<br>
3 has worked the way it does now has used the language reference, the<br>
source code, the disassembler, or the debugger to formulate a similar<br>
mental model of how they expect comprehensions and generator<br>
expressions to behave.<br>
<br>
I'll grant that I may be unusual in thinking about comprehensions and<br>
generator expressions the way I do, and I definitely accept that most<br>
folks simply don't think about the subtleties of how they handle<br>
scopes in the first place, but I *don't* accept the assertion that I'm<br>
unique in thinking about them that way. There are simply too many edge<br>
cases in their current runtime behaviour where the "Aha!" moment at<br>
the end of a debugging effort is going to be the realisation that<br>
they're implemented as an implicitly nested scope, and we've had a<br>
decade of Python 3 use where folks prone towards writing overly clever<br>
comprehensions have been in a position to independently make that<br>
discovery.<br>
<br>
>> The early iterations of PEP 572 tried to duck this whole realm of<br>
>> potential semantic inconsistencies by introducing sublocal scoping<br>
<br>
> There was also another variant in some iteration or PEP 572, after sublocal<br>
> scopes were already eliminated -- a change to comprehensions that would<br>
> evaluate the innermost iterable in the implicit function. This would make<br>
> the explanation of inline assignment in comprehensions consistent again<br>
> (they were always local to the comprehension in that iteration of the PEP),<br>
> at the cost of a backward incompatibility that was ultimately withdrawn.<br>
<br>
Yeah, the current "given" draft has an open question around the idea<br>
of having the presence of a "given" clause pull the outermost iterable<br>
evaluation inside the nested scope. It still doesn't really solve the<br>
problem, though, so I think I'd actually consider<br>
PEP-572-with-explicit-parent-local-scoping-support the version of<br>
assignment expressions that most cleanly handles the interaction with<br>
comprehension scopes without making that interaction rely on opaque<br>
magic (instead, it would be relying on an implicit target scope<br>
declaration, the same as any other name binding - the only unusual<br>
aspect is that the implicit declaration would be "parentlocal NAME"<br>
rather than the more typical local variable declaration).<br>
<br>
Cheers,<br>
Nick.<br>
<br>
-- <br>
Nick Coghlan | <a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a> | Brisbane, Australia<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido">python.org/~guido</a>)</div>