<div dir="ltr"><div><div><div><div><div><div><div><div class="gmail_extra"><div class="gmail_quote">On 10 May 2018 at 23:47, Tim Peters <span dir="ltr"><<a href="mailto:tim.peters@gmail.com" target="_blank">tim.peters@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">...<br>
<br>
[Guido]<br>
<span class="gmail-">>> You should really read Tim's initial post in this thread, where he<br>
>> explains his motivation.<br>
<br>
</span>[Nick]<br>
<span class="gmail-">> I did, and then I talked him out of it by pointing out how confusing it<br>
> would be to have the binding semantics of "x := y" be context dependent.<br>
<br>
</span>Ya, that was an effective Jedi mind trick when I was overdue to go to sleep ;-)<br>
<br>
To a plain user, there's nothing about a listcomp or genexp that says<br>
"new function introduced here". It looks like, for all the world,<br>
that it's running _in_ the block that contains it. It's magical<br>
enough that `for` targets magically become local. But that's almost<br>
never harmful magic, and often helpful, so worth it.<br>
<br>
<br>
>...<br>
<span class="gmail-">> It *is* different, because ":=" normally binds the same as any other name<br>
> binding operation including "for x in y:" (i.e. it creates a local<br>
> variable), while at comprehension scope, the proposal has now become for "x<br>
> := y" to create a local variable in the containing scope, while "for x in y"<br>
> doesn't.<br>
<br>
</span>":=" target names in a genexp/listcmp are treated exactly the same as<br>
any other non-for-target name: they resolve to the same scope as they<br>
resolve to in the block that contains them. The only twist is that if<br>
such a name `x` isn't otherwise known in the block, then `x` is<br>
established as being local to the block (which incidentally also<br>
covers the case when the genexp/listcomp is at module level, where<br>
"local to the block" and "global to the block" mean the same thing).<br>
Class scope may be an exception (I cheerfully never learned anything<br>
about how class scope works, because I don't write insane code ;-) ).<br></blockquote><div><br><br></div></div>That's all well and good, but it is *completely insufficient for the language specification*. For the language spec, we have to be able to tell implementation authors exactly how all of the "bizarre edge case" that you're attempting to hand wave away should behave by updating <a href="https://docs.python.org/dev/reference/expressions.html#displays-for-lists-sets-and-dictionaries">https://docs.python.org/dev/reference/expressions.html#displays-for-lists-sets-and-dictionaries</a> appropriately. It isn't 1995 any more - while CPython is still the reference implementation for Python, we're far from being the only implementation, which means we have to be a lot more disciplined about how much we leave up to the implementation to define.<br><br></div><div class="gmail_extra">The expected semantics for locals() are already sufficiently unclear that they're a source of software bugs (even in CPython) when attempting to run things under a debugger or line profiler (or anything else that sets a trace function). See <a href="https://www.python.org/dev/peps/pep-0558/">https://www.python.org/dev/peps/pep-0558/</a> for details.<br><br></div><div class="gmail_extra">"Comprehension scopes are already confusing, so it's OK to dial their weirdness all the way up to 11" is an *incredibly* strange argument to be attempting to make when the original better defined sublocal scoping proposal was knocked back as being overly confusing (even after it had been deliberately simplified by prohibiting nonlocal access to sublocals).<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Right now, the learning process for picking up the details of comprehension scopes goes something like this:<br><br></div><div class="gmail_extra">* make the technically-incorrect-but-mostly-reliable-in-the-absence-of-name-shadowing assumption that "[x for x in data]" is semantically equivalent to a for loop (especially common for experienced Py2 devs where this really was the case!):<br><br></div><div class="gmail_extra"> _result = []<br></div> for x in data:<br><div class="gmail_extra"> _result.append(x)<br></div> <br><div class="gmail_extra">* discover that "[x for x in data]" is actually semantically equivalent to "list(x for x in data)" (albeit without the name lookup and optimised to avoid actually creating the generator-iterator)<br></div><div class="gmail_extra">* make the still-technically-incorrect-but-even-more-reliable assumption that the generator expression "(x for x in data)" is equivalent to<br><br></div><div class="gmail_extra"> def _genexp():<br></div><div class="gmail_extra"> for x in data:<br></div><div class="gmail_extra"> yield x<br><br></div><div class="gmail_extra"> _result = _genexp()<br><br></div><div class="gmail_extra">* *maybe* discover that even the above expansion isn't quite accurate, and that the underlying semantic equivalent is actually this (one way to discover this by accident is to have a name error in the outermost iterable expression):<br><div class="gmail_extra"><br></div><div class="gmail_extra"> def _genexp(_outermost_iter):<br></div><div class="gmail_extra"> for x in _outermost_iter:<br></div><div class="gmail_extra"> yield x<br><br></div><div class="gmail_extra"> _result = _genexp(_outermost_iter)<br><br></div><div class="gmail_extra">* and then realise that the optimised list comprehension form is essentially this:<br><br></div><div class="gmail_extra"> def _listcomp(_outermost_iter):<br></div><div class="gmail_extra"> result = []<br></div><div class="gmail_extra"> for x in _outermost_iter:<br></div><div class="gmail_extra"> result.append(x)<br></div><div class="gmail_extra"> return result<br><br></div><div class="gmail_extra"> _result = _listcomp(data)<br></div><br></div>Now that "yield" in comprehensions has been prohibited, you've learned all the edge cases at that point - all of the runtime behaviour of things like name references, locals(), lambda expressions that close over the iteration variable, etc can be explained directly in terms of the equivalent functions and generators, so while comprehension iteration variable hiding may *seem* magical, it's really mostly explained by the deliberate semantic equivalence between the comprehension form and the constructor+genexp form. (That's exactly how PEP 3100 describes the change: "Have list comprehensions be syntactic sugar for passing an
equivalent generator expression to <tt class="gmail-docutils gmail-literal">list()</tt>; as a consequence the
loop variable will no longer be exposed")<br><br></div>As such, any proposal to have name bindings behave differently in comprehension and generator expression scope from the way they would behave in the equivalent nested function definitions *must be specified to an equivalent level of detail as the status quo*.<br><br></div>All of the attempts at such a definition that have been made so far have been riddled with action and a distance and context-dependent compilation requirements:<br><br></div>* whether to implicitly declare the binding target as nonlocal or global depends on whether or not you're at module scope or inside a function<br></div>* the desired semantics at class scope have been left largely unclear<br></div><div>* the desired semantics in the case of nested comprehensions and generator expressions has been left entirely unclear<br></div><div><br></div>Now, there *are* ways to resolve these problems in a coherent way, and that would be to define "parent local scoping" as a new scope type, and introduce a corresponding "parentlocal NAME" compiler declaration to explicitly request those semantics for bound names (allowing the expansions of comprehensions and generator expressions as explicitly nested functions to be adjusted accordingly).<br><br></div><div>But the PEP will need to state explicitly that that's what it is doing, and fully specify how those new semantics are expected to work in *all* of the existing scope types, not just the two where the desired behaviour is relatively easy to define in terms of nonlocal and global.<br></div><div><br><div>Regards,<br></div><div>Nick.<br></div><div><br><div><div class="gmail_extra">-- <br><div class="gmail_signature">Nick Coghlan | <a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a> | Brisbane, Australia</div>
</div></div></div></div></div><div><div><div><div><div></div></div></div></div></div></div>