[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
Guido van Rossum
guido at python.org
Wed Jun 27 18:31:42 EDT 2018
So IIUC you are okay with the behavior described by the PEP but you want an
explicit language feature to specify it?
I don't particularly like adding a `parentlocal` statement to the language,
because I don't think it'll be generally useful. (We don't have `goto` in
the language even though it could be used in the formal specification of
`if`, for example. :-)
But as a descriptive mechanism to make the PEP's spec clearer I'm fine with
it. Let's call it `__parentlocal` for now. It would work a bit like
`nonlocal` but also different, since in the normal case (when there's no
matching `nonlocal` in the parent scope) it would make the target a local
in that scope rather than trying to look for a definition of the target
name in surrounding (non-class, non-global) scopes. Also if there's a
matching `global` in the parent scope, `__parentlocal` itself changes its
meaning to `global`. If you want to push a target through several level of
target scopes you can do that by having a `__parentlocal` in each scope
that it should push through (this is needed for nested comprehensions, see
below).
Given that definition of `__parentlocal`, in first approximation the
scoping rule proposed by PEP 572 would then be: In comprehensions (which in
my use in the PEP 572 discussion includes generator expressions) the
targets of inline assignments are automatically endowed with a
`__parentlocal` declaration, except inside the "outermost iterable" (since
that already runs in the parent scope).
There would have to be additional words when comprehensions themselves are
nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's
intention is that inline assignments anywhere there end up targeting the
scope containing the outermost comprehension. But this can all be expressed
by adding `__parentlocal` for various variables in various places
(including in the "outermost iterable" of inner comprehensions).
I'd also like to keep the rule prohibiting use of the same name as a
comprehension loop control variable and as an inline assignment target;
this rule would also prohibit shenanigans with nested comprehensions (for
any set of nested comprehensions, any name that's a loop control variable
in any of them cannot be an inline assignment target in any of them). This
would also apply to the "outermost iterable".
Does this help at all, or did I miss something?
--Guido
On Wed, Jun 27, 2018 at 5:27 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 26 June 2018 at 02:27, Guido van Rossum <guido at python.org> wrote:
> > [This is my one reply in this thread today. I am trying to limit the
> amount
> > of time I spend to avoid another overheated escalation.]
>
> Aye, I'm trying to do the same, and deliberately spending some
> evenings entirely offline is helping with that :)
>
> > On Mon, Jun 25, 2018 at 4:44 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>
> >> Right, the proposed blunt solution to "Should I use 'NAME = EXPR' or
> >> 'NAME := EXPR'?" bothers me a bit, but it's the implementation
> >> implications of parent local scoping that I fear will create a
> >> semantic tar pit we can't get out of later.
> >
> > Others have remarked this too, but it really bother me that you are
> focusing
> > so much on the implementation of parent local scoping rather than on the
> > "intuitive" behavior which is super easy to explain -- especially to
> someone
> > who isn't all that familiar (or interested) with the implicit scope
> created
> > for the loop control variable(s). According to Steven (who noticed that
> this
> > is barely mentioned in most tutorials about comprehensions) that is most
> > people, however very few of them read python-dev.
> >
> > It's not that much work for the compiler, since it just needs to do a
> little
> > bit of (new) static analysis and then it can generate the bytecode to
> > manipulate closure(s). The runtime proper doesn't need any new
> > implementation effort. The fact that sometimes a closure must be
> introduced
> > where no explicit initialization exists is irrelevant to the runtime --
> this
> > only affects the static analysis, at runtime it's no different than if
> the
> > explicit initialization was inside `if 0`.
>
> One of the things I prize about Python's current code generator is how
> many of the constructs can be formulated as simple content-and-context
> independent boilerplate removal, which is why parent local scoping (as
> currently defined in PEP 572) bothers me: rather than being a new
> primitive in its own right, the PEP instead makes the notion of "an
> assignment expression in a comprehension or generator expression" a
> construct that can't readily decomposed into lower level building
> blocks the way that both assignment expressions on their own and
> comprehensions and generator expressions on their own can be. Instead,
> completely new language semantics arise from the interaction between
> two otherwise independent features.
>
> Even changes as complicated as PEP 343's with statement, PEP 380's
> yield from, and PEP 492's native coroutines all include examples of
> how they could be written *without* the benefit of the new syntax.
>
> By contrast, PEP 572's parent local scoping can't currently be defined
> that way. Instead, to explain how the code generator is going to be
> expected to handle comprehensions, you have to take the current
> comprehension semantics and add two new loops to link up the bound
> names correctly::
>
> [item := x for x in items]
>
> becomes:
>
> # Each bound name gets declared as local in the parent scope
> if 0:
> for item in (): pass
> def _list_comp(_outermost_iter):
> # Each bound name gets declared as:
> # - nonlocal if outer scope is a function scope
> # - global item if outer scope is a module scope
> # - an error, otherwise
> _result = []
> for x in _outermost_iter:
> _result.append(x)
> return _result
>
> _expr_result = _list_comp(items)
>
> This is why my objections would be reduced significantly if the PEP
> explicitly admitted that it was defining a new kind of scoping
> semantics, and actually made those semantics available as an explicit
> "parentlocal NAME" declaration (behind a "from __future__ import
> parent_locals" guard), such that the translation of the above example
> to an explicitly nested scope could just be the visually
> straightforward::
>
> def _list_comp(_outermost_iter):
> parentlocal item
> _result = []
> for x in _outermost_iter:
> item = x
> _result.append(x)
> return _result
>
> _expr_result = _list_comp(items)
>
> That splits up the learning process for anyone trying to really
> understand how this particular aspect of Python's code generation
> works into two distinct pieces:
>
> - "assignment expressions inside comprehensions and generator
> expressions use parent local scoping"
> - "parent local scoping works <the way that PEP 572 defines it>"
>
> If the PEP did that, we could likely even make parent locals work
> sensibly for classes by saying that "parent local" for a method
> definition in a class body refers to the closure namespace where we
> already stash __class__ references for the benefit of zero-arg super
> (this would also be a far more robust way of defining private class
> variables than name mangling is able to offer).
>
> Having parent locals available as a language level concept (rather
> than solely as an interaction between assignment expressions and
> implicitly nested scopes) also gets us to a point where
> context-independent code thunks that work both at module level and
> inside another function can be built as nested functions which declare
> all their working variables as parentlocal (you still need to define
> the thunks inline in the scope you want them to affect, since this
> isn't dynamic scoping, but when describing the code, you don't need to
> say "as a module level function define it this way, as a nested
> function define it that way").
>
> An explicit "parentlocal NAME" concept at the PEP 572 layer would also
> change the nature of the draft "given" proposal from competing with
> PEP 572, to instead being a follow-up proposal that focused on
> providing control of target name declarations in lambda expressions,
> comprehensions, and generator expressions such that:
>
> - (lambda arg: value := arg given parentlocal value) # Exports "value"
> to parent scope
> - any(x for x in items given parentlocal x) # Exports "x" to parent scope
> - [y for x in data if (y := f(x)) given y] # *Avoids* exporting "y" to
> parent scope
>
> With parent local scoping in the mix the proposed "given" syntax could
> also dispense with initialiser and type hinting support entirely and
> instead only allow:
>
> - "... given NAME" (always local, no matter the default scoping)
> - "... given parentlocal NAME" (always parent local, declaring if
> necessary)
> - "... given nonlocal NAME" (always nonlocal, error if not declared in
> outer scope)
> - "... given global NAME" (always global, no matter how nested the
> current scope is)
> - "... given (TARGET1, TARGET2, ...)" (declaring multiple assignment
> targets)
>
> If you want an initialiser or a type hint, then you'd use parentlocal
> semantics. If you want to keep names local (e.g. to avoid exporting
> them as part of a module's public API) then you can do that, too.
>
> >> Unfortunately, I think the key rationale for (b) is that if you
> >> *don't* do something along those lines, then there's a different
> >> strange scoping discrepancy that arises between the non-comprehension
> >> forms of container displays and the comprehension forms:
> >>
> >> (NAME := EXPR,) # Binds a local
> >> tuple(NAME := EXPR for __ in range(1)) # Doesn't bind a local
> >> [...]
> >> Those scoping inconsistencies aren't *new*, but provoking them
> >> currently involves either class scopes, or messing about with
> >> locals().
> >
> > In what sense are they not new? This syntax doesn't exist yet.
>
> The simplest way to illustrate the scope distinction today is with
> "len(locals())":
>
> >>> [len(locals()) for i in range(1)]
> [2]
> >>> [len(locals())]
> [7]
>
> But essentially nobody ever does that, so the distinction doesn't
> currently matter.
>
> By contrast, where assignment expressions bind their targets matters a
> *lot*, so PEP 572 makes the existing scoping oddities a lot more
> significant.
>
> > You left out another discrepancy, which is more likely to hit people in
> the
> > face: according to your doctrine, := used in the "outermost iterable"
> would
> > create a local in the containing scope, since that's where the outermost
> > iterable is evaluated. So in this example
> >
> > a = [x := i+1 for i in range(y := 2)]
> >
> > the scope of x would be the implicit function (i.e. it wouldn't leak)
> while
> > the scope of y would be the same as that of a. (And there's an even more
> > cryptic example, where the same name is assigned in both places.)
>
> Yeah, the fact it deals with this problem nicely is one aspect of the
> parent local scoping that I find genuinely attractive.
>
> >> Parent local scoping tries to mitigate the surface inconsistency by
> >> changing how write semantics are defined for implicitly nested scopes,
> >> but that comes at the cost of making those semantics inconsistent with
> >> explicitly nested scopes and with the read semantics of implicitly
> >> nested scopes.
> >
> >
> > Nobody thinks about write semantics though -- it's simply not the right
> > abstraction to use here, you've introduced it because that's how *you*
> think
> > about this.
>
> The truth of the last part of that paragraph means that the only way
> for the first part of it to be true is to decide that my way of
> thinking is *so* unusual that nobody else in the 10 years that Python
> 3 has worked the way it does now has used the language reference, the
> source code, the disassembler, or the debugger to formulate a similar
> mental model of how they expect comprehensions and generator
> expressions to behave.
>
> I'll grant that I may be unusual in thinking about comprehensions and
> generator expressions the way I do, and I definitely accept that most
> folks simply don't think about the subtleties of how they handle
> scopes in the first place, but I *don't* accept the assertion that I'm
> unique in thinking about them that way. There are simply too many edge
> cases in their current runtime behaviour where the "Aha!" moment at
> the end of a debugging effort is going to be the realisation that
> they're implemented as an implicitly nested scope, and we've had a
> decade of Python 3 use where folks prone towards writing overly clever
> comprehensions have been in a position to independently make that
> discovery.
>
> >> The early iterations of PEP 572 tried to duck this whole realm of
> >> potential semantic inconsistencies by introducing sublocal scoping
>
> > There was also another variant in some iteration or PEP 572, after
> sublocal
> > scopes were already eliminated -- a change to comprehensions that would
> > evaluate the innermost iterable in the implicit function. This would make
> > the explanation of inline assignment in comprehensions consistent again
> > (they were always local to the comprehension in that iteration of the
> PEP),
> > at the cost of a backward incompatibility that was ultimately withdrawn.
>
> Yeah, the current "given" draft has an open question around the idea
> of having the presence of a "given" clause pull the outermost iterable
> evaluation inside the nested scope. It still doesn't really solve the
> problem, though, so I think I'd actually consider
> PEP-572-with-explicit-parent-local-scoping-support the version of
> assignment expressions that most cleanly handles the interaction with
> comprehension scopes without making that interaction rely on opaque
> magic (instead, it would be relying on an implicit target scope
> declaration, the same as any other name binding - the only unusual
> aspect is that the implicit declaration would be "parentlocal NAME"
> rather than the more typical local variable declaration).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
>
--
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180627/d88c39c3/attachment-0001.html>
More information about the Python-Dev
mailing list