[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Steven D'Aprano]

On Sun, Jun 24, 2018 at 04:33:38PM +1000, Nick Coghlan wrote:
[...]
> > Making the intentional choice to use an assignment expression is not
> > really "implicit" in any meaningful sense.
> 
> No, it's actually implicit: there's an extra "global NAME" or
> "nonlocal NAME" in the equivalent code for a comprehension that isn't
> there in the as-written source code, and doesn't get emitted for a
> regular assignment expression or for the iteration variable in a
> comprehension - it only shows up due to the defined interaction
> between comprehensions and assignment expressions.

You seem to be talking about an implementation which could change in the 
future. I'm talking semantics of the proposed language feature. As a 
programmer writing Python code, I have no visibility into the 
implementation. The implementation could change ten times a day for all 
I care, so long as the semantics remain the same.

I want the desired semantics to drive the implementation, not the 
other way around.

You seem to want the implementation to drive the semantics, by 
eliminating the proposed feature because it doesn't match your 
deep understanding of the implementation as a nested function.

I want this feature because its useful, and without it the use-cases 
for assignment expressions are significantly reduced.

As far as "implicit", for the sake of the discussion, I'll grant you 
that one. Okay, the proposed behaviour will implicitly enable 
comprehensions to export their state.

Now what? Is that a good thing or a bad thing?

If "implicit" (with or without the scare quotes) is such a bad thing to 
be avoided, why are comprehensions implemented using an implicit 
function?


> The problem I have with PEP 572 is that it proposes *breaking that
> otherwise universal pattern* - instead of having assignment
> expressions in comprehensions implicitly declare the name as local in
> the nested comprehension scope, it instead has them:

You talk about "nested comprehension scope", and that's a critical 
point, but I'm going to skip answering that for now. I have a draft 
email responding to another of your posts on that topic, which I hope to 
polish in the next day.


> 1. implicitly declare the name as global or as nonlocal in the
> comprehension (or else raise an exception), depending on the nature of
> the parent scope where the comprehension is used
> 2. in the nonlocal reference case, amend the symbol table analysis to
> act like there was a preceding "if 0:\n for NAME in ():\n pass" in the
> parent scope (so the compiler knows which outer function scope to
> target)

If it is okay for you to amend the list comprehension to behave as if it 
were wrapped in an implicit nested function, why shouldn't it be okay to 
behave as if assignments inside the comprehension included an implicit 
nonlocal declaration?


> The rationale being given for why that is OK is:
> 
> 1. "Everyone" thinks comprehensions are just a for loop (even though
> that hasn't been true for the better part of a decade, and was never
> true for generator expressions)

Obviously "everyone" is an exaggeration, but, yes, I stand by that -- 
most people don't even give comprehension scope a thought until they 
get bitten by it.

Either because (Python 2) they don't realise the loop variable is local 
to their current scope:

http://www.librador.com/2014/07/10/Variable-scope-in-list-comprehension-vs-generator-expression/

or (Python 3) they get bitten by the change:

https://old.reddit.com/r/Python/comments/425qmb/strange_python_27_34_difference/

(As is so often the case, whatever behaviour we choose, we're going to 
surprise somebody.)

It is hardly surprising that people don't think too hard about scoping 
of comprehensions. Without a way to perform assignments inside 
comprehensions, aside from the loop variables themselves, there's 
nothing going on inside a comprehension where it makes a visible 
difference whether it is a local scope or a sublocal scope.

*IF* assignment expressions are introduced, that is going to change. We 
have some choices:

1. Keep assignment expressions encapsulated in their implicit function, 
and be prepared for people to be annoyed because (with no way to declare 
them global or non-local inside an expression), they can't use them to 
get data in and out of the comprehension.


2. Allow assignment expressions to be exported out of the comprehension, 
and be prepared for people to be annoyed because they clobbered a local.

(But for the reasons Tim Peters has already set out, I doubt this will 
happen often.)


3. Allow some sort of extra comprehension syntax to allow 
global/nonlocal declarations inside comprehensions.

    x = 1
    [nonlocal x := x+i for i in sequence]

(Hmmm... I thought I would hate that more than I actually do.)


4. Have some sort of cunning plan whereby if the variable in question 
exists in the local scope, it is implicitly local inside the 
comprehension:

    x = 1
    [x := i+1 for i in (1, 2)]
    assert x == 3

but if it doesn't, then the variable is implicitly sublocal inside the 
comprehension:

    del x
    [x := i+1 for i in (1, 2)]
    x  # raises NameError



Remember, the driving use-case which started this (ever-so-long) 
discussion was the ability to push data into a comprehension and then 
update it on each iteration, something like this:

    x = initial_value()
    results = [x := transform(x, i) for i in sequence]

Please, Nick, take your implementor's hat off, forget everything you 
know about the implementation of comprehensions and their implicit 
nested function, and tell me that doesn't look like it should work.




-- 
Steve