[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Terry Reedy]

On 6/26/2018 10:36 PM, Guido van Rossum wrote:
> [This is my one response today]

Thank you for clearly presenting how you see 'comprehension', 'generator 
expression' and by implication 'equivalent code'.  The latter can either 
be a definition or an explanation.  The difference is subtle but real, 
and, I believe, the essence of the disagreement over iteration 
variables.  If the code equivalent to a comprehension is its definition, 
like a macro expansion, then survival of the iteration variable is to be 
expected.  If the equivalent code is an explanation of the *result* of 
evaluating a *self-contained expression*, then leakage is easily seen a 
wart, just as leakage of temporaries from any other expression would be.

My interpretation of what you say below is that you always wanted, for 
instance, [i*i for i in iterable] == [j*j for j in iterable] to be true, 
and saw the leakage making this not quite true as a wart.  In other 
words, within comprehensions (including generator expressions) 
iterations names should be regarded as dummy placeholders and not part 
of the value.

If this is correct, the list comprehension syntax could have been
   [\0 * \0 for \0 in iterable]
with  \1, \2, ... used as needed. (I am using the regex back-reference 
notation in a way similar to the use of str.format forward reference 
notation.)

I will stop here for now, as it is 1:30 am for me.

Terry

> On Mon, Jun 25, 2018 at 12:40 PM Terry Reedy <tjreedy at udel.edu 
> <mailto:tjreedy at udel.edu>> wrote:
> 
>     On 6/24/2018 7:25 PM, Guido van Rossum wrote:
>      > I'd wager that the people who might be most horrified about it
> 
>     the (b) scoping rule change
> 
>      > would be people who feel strongly that the change to the
>      > comprehension scope rules in Python 3 is a big improvement,
> 
>     I might not be one of those 'most horrified' by (b), but I increasingly
>     don't like it, and I was at best -0 on the comprehension scope change.
>     To me, iteration variable assignment in the current scope is a
>     non-problem.  So to me the change was mostly useless churn.  Little
>     benefit, little harm.  And not worth fighting when others saw a benefit.
> 
> 
> Fair enough, and by itself this might not have been enough reason to 
> make the change. But see below.
> 
>     However, having made the change to nested scopes, I think we should
>     stick with them.  Or repeal them.  (I believe there is another way to
>     isolate iteration names -- see  below).  To me, (b) amounts to half
>     repealing the nested scope change, making comprehensions half-fowl,
>     half-fish chimeras.
> 
> 
> That depends on how you see it -- to me (b) just means that there's an 
> implicit nonlocal[1] to make the assignment have the (desirable) 
> side-effect.
> 
> The key thing to consider here is whether that side-effect is in fact 
> desirable. For me, the side-effect of the comprehension's loop control 
> variable was never desirable -- it was just an implementation detail 
> leaking out. (And that's different from leaking a regular for-loop's 
> control variable -- since we have 'break' (and 'else') there are some 
> legitimate use cases. But comprehensions try to be expressions, and here 
> the side effect is at best useless and at worst a nasty surprise.)
> 
>      > and who are familiar with the difference in implementation
>      > of comprehensions (though not generator expressions) in Python 2
>     vs. 3.
> 
>     That I pretty much am, I think.  In Python 2, comprehensions (the fish)
>     were, at least in effect, expanded in-line to a normal for loop.
>     Generator expressions (the fowls) were different.  They were, and still
>     are, expanded into a temporary generator function whose return value is
>     dropped back into the original namespace.  Python 3 turned
>     comprehensions (with 2 news varieties thereof) into fowls also,
>     temporary functions whose return value is dropped back in the original
>     namespace.  The result is that a list comprehension is equivalent to
>     list(generator_ expression), even though, for efficiency, it is not
>     implemented that way.  (To me, this unification is more a benefit than
>     name hiding.)
> 
> 
> Right, and this consistency convinced me that the change was worth it. I 
> just really like to be able to say "[... for ...]" is equivalent to 
> "list(... for ...)", and similar for set and dict.
> 
>     (b) proposes to add extra hidden code in and around the temporary
>     function to partly undo the isolation.
> 
> 
> But it just adds a nonlocal declaration. There's always some hidden code 
> ('def' and 'return' at the very least).
> 
>     list comprehensions would no
>     longer be equivalent to list(generator_expression), unless
>     generator_expressions got the same treatment, in which case they would
>     no longer be equivalent to calling the obvious generator function.
>     Breaking either equivalence might break someone's code.
> 
> 
> Ah, there's the rub! I should probably apologize for not clarifying my 
> terminology more. In the context of PEP 572, when I say "comprehensions" 
> I include generators! PEP 572 states this explicitly 
> (https://github.com/python/peps/blame/master/pep-0572.rst#L201-L202).
> 
> Certainly PEP 572 intends to add that implicit nonlocal to both 
> comprehensions and generator expressions. (I just got really tired of 
> writing that phrase over and over, and at some point I forgot that this 
> is only a parenthetical remark added in the PEP's latest revision, and 
> not conventional terminology -- alas. :-)
> 
> Part (b) of PEP 572 does several things of things to *retain* consistency:
> 
> - The target of := lives in the same scope regardless of whether it 
> occurs in a comprehension, a generator expression, or just in some other 
> expression.
> 
> - When it occurs in a comprehension or generator expression, the scope 
> is the same regardless of whether it occurs in the "outermost iterable" 
> or not.
> 
> If we didn't have (b) the target would live in the comprehension/genexpr 
> scope if it occurred in a comprehension/genexp but outside its 
> "outermost iterable", and in the surrounding scope otherwise.
> 
>     ---
> 
>     How loop variables might be isolated without a nested scope: After a
>     comprehension is parsed, so that names become strings, rename the loop
>     variables to something otherwise illegal.  For instance, i could become
>     '<i>', just as lambda becomes '<lambda>' as the name of the resulting
>     function.  Expand the comprehension as in Python 2, except for deleting
>     the loop names along with the temporary result name.
> 
>     Assignment expressions within a comprehension would become assignment
>     expressions within the for loop expansion and would automatically
>     add or
>     replace values in the namespace containing the comprehension.  In other
>     words, I am suggesting that if we want name expressions in
>     comprehensions to act as they would in Python 2, then we should
>     consider
>     reverting to an altered version of the Python 2 expansion.
> 
> 
> Possibly this is based on a misunderstanding of my use of 
> "comprehensions". Also, since your trick can only be used for 
> list/set/dict comprehensions, but not for generator expressions (at 
> least I assume you don't want it there) it would actually *reduce* 
> consistency between list/set/dict comprehensions and generator expressions.
> 
>     ---
> 
>     In any case, I think (b) should be a separate PEP linked to a PEP for
>     (a).  The decision for (a) could be reject (making (b) moot), accept
>     with (b), or accept unconditionally (but still consider (b)).
> 
> 
> For me personally, (b) makes the PEP more consistent, so I'm not in 
> favor of breaking up the PEP. But we can certainly break up the 
> discussion -- that's why I started using the labels (a) and (b).
> ----------
> [1] Sometimes it's an implicit global instead of an implicit nonlocal -- 
> when there's already a global for the same variable in the target scope.
> 
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> 
> 


-- 
Terry Jan Reedy