[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Steven D'Aprano]

On Sat, Jun 23, 2018 at 12:22:33AM +1000, Nick Coghlan wrote:

[...]
> * for the reactions to my description of the currently proposed parent
> local scoping behaviour in comprehensions, I'd use the word
> "horrified", and feel I wasn't overstating the response :)

Without knowing how you worded the question, and the reasons for this 
horrified reaction, I'm afraid that isn't really helpful. It is nothing 
more than an appeal to emotion:

https://en.wikipedia.org/wiki/Wisdom_of_repugnance

Such strong emotions as "horrified" are typically a sign of an 
immediate, emotional gut reaction, not careful thought. We often see 
those sorts of reactions attached to the most objectively trivial 
matters. Immediate gut reactions are rarely a good guide because they 
tend to over-value the status quo, exaggerate the difficulty and costs 
of change, and under-estimate the benefits.

Speaking personally, I've learned to question my immediately gut 
reaction. (And I remember to do so at least half the time.) PEP 572 is 
an example: when the issue was first raised back in February, my gut 
reaction was "Not in MY Python!!!" but by taking it seriously and 
running through some examples over the course of the discussion, I 
realised that, actually, I cautiously favour the idea.

Of course, matters of *personal taste* cannot be anything but gut 
reaction, but in those matters, what one person holds strongly another 
can legitimately reject strongly. We ought to try to look beyond 
personal taste, and try (even if only imperfectly) to consider rational 
reasons for and against a proposal. If we do, reactions like "horrified" 
are rarely justified. It's just a minor feature in a programming 
language, the world will go on one way or the other, and Python already 
has trickier gotchas.


> While I try to account for the fact that I implemented the current
> comprehension semantics for the 3.x series, and am hence biased
> towards considering them the now obvious interpretation,

While we certainly don't want to make "non-obvious" a virtue for its own 
sake, obviousness (obvious to who?) ought to take a distant second place 
to *useful*. Otherwise we'd have to give up an awful lot of existing 
Python, starting with the fundamental execution model.

(Oh, the number and length of arguments about whether Python uses call 
by value or call by reference, why mutable defaults and [[]]*3 are 
"broken"... if you think Python's execution model is "obvious" you've 
been using Python too long ;-)

But as Tim Peters has said on a number of occasions, nobody is 
suggesting changing the interpretation of current comprehension 
semantics. Comprehension loop variables will continue to remain 
isolated to the comprehension.

(And for the record, that makes *comprehensions* a weird special case, 
not assignment expressions. All other expressions run in the current 
lexical scope. Comprehensions introduce an implicit, invisible, 
sub-local scope that doesn't match up with a change in indentation as 
class and def statements do.)

The behaviour in question is a matter of *assignment expression* 
semantics, not comprehensions. And honestly, I don't see why the 
proposed behaviour is "horrifying". Here's the high-level overview:

- at the top level of a module, assignment expressions assign in
  the global scope;

- inside a class, assignment expressions assign in the class scope;

- inside a function, assignment expressions assign in the function
  local scope (unless declared global or nonlocal);

- inside a comprehension, assignment expressions assign in the 
  surrounding lexical scope (the surrounding function, class or
  module).


The first three are the same as ordinary statement assignment. The last 
one is what you would expect if you treat comprehensions as any other 
expression which run in the current lexical scope. (The current function 
or class or module.) Even if we treat it as a "weird special case" (I 
don't think it is, but for the sake of the argument let's say it is) its 
not hard to explain.

As I discuss below, you can get a very long way indeed working with 
comprehensions without once thinking about the scope they run in. By the 
time you need to think about comprehension scope, it shouldn't be hard 
to deal with the rule:

- loop variables are hidden in a comprehension private scope;
- explicit assignment expression variables are not.

This is not async, or metaclasses, or even Unicode.


[...]
> plenty of
> functional-language-inspired documentation to instead encourage folks
> to view comprehensions as tightly encapsulated declarative container
> construction syntax.

I can't say I've done a broad survey, but the third-party documentation 
I've read on comprehensions typically glosses over the scoping issues 
without mentioning them. To the extent that scoping is even hinted at, 
comprehensions are treated as expressions which are exactly equivalent 
to re-writing them as a for-loop in the current scope.

This is a typical example, found as the top result on googling for 
"python comprehensions":

https://www.google.com/search?q=python+comprehensions

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python

Nothing is mentioned about scope, and it repeats the inaccurate but 
simple equivalency:

for item in list:
    if conditional:
        expression

But perhaps that tutorial is too old. Okay this recent one is only a 
little more than a year old:

https://hackernoon.com/list-comprehension-in-python-8895a785550b

Again, no mention of scoping issues, comprehensions are simply 
expressions which presumably run in the same scope as any other 
expression.

I think you over-estimate how many newcomers to Python are even aware 
that the scope of comprehensions is something to consider.


> I'm currently working on a concept proposal at
> https://github.com/ncoghlan/peps/pull/2 that's much closer to PEP 572
> than any of my previous `given` based suggestions:
[...]

I look forward to reading it, and I promise I won't go by my gut 
reaction :-)



-- 
Steve