[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Steven D'Aprano]

On Thu, Jun 28, 2018 at 03:42:49PM -0700, Chris Barker via Python-Dev wrote:

> If we think hardly anyone is ever going to do that -- then I guess it
> doesn't matter how it's handled.

That's how you get a language with surprising special cases, gotchas and 
landmines in its behaviour. (Cough PHP cough.)

It is one thing when gotchas occur because nobody thought of them, or 
because there is nothing you can do about them. But I do not think it is 
a good idea to *intentionally* leave gotchas lying around because "oh, I 
didn't think anyone would ever do that...".

*wink*


[...]
> but here the keyword "nonlocal" is used -- you are clearly declaring that
> you are messing with a nonlocal name here -- that is a lot more obvious
> than simply using a :=

But from the point of view of somebody reading the code, there is no 
need for a nonlocal declaration, since the assignment is just a local 
assignment.

Forget the loop variable -- it is a special case where "practicality 
beats purity" and it makes sense to have it run in a sublocal scope. 
Everything else is just a local, regardless of whether it is in a 
comprehension or not.


> And "nonlocal" is not used that often, and when it is it's for careful
> closure trickery -- I'm guessing := will be far more common. And, of
> course, when a newbie encounters it, they can google it and see what it
> means -- far different that seeing a := in a comprehension and
> understanding (by osmosis??) that it might make changes in the local scope.

I've given reasons why I believe that people will expect assignments in 
comprehensions to occur in the local scope. Aside from the special case 
of loop variables, people don't think of comprehensions as a separate 
scope.

There's no Comprehension Sublocal-Local-Enclosing Local-Global-Builtin 
scoping rule. (Do you want there to be?) Even *class scope* comes as an 
unfamiliar surprise to people.

I do not believe that people will "intuitively" expect assignments in a 
comprehension to disappear when the comprehension finishes -- I expect 
that most of the time they won't even think about it, but when they do, 
they'll expect it to hang around like *every other use* of assignment 
expressions.


> And  I don't think you can even do that with generator expressions now --
> as they can only contain expressions.

It makes me cry to think of the hours I spent -- and the brownie points 
I lost with my wife -- showing how you can already simulate this with 
locals() or globals(). Did nobody read it? :-(

https://mail.python.org/pipermail/python-dev/2018-June/154114.html

Yes, you can do this *right now*. We just don't because playing around 
with locals() is a dodgy thing to do.


> Maybe it's only comprehensions, and maybe it'll be rare to have a confusing
> version of those, so it'll be no big deal, but this thread started talking
> about educators' take on this -- and as an educator, I think this really
> does complicate the language.

See my recent post here:

https://mail.python.org/pipermail/python-dev/2018-June/154184.html

I strongly believe that the "comprehensions are local, like everything 
else" scenario is simpler and less surprising and easier to explain than 
hiding assignments inside a sublocal comprehension scope that hardly 
anyone even knows exists.

Especially if we end up doing it inconsistently and let variables 
sometimes leak.


> Python got much of it's "fame" by being "executable pseudo code" -- its
> been moving farther and farther away from those roots. That's generally a
> good thing, as we've gain expressiveness in exchangel, but we shouldn't
> pretend it isn't happening, or that this proposal doesn't contribute to
> that trend.

I think there are two distinct forms of "complication" here.

1. Are assignment expressions in isolation complicated?

2. Given assignment expressions, can people write obfuscated,
   complex code?


Of course the answer to Q2 is yes, the opportunity will be there. 
Despite my general cynicism about my fellow programmers, I actually do 
believe that the Python community does a brilliant job of self-policing 
to prevent the worst excesses of obfuscatory one-liners. I don't think 
that will change.

So I think Q1 is the critical one. And I think the answer is, no, 
they're conceptually bloody simple. They evaluate the expression on the 
right, assign it to the name on the left, and return that value.

Here is a question and answer:

    Question: after ``result = (x := 2) + 3``, what is the value of x?

    Answer: 2.

    Question: what if we put the assignment inside a function call? 
    ``f((x:=2), x+3)``

    Answer: still 2.

    Question: how about inside a list display? ``[1, x:=2, 3]``

    Answer: still 2.

    Question: what about a dict display? ``{key: x:=2}`` A tuple? A set?

    Answer: still 2 to all of them.

    Question: how about a list comprehension?

    Answer: ah, now, that's complicated, it depends on which bit of the 
    comprehension you put it in, the answer could be that x is 2 as you 
    would expect, or it could be undefined.


Yes, I can see why as an educator you would prefer that over *my* 
answer, which would be:

    Answer: it's still ^&*@^!$ two, what did you think it would be???

*wink*



> > > Maybe it’s just me, but re-binding a name seems like a whole new
> > > category of side effect.
> >
> > With no trickery at all, you've always been able to rebind attributes, and
> > mutate containers, in comprehensions and genexps.  Because `for` targets
> > aren't limited to plain names; e.g.,
> >
> >     g = (x+y for object.attribute, a[i][j] in zip(range(3), range(3)))
> >
> 
> sure, but you are explicitly using the names "object" and "a" here -- so
> while side effects in comprehension are discouraged, it's not really a
> surprised that namespaces specifically named are changed.

Trust me on this Chris, assuming the arguments over this PEP are 
finished by then (I give 50:50 odds *wink*) by the time 3.8 comes out 
you'll be saying 

    "sure, but you are explicitly assigning to a local variable
    with the := operator, it's not really a surprise that the
    local variable specifically named is being changed, it would
    be weird if it wasn't..."

and you'll have forgotten that there was ever a time we seriously 
discussed comprehension-scope as a thing.

:-)



[example with the loop variable of a comprehension]
> doesn't change x in the local scope -- if that was a good idea, why is a
> good idea to have := in a comprehension effect the local scope??
> 
> But maybe it is just me.

Some of us don't think that it was a good idea *wink*

But I know when I'm outvoted and when "practicality beats purity".

Loop variables are semantically special. They tend to have short, often 
one-character names, taken from a small set of common examples (i, j, x 
are especially common). We don't tend to think of them as quite the same 
as ordinary variables: once the loop is complete, we typically ignore 
the loop variable.

It takes a conscious effort to remember that it actually still hangs 
around. I've written code like this:

    for x in sequence:
        last_x = x
        do_stuff()
    # outside the loop
    process(last_x)

and then looked at it a day later and figuratively kicked myself. What 
the hell was I thinking?

So it is hardly surprising that people would sometimes write:

    for x in sequence:
        L = [expression for x in something]
        process(x)

and be unpleasantly surprised in Python 2.

But this is not likely to happen BY ACCIDENT with assignment 
expressions, and if it does, well, the answer is, change the variable 
name to something a little less generic, or at least *different*:

    for x in sequence:
        L = [x := a+1 for a in something]
        process(x)

I am reluctantly forced to agree that, purity be damned, burying the 
loop variable inside a hidden implicit function scope is the right thing 
to do, even if it is sometimes annoying. But the reasons for doing do 
don't apply to explicit assignment expressions.

We have no obligation to protect people from every accidental name 
clobbering caused by carelessness and poor choice of names.


py> import math as module, string as module, sys as module
py> module
<module 'sys' (built-in)>


>  I suppose we need to go back and look at the "real" examples of where/how
> folks think they'll use := in comprehensions, and see how confusing it may
> be.

One the simplest but most useful examples is as a debugging aide.

    [function(x) for x in whatever]

crashes part of the way through and raises an exception. How to debug? 
If it were a for-loop, one simple solution would be to wrap it in a 
try...except and then print the last seen value of x.

    try:
        for x in whatever:
            function(x)
    except:
        print(x)

Can't do that with a list comprehension, not in Python 3. (It works in 
Python 2.) Assignment expressions to the rescue!

    try:
        [function(a := x) for x in whatever]
     except:
        print(a)


That's simple and lightweight enough that with a bit of polishing, you 
could even leave it in production code to log a hard-to-track down 
error.

Oh, and it doesn't easily translate to a map(), since lambda functions 
do execute in their own scope:

    map(lambda x: function(a := x), whatever)

Another reason to prefer comprehensions to map.



> One of these conversations was started with an example something like this:
> 
> [(f(x), g(f(x))) for x in an_iterable]
> 
> The OP didn't like having to call f() twice. So that would become:
> 
> [ (temp:=f(x), g(temp)) for x in an_iterable]
> 
> so now the question is: should "temp" be created / changed in the enclosing
> local scope?

The comprehension here is a red herring.

    result = (temp := f(x), g(temp))

Should temp be a local, or should tuple displays exist in their own, 
special, sublocal scope?

In either case, there's no *specific* advantage to letting temp "leak" 
(scare quotes) out of the expression. But there's no disadvantage 
either, and if we pick a less prejudicial name, it might actually be 
advantagous:

    result = (useful_data := f(x), g(useful_data))
    process(useful_data)

Using a comprehension isn't special enough to change the rule that 
assignment expressions are local (unless declared global).


-- 
Steve