[Python-Dev] PEP 572: Assignment Expressions

[Steven D'Aprano]

On Sun, Apr 22, 2018 at 12:45:36AM +1000, Chris Angelico wrote:

> > The reason I want items to "leak" into the surrounding scope is mostly
> > so that the initial value for it can be set with a simple assignment
> > outside the comprehension:
> >
> >     items = (1, 2, 3)
> >     [ ... items := items*2 ... ]
> >
> > and the least magical way to do that is to just make items an ordinary
> > local variable.
> 
> You can't have your cake and eat it too. Iteration variables and names
> bound by assignment expressions are both set inside the comprehension.

You say that as if it were a law of physics, rather than an 
implementation choice.


> Either they both are local, or they both leak - or else we have a
> weird rule like "the outermost iterable is magical and special".

We *already* have the rule that the outermost iterable is special, 
except it isn't a rule precisely, since (as far as I know) it isn't 
documented anywhere, nor was it ever planned as a feature. It's just an 
accidental(?) consequence of the implementation choices made.


py> spam = [(1,2), (3, 4)]
py> [spam for x in spam for spam in x]  # first loop is magic
[1, 2, 3, 4]


but:


py> spam = [(1,2), (3, 4)]
py> [spam for _ in [1] for x in spam for spam in x]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
UnboundLocalError: local variable 'spam' referenced before assignment


However the second example worked fine in Python 2. Changing the 
implementation of comprehensions to be like generator expressions and 
avoid leaking the loop variables seems to have had the (accidental?) 
side-effect of making the first loop magical.


[...]
> PEP 572 corrects this by making it behave the way that you, and many
> other people, expect. Current behaviour is surprising because the
> outermost iterable is special and magical.

This shouldn't be PEP 572's job.

It's unfair on you to be shouldered with sheparding through what is
effectively two complex PEPs ("assignment-expressions" plus "fix 
comprehension scoping") in one. Especially if you had no idea at the 
start that this is what is involved.

And it's even more unfair on those who may not care two hoots about 
assignment-expressions, but would be really interested in comprehension 
scoping if only they knew we were talking about that.

And it makes this PEP harder work for readers who don't care about 
comprehension scoping.

I think that we should be able to make any of the following choices (or 
no choice at all) regarding comprehensions:

* no change: semantics remains underspecified, defined by 
  "whatever the implementation does";
* lock in the current behaviour as a language promise;
* change the behaviour and make it a language promise;

regardless of what is decided about PEP 572.


[...]
> > Are you sure about this example?
> 
> Yes, I'm sure. You may notice that I didn't iterate over the genexps
> in my example.

No, I didn't notice that was your intent. I thought it was just 
short-hand.


> The first one will bomb out, even without iteration;

And that's yet another oddity, one I didn't think of. It's downright 
bizarre that these two genexps behave differently:

spam = [1, 2]
eggs = 12
(x+y for x in spam for y in eggs)  # okay
(x+y for y in eggs for x in spam)  # TypeError

and I'd be surprised to learn that this behavour was planned in advance. 
("Early binding and ahead-of-time type-testing for the first loop, late 
binding and just-in-time type-testing for the second loop. All in 
favour?")

But it is what it is, and who knows, maybe we decide we *want* this 
behaviour, bizarre as it is. It isn't clear to me that:


1. it's necessarily "broken" and needs fixing;

2. if if does need fixing, it needs to be fixed *right* now;

3. that acceptance or rejection of PEP 572 needs to hinge on the 
   decision about comprehensions;

4. and especially that a change to comprehensions ought to be
   smuggled in via an unrelated PEP.

(I know that 4 is not your intention, but that's the way it may appear.)


-- 
Steve