[Python-Dev] The new and improved PEP 572, same great taste with 75% less complexity!

[Steven D'Aprano]

On Tue, Apr 24, 2018 at 11:55:16PM -0700, Nathaniel Smith wrote:

> These examples all make me very nervous. The order of execution in
> comprehensions is pretty confusing to start with (right to left,
> except when it's left to right!).

I don't think comprehensions are ever right to left. With one small 
irregularity, execution follows Python's normal order, namely (mostly) 
left to right except where precedence requires something different, the 
if operator etc.

The one small irregularity is that the values produced by 
the comprehension are written first, rather than last. That is, given:

    [expression for item in iterable ...]

the initial expression doesn't get evaluated until the loop is entered. 
But the expression still evaluates in left-to-write order (modulo 
operator precedence etc) and everything following the first "for" 
keywords also evaluates in left-to-right order.


[...]
> Concretely, I find it unnerving that two of these work, and one doesn't:
> 
> # assignment on the right of usage
> results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0]

This shouldn't be surprising. We place the expression first because 
that's the most important part of the comprehension, the values it 
generates. Everything from the "for" onwards is just scaffolding needed 
to produce those values, its the leading expression that matters.

But clearly we can't expect to evaluate the expression *before* the 
loop, even though we want it written first. So although the 
comprehension is written expression first, the expression is actually 
evaluated *last*.

    results = [ for x in input_data 
                    if (y := f(x)) > 0 
                        (x, y, x/y)
              ]


So when reading a comprehension in execution order:

- look ahead to the first "for" keyword;

- read to the end of the comprehension, using normal left-to-right
  execution order (modulo the usual exceptions);

- jump back to the expression at the start;

- read in normal left-to-right execution order.

So aside from that small anomaly of the expression coming first, 
comprehensions use the same order as regular Python code. Binding 
expressions won't change that.


> I could probably figure it out if necessary, but even in
> simple cases like f(g(), h()), then do you really know for certain off
> the top of your head whether g() or h() runs first?

Of course. Left-to-right execution order (modulo the usual...) is a 
language guarantee.

> Does the average user?

I dare say the average user probably doesn't even think about it, and 
merely assumes that everything is left to right until they learn 
differently. In this case, that reasonable default position is correct. 
Everything is left to right until you learn differently.


> With code like f(a := g(), h(a)) this suddenly matters a lot!

That's hardly different from any other precedence issue, and even when 
precedence is specified by the language, sometimes adding a few extra 
brackets makes things clearer even when they're not needed:

    func(arg, (spam*n == token), (a := g()), h(a))


> But comprehensions suffer from a particularly extreme version of this,
> so it worries me that they're being put forward as the first set of
> motivating examples.

Ha, I think that comprehensions are one of the weaker motivating 
examples, but it was a discussion on Python-Ideas about adding special 
syntax *only* to comprehensions which lead to Chris writing the PEP. So 
it is purely an accident of history why comprehensions are the first 
example in the PEP.


> >     # Capturing regular expression match objects
> >     # See, for instance, Lib/pydoc.py, which uses a multiline spelling
> >     # of this effect
> >     if match := re.search(pat, text):
> >         print("Found:", match.group(0))
> 
> Now this is a genuinely compelling example! re match objects are
> always awkward to work with. But this feels like a very big hammer to
> make re.match easier to use :-). I wonder if there's anything more
> focused we could do here?

A reasonable point.


[...]
> However, I do think it'd be kinda confusing if we had:
> 
> if EXPR as X:
> while EXPR as X:
> with EXPR as X:
> 
> and the first two assign the value of EXPR to X, while the last one
> does something more subtle. Or maybe it'd be fine?

It would be fine, right up to the point that it wasn't, and then it 
would be a brain melting bug magnet that would cause programmers to 
curse us onto the 20th generation.



-- 
Steve