<div dir="ltr">With the new name restriction on the LHS, I'm now -0 on this. While I don't think the benefits outweigh the overhead cost of pushing Python closer to not fitting in my brain, I would admittedly use this if provided to me. (I also put this in the bucket of consenting adult features; ripe for abuse, but common sense should prevail much like with every other feature we have added that could get over-used, e.g. decorators.)<br><div><br><div class="gmail_quote"><div dir="ltr">On Tue, 24 Apr 2018 at 08:36 Chris Angelico <<a href="mailto:rosuav@gmail.com">rosuav@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The most notable change since last posting is that the assignment<br>
target is no longer as flexible as with the statement form of<br>
assignment, but is restricted to a simple name.<br>
<br>
Note that the reference implementation has not been updated.<br>
<br>
ChrisA<br>
<br>
<br>
PEP: 572<br>
Title: Assignment Expressions<br>
Author: Chris Angelico <<a href="mailto:rosuav@gmail.com" target="_blank">rosuav@gmail.com</a>><br>
Status: Draft<br>
Type: Standards Track<br>
Content-Type: text/x-rst<br>
Created: 28-Feb-2018<br>
Python-Version: 3.8<br>
Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018, 17-Apr-2018,<br>
25-Apr-2018<br>
<br>
<br>
Abstract<br>
========<br>
<br>
This is a proposal for creating a way to assign to variables within an<br>
expression. Additionally, the precise scope of comprehensions is adjusted, to<br>
maintain consistency and follow expectations.<br>
<br>
<br>
Rationale<br>
=========<br>
<br>
Naming the result of an expression is an important part of programming,<br>
allowing a descriptive name to be used in place of a longer expression,<br>
and permitting reuse. Currently, this feature is available only in<br>
statement form, making it unavailable in list comprehensions and other<br>
expression contexts. Merely introducing a way to assign as an expression<br>
would create bizarre edge cases around comprehensions, though, and to avoid<br>
the worst of the confusions, we change the definition of comprehensions,<br>
causing some edge cases to be interpreted differently, but maintaining the<br>
existing behaviour in the majority of situations.<br>
<br>
<br>
Syntax and semantics<br>
====================<br>
<br>
In any context where arbitrary Python expressions can be used, a **named<br>
expression** can appear. This is of the form ``name := expr`` where<br>
``expr`` is any valid Python expression, and ``name`` is an identifier.<br>
<br>
The value of such a named expression is the same as the incorporated<br>
expression, with the additional side-effect that the target is assigned<br>
that value::<br>
<br>
# Handle a matched regex<br>
if (match := pattern.search(data)) is not None:<br>
...<br>
<br>
# A more explicit alternative to the 2-arg form of iter() invocation<br>
while (value := read_next_item()) is not None:<br>
...<br>
<br>
# Share a subexpression between a comprehension filter clause and its output<br>
filtered_data = [y for x in data if (y := f(x)) is not None]<br>
<br>
<br>
Differences from regular assignment statements<br>
----------------------------------------------<br>
<br>
Most importantly, since ``:=`` is an expression, it can be used in contexts<br>
where statements are illegal, including lambda functions and comprehensions.<br>
<br>
An assignment statement can assign to multiple targets, left-to-right::<br>
<br>
x = y = z = 0<br>
<br>
The equivalent assignment expression is parsed as separate binary operators,<br>
and is therefore processed right-to-left, as if it were spelled thus::<br>
<br>
assert 0 == (x := (y := (z := 0)))<br>
<br>
Statement assignment can include annotations. This would be syntactically<br>
noisy in expressions, and is of minor importance. An annotation can be<br>
given separately from the assignment if needed::<br>
<br>
x:str = "" # works<br>
(x:str := "") # SyntaxError<br>
x:str # possibly before a loop<br>
(x := "") # fine<br>
<br>
Augmented assignment is not supported in expression form::<br>
<br>
>>> x +:= 1<br>
File "<stdin>", line 1<br>
x +:= 1<br>
^<br>
SyntaxError: invalid syntax<br>
<br>
Statement assignment is able to set attributes and subscripts, but<br>
expression assignment is restricted to names. (This restriction may be<br>
relaxed in a future version of Python.)<br>
<br>
Otherwise, the semantics of assignment are identical in statement and<br>
expression forms.<br>
<br>
<br>
Alterations to comprehensions<br>
-----------------------------<br>
<br>
The current behaviour of list/set/dict comprehensions and generator<br>
expressions has some edge cases that would behave strangely if an assignment<br>
expression were to be used. Therefore the proposed semantics are changed,<br>
removing the current edge cases, and instead altering their behaviour *only*<br>
in a class scope.<br>
<br>
As of Python 3.7, the outermost iterable of any comprehension is evaluated<br>
in the surrounding context, and then passed as an argument to the implicit<br>
function that evaluates the comprehension.<br>
<br>
Under this proposal, the entire body of the comprehension is evaluated in<br>
its implicit function. Names not assigned to within the comprehension are<br>
located in the surrounding scopes, as with normal lookups. As one special<br>
case, a comprehension at class scope will **eagerly bind** any name which<br>
is already defined in the class scope.<br>
<br>
A list comprehension can be unrolled into an equivalent function. With<br>
Python 3.7 semantics::<br>
<br>
numbers = [x + y for x in range(3) for y in range(4)]<br>
# Is approximately equivalent to<br>
def <listcomp>(iterator):<br>
result = []<br>
for x in iterator:<br>
for y in range(4):<br>
result.append(x + y)<br>
return result<br>
numbers = <listcomp>(iter(range(3)))<br>
<br>
Under the new semantics, this would instead be equivalent to::<br>
<br>
def <listcomp>():<br>
result = []<br>
for x in range(3):<br>
for y in range(4):<br>
result.append(x + y)<br>
return result<br>
numbers = <listcomp>()<br>
<br>
When a class scope is involved, a naive transformation into a function would<br>
prevent name lookups (as the function would behave like a method)::<br>
<br>
class X:<br>
names = ["Fred", "Barney", "Joe"]<br>
prefix = "> "<br>
prefixed_names = [prefix + name for name in names]<br>
<br>
With Python 3.7 semantics, this will evaluate the outermost iterable at class<br>
scope, which will succeed; but it will evaluate everything else in a function::<br>
<br>
class X:<br>
names = ["Fred", "Barney", "Joe"]<br>
prefix = "> "<br>
def <listcomp>(iterator):<br>
result = []<br>
for name in iterator:<br>
result.append(prefix + name)<br>
return result<br>
prefixed_names = <listcomp>(iter(names))<br>
<br>
The name ``prefix`` is thus searched for at global scope, ignoring the class<br>
name. Under the proposed semantics, this name will be eagerly bound; and the<br>
same early binding then handles the outermost iterable as well. The list<br>
comprehension is thus approximately equivalent to::<br>
<br>
class X:<br>
names = ["Fred", "Barney", "Joe"]<br>
prefix = "> "<br>
def <listcomp>(names=names, prefix=prefix):<br>
result = []<br>
for name in names:<br>
result.append(prefix + name)<br>
return result<br>
prefixed_names = <listcomp>()<br>
<br>
With list comprehensions, this is unlikely to cause any confusion. With<br>
generator expressions, this has the potential to affect behaviour, as the<br>
eager binding means that the name could be rebound between the creation of<br>
the genexp and the first call to ``next()``. It is, however, more closely<br>
aligned to normal expectations. The effect is ONLY seen with names that<br>
are looked up from class scope; global names (eg ``range()``) will still<br>
be late-bound as usual.<br>
<br>
One consequence of this change is that certain bugs in genexps will not<br>
be detected until the first call to ``next()``, where today they would be<br>
caught upon creation of the generator.<br>
<br>
<br>
Recommended use-cases<br>
=====================<br>
<br>
Simplifying list comprehensions<br>
-------------------------------<br>
<br>
A list comprehension can map and filter efficiently by capturing<br>
the condition::<br>
<br>
results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0]<br>
<br>
Similarly, a subexpression can be reused within the main expression, by<br>
giving it a name on first use::<br>
<br>
stuff = [[y := f(x), x/y] for x in range(5)]<br>
<br>
# There are a number of less obvious ways to spell this in current<br>
# versions of Python, such as:<br>
<br>
# Inline helper function<br>
stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)]<br>
<br>
# Extra 'for' loop - potentially could be optimized internally<br>
stuff = [[y, x/y] for x in range(5) for y in [f(x)]]<br>
<br>
# Using a mutable cache object (various forms possible)<br>
c = {}<br>
stuff = [[c.update(y=f(x)) or c['y'], x/c['y']] for x in range(5)]<br>
<br>
In all cases, the name is local to the comprehension; like iteration variables,<br>
it cannot leak out into the surrounding context.<br>
<br>
<br>
Capturing condition values<br>
--------------------------<br>
<br>
Assignment expressions can be used to good effect in the header of<br>
an ``if`` or ``while`` statement::<br>
<br>
# Proposed syntax<br>
while (command := input("> ")) != "quit":<br>
print("You entered:", command)<br>
<br>
# Capturing regular expression match objects<br>
# See, for instance, Lib/pydoc.py, which uses a multiline spelling<br>
# of this effect<br>
if match := re.search(pat, text):<br>
print("Found:", match.group(0))<br>
<br>
# Reading socket data until an empty string is returned<br>
while data := sock.read():<br>
print("Received data:", data)<br>
<br>
# Equivalent in current Python, not caring about function return value<br>
while input("> ") != "quit":<br>
print("You entered a command.")<br>
<br>
# To capture the return value in current Python demands a four-line<br>
# loop header.<br>
while True:<br>
command = input("> ");<br>
if command == "quit":<br>
break<br>
print("You entered:", command)<br>
<br>
Particularly with the ``while`` loop, this can remove the need to have an<br>
infinite loop, an assignment, and a condition. It also creates a smooth<br>
parallel between a loop which simply uses a function call as its condition,<br>
and one which uses that as its condition but also uses the actual value.<br>
<br>
<br>
Rejected alternative proposals<br>
==============================<br>
<br>
Proposals broadly similar to this one have come up frequently on python-ideas.<br>
Below are a number of alternative syntaxes, some of them specific to<br>
comprehensions, which have been rejected in favour of the one given above.<br>
<br>
<br>
Alternative spellings<br>
---------------------<br>
<br>
Broadly the same semantics as the current proposal, but spelled differently.<br>
<br>
1. ``EXPR as NAME``::<br>
<br>
stuff = [[f(x) as y, x/y] for x in range(5)]<br>
<br>
Since ``EXPR as NAME`` already has meaning in ``except`` and ``with``<br>
statements (with different semantics), this would create unnecessary<br>
confusion or require special-casing (eg to forbid assignment within the<br>
headers of these statements).<br>
<br>
2. ``EXPR -> NAME``::<br>
<br>
stuff = [[f(x) -> y, x/y] for x in range(5)]<br>
<br>
This syntax is inspired by languages such as R and Haskell, and some<br>
programmable calculators. (Note that a left-facing arrow ``y <- f(x)`` is<br>
not possible in Python, as it would be interpreted as less-than and unary<br>
minus.) This syntax has a slight advantage over 'as' in that it does not<br>
conflict with ``with`` and ``except`` statements, but otherwise is<br>
equivalent.<br>
<br>
3. Adorning statement-local names with a leading dot::<br>
<br>
stuff = [[(f(x) as .y), x/.y] for x in range(5)] # with "as"<br>
stuff = [[(.y := f(x)), x/.y] for x in range(5)] # with ":="<br>
<br>
This has the advantage that leaked usage can be readily detected, removing<br>
some forms of syntactic ambiguity. However, this would be the only place<br>
in Python where a variable's scope is encoded into its name, making<br>
refactoring harder.<br>
<br>
4. Adding a ``where:`` to any statement to create local name bindings::<br>
<br>
value = x**2 + 2*x where:<br>
x = spam(1, 4, 7, q)<br>
<br>
Execution order is inverted (the indented body is performed first, followed<br>
by the "header"). This requires a new keyword, unless an existing keyword<br>
is repurposed (most likely ``with:``). See PEP 3150 for prior discussion<br>
on this subject (with the proposed keyword being ``given:``).<br>
<br>
5. ``TARGET from EXPR``::<br>
<br>
stuff = [[y from f(x), x/y] for x in range(5)]<br>
<br>
This syntax has fewer conflicts than ``as`` does (conflicting only with the<br>
``raise Exc from Exc`` notation), but is otherwise comparable to it. Instead<br>
of paralleling ``with expr as target:`` (which can be useful but can also be<br>
confusing), this has no parallels, but is evocative.<br>
<br>
<br>
Special-casing conditional statements<br>
-------------------------------------<br>
<br>
One of the most popular use-cases is ``if`` and ``while`` statements. Instead<br>
of a more general solution, this proposal enhances the syntax of these two<br>
statements to add a means of capturing the compared value::<br>
<br>
if re.search(pat, text) as match:<br>
print("Found:", match.group(0))<br>
<br>
This works beautifully if and ONLY if the desired condition is based on the<br>
truthiness of the captured value. It is thus effective for specific<br>
use-cases (regex matches, socket reads that return `''` when done), and<br>
completely useless in more complicated cases (eg where the condition is<br>
``f(x) < 0`` and you want to capture the value of ``f(x)``). It also has<br>
no benefit to list comprehensions.<br>
<br>
Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction<br>
of possible use-cases, even in ``if``/``while`` statements.<br>
<br>
<br>
Special-casing comprehensions<br>
-----------------------------<br>
<br>
Another common use-case is comprehensions (list/set/dict, and genexps). As<br>
above, proposals have been made for comprehension-specific solutions.<br>
<br>
1. ``where``, ``let``, or ``given``::<br>
<br>
stuff = [(y, x/y) where y = f(x) for x in range(5)]<br>
stuff = [(y, x/y) let y = f(x) for x in range(5)]<br>
stuff = [(y, x/y) given y = f(x) for x in range(5)]<br>
<br>
This brings the subexpression to a location in between the 'for' loop and<br>
the expression. It introduces an additional language keyword, which creates<br>
conflicts. Of the three, ``where`` reads the most cleanly, but also has the<br>
greatest potential for conflict (eg SQLAlchemy and numpy have ``where``<br>
methods, as does ``tkinter.dnd.Icon`` in the standard library).<br>
<br>
2. ``with NAME = EXPR``::<br>
<br>
stuff = [(y, x/y) with y = f(x) for x in range(5)]<br>
<br>
As above, but reusing the `with` keyword. Doesn't read too badly, and needs<br>
no additional language keyword. Is restricted to comprehensions, though,<br>
and cannot as easily be transformed into "longhand" for-loop syntax. Has<br>
the C problem that an equals sign in an expression can now create a name<br>
binding, rather than performing a comparison. Would raise the question of<br>
why "with NAME = EXPR:" cannot be used as a statement on its own.<br>
<br>
3. ``with EXPR as NAME``::<br>
<br>
stuff = [(y, x/y) with f(x) as y for x in range(5)]<br>
<br>
As per option 2, but using ``as`` rather than an equals sign. Aligns<br>
syntactically with other uses of ``as`` for name binding, but a simple<br>
transformation to for-loop longhand would create drastically different<br>
semantics; the meaning of ``with`` inside a comprehension would be<br>
completely different from the meaning as a stand-alone statement, while<br>
retaining identical syntax.<br>
<br>
Regardless of the spelling chosen, this introduces a stark difference between<br>
comprehensions and the equivalent unrolled long-hand form of the loop. It is<br>
no longer possible to unwrap the loop into statement form without reworking<br>
any name bindings. The only keyword that can be repurposed to this task is<br>
``with``, thus giving it sneakily different semantics in a comprehension than<br>
in a statement; alternatively, a new keyword is needed, with all the costs<br>
therein.<br>
<br>
<br>
Lowering operator precedence<br>
----------------------------<br>
<br>
There are two logical precedences for the ``:=`` operator. Either it should<br>
bind as loosely as possible, as does statement-assignment; or it should bind<br>
more tightly than comparison operators. Placing its precedence between the<br>
comparison and arithmetic operators (to be precise: just lower than bitwise<br>
OR) allows most uses inside ``while`` and ``if`` conditions to be spelled<br>
without parentheses, as it is most likely that you wish to capture the value<br>
of something, then perform a comparison on it::<br>
<br>
pos = -1<br>
while pos := buffer.find(search_term, pos + 1) >= 0:<br>
...<br>
<br>
Once find() returns -1, the loop terminates. If ``:=`` binds as loosely as<br>
``=`` does, this would capture the result of the comparison (generally either<br>
``True`` or ``False``), which is less useful.<br>
<br>
While this behaviour would be convenient in many situations, it is also harder<br>
to explain than "the := operator behaves just like the assignment statement",<br>
and as such, the precedence for ``:=`` has been made as close as possible to<br>
that of ``=``.<br>
<br>
<br>
Migration path<br>
==============<br>
<br>
The semantic changes to list/set/dict comprehensions, and more so to generator<br>
expressions, may potentially require migration of code. In many cases, the<br>
changes simply make legal what used to raise an exception, but there are some<br>
edge cases that were previously legal and now are not, and a few corner cases<br>
with altered semantics.<br>
<br>
<br>
The Outermost Iterable<br>
----------------------<br>
<br>
As of Python 3.7, the outermost iterable in a comprehension is special: it is<br>
evaluated in the surrounding context, instead of inside the comprehension.<br>
Thus it is permitted to contain a ``yield`` expression, to use a name also<br>
used elsewhere, and to reference names from class scope. Also, in a genexp,<br>
the outermost iterable is pre-evaluated, but the rest of the code is not<br>
touched until the genexp is first iterated over. Class scope is now handled<br>
more generally (see above), but if other changes require the old behaviour,<br>
the iterable must be explicitly elevated from the comprehension::<br>
<br>
# Python 3.7<br>
def f(x):<br>
return [x for x in x if x]<br>
def g():<br>
return [x for x in [(yield 1)]]<br>
# With PEP 572<br>
def f(x):<br>
return [y for y in x if y]<br>
def g():<br>
sent_item = (yield 1)<br>
return [x for x in [sent_item]]<br>
<br>
This more clearly shows that it is g(), not the comprehension, which is able<br>
to yield values (and is thus a generator function). The entire comprehension<br>
is consistently in a single scope.<br>
<br>
The following expressions would, in Python 3.7, raise exceptions immediately.<br>
With the removal of the outermost iterable's special casing, they are now<br>
equivalent to the most obvious longhand form::<br>
<br>
gen = (x for x in rage(10)) # NameError<br>
gen = (x for x in 10) # TypeError (not iterable)<br>
gen = (x for x in range(1/0)) # ZeroDivisionError<br>
<br>
def <genexp>():<br>
for x in rage(10):<br>
yield x<br>
gen = <genexp>() # No exception yet<br>
tng = next(gen) # NameError<br>
<br>
<br>
Open questions<br>
==============<br>
<br>
Importing names into comprehensions<br>
-----------------------------------<br>
<br>
A list comprehension can use and update local names, and they will retain<br>
their values from one iteration to another. It would be convenient to use<br>
this feature to create rolling or self-effecting data streams::<br>
<br>
progressive_sums = [total := total + value for value in data]<br>
<br>
This will fail with UnboundLocalError due to ``total`` not being initalized.<br>
Simply initializing it outside of the comprehension is insufficient - unless<br>
the comprehension is in class scope::<br>
<br>
class X:<br>
total = 0<br>
progressive_sums = [total := total + value for value in data]<br>
<br>
At other scopes, it may be beneficial to have a way to fetch a value from the<br>
surrounding scope. Should this be automatic? Should it be controlled with a<br>
keyword? Hypothetically (and using no new keywords), this could be written::<br>
<br>
total = 0<br>
progressive_sums = [total := total + value<br>
import nonlocal total<br>
for value in data]<br>
<br>
Translated into longhand, this would become::<br>
<br>
total = 0<br>
def <listcomp>(total=total):<br>
result = []<br>
for value in data:<br>
result.append(total := total + value)<br>
return result<br>
progressive_sums = <listcomp>()<br>
<br>
ie utilizing the same early-binding technique that is used at class scope.<br>
<br>
<br>
Frequently Raised Objections<br>
============================<br>
<br>
Why not just turn existing assignment into an expression?<br>
---------------------------------------------------------<br>
<br>
C and its derivatives define the ``=`` operator as an expression, rather than<br>
a statement as is Python's way. This allows assignments in more contexts,<br>
including contexts where comparisons are more common. The syntactic similarity<br>
between ``if (x == y)`` and ``if (x = y)`` belies their drastically different<br>
semantics. Thus this proposal uses ``:=`` to clarify the distinction.<br>
<br>
<br>
This could be used to create ugly code!<br>
---------------------------------------<br>
<br>
So can anything else. This is a tool, and it is up to the programmer to use it<br>
where it makes sense, and not use it where superior constructs can be used.<br>
<br>
<br>
With assignment expressions, why bother with assignment statements?<br>
-------------------------------------------------------------------<br>
<br>
The two forms have different flexibilities. The ``:=`` operator can be used<br>
inside a larger expression; the ``=`` statement can be augmented to ``+=`` and<br>
its friends, can be chained, and can assign to attributes and subscripts.<br>
<br>
<br>
Why not use a sublocal scope and prevent namespace pollution?<br>
-------------------------------------------------------------<br>
<br>
Previous revisions of this proposal involved sublocal scope (restricted to a<br>
single statement), preventing name leakage and namespace pollution. While a<br>
definite advantage in a number of situations, this increases complexity in<br>
many others, and the costs are not justified by the benefits. In the interests<br>
of language simplicity, the name bindings created here are exactly equivalent<br>
to any other name bindings, including that usage at class or module scope will<br>
create externally-visible names. This is no different from ``for`` loops or<br>
other constructs, and can be solved the same way: ``del`` the name once it is<br>
no longer needed, or prefix it with an underscore.<br>
<br>
Names bound within a comprehension are local to that comprehension, even in<br>
the outermost iterable, and can thus be used freely without polluting the<br>
surrounding namespace.<br>
<br>
(The author wishes to thank Guido van Rossum and Christoph Groth for their<br>
suggestions to move the proposal in this direction. [2]_)<br>
<br>
<br>
Style guide recommendations<br>
===========================<br>
<br>
As expression assignments can sometimes be used equivalently to statement<br>
assignments, the question of which should be preferred will arise. For the<br>
benefit of style guides such as PEP 8, two recommendations are suggested.<br>
<br>
1. If either assignment statements or assignment expressions can be<br>
used, prefer statements; they are a clear declaration of intent.<br>
<br>
2. If using assignment expressions would lead to ambiguity about<br>
execution order, restructure it to use statements instead.<br>
<br>
<br>
Acknowledgements<br>
================<br>
<br>
The author wishes to thank Guido van Rossum and Nick Coghlan for their<br>
considerable contributions to this proposal, and to members of the<br>
core-mentorship mailing list for assistance with implementation.<br>
<br>
<br>
References<br>
==========<br>
<br>
.. [1] Proof of concept / reference implementation<br>
(<a href="https://github.com/Rosuav/cpython/tree/assignment-expressions" rel="noreferrer" target="_blank">https://github.com/Rosuav/cpython/tree/assignment-expressions</a>)<br>
.. [2] Pivotal post regarding inline assignment semantics<br>
(<a href="https://mail.python.org/pipermail/python-ideas/2018-March/049409.html" rel="noreferrer" target="_blank">https://mail.python.org/pipermail/python-ideas/2018-March/049409.html</a>)<br>
<br>
<br>
Copyright<br>
=========<br>
<br>
This document has been placed in the public domain.<br>
<br>
<br>
..<br>
Local Variables:<br>
mode: indented-text<br>
indent-tabs-mode: nil<br>
sentence-end-double-space: t<br>
fill-column: 70<br>
coding: utf-8<br>
End:<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/brett%40python.org" rel="noreferrer" target="_blank">https://mail.python.org/mailman/options/python-dev/brett%40python.org</a><br>
</blockquote></div></div></div>