[Python-Dev] The new and improved PEP 572, same great taste with 75% less complexity!
Brett Cannon
brett at python.org
Thu Apr 26 14:20:58 EDT 2018
With the new name restriction on the LHS, I'm now -0 on this. While I don't
think the benefits outweigh the overhead cost of pushing Python closer to
not fitting in my brain, I would admittedly use this if provided to me. (I
also put this in the bucket of consenting adult features; ripe for abuse,
but common sense should prevail much like with every other feature we have
added that could get over-used, e.g. decorators.)
On Tue, 24 Apr 2018 at 08:36 Chris Angelico <rosuav at gmail.com> wrote:
> The most notable change since last posting is that the assignment
> target is no longer as flexible as with the statement form of
> assignment, but is restricted to a simple name.
>
> Note that the reference implementation has not been updated.
>
> ChrisA
>
>
> PEP: 572
> Title: Assignment Expressions
> Author: Chris Angelico <rosuav at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 28-Feb-2018
> Python-Version: 3.8
> Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018,
> 17-Apr-2018,
> 25-Apr-2018
>
>
> Abstract
> ========
>
> This is a proposal for creating a way to assign to variables within an
> expression. Additionally, the precise scope of comprehensions is adjusted,
> to
> maintain consistency and follow expectations.
>
>
> Rationale
> =========
>
> Naming the result of an expression is an important part of programming,
> allowing a descriptive name to be used in place of a longer expression,
> and permitting reuse. Currently, this feature is available only in
> statement form, making it unavailable in list comprehensions and other
> expression contexts. Merely introducing a way to assign as an expression
> would create bizarre edge cases around comprehensions, though, and to avoid
> the worst of the confusions, we change the definition of comprehensions,
> causing some edge cases to be interpreted differently, but maintaining the
> existing behaviour in the majority of situations.
>
>
> Syntax and semantics
> ====================
>
> In any context where arbitrary Python expressions can be used, a **named
> expression** can appear. This is of the form ``name := expr`` where
> ``expr`` is any valid Python expression, and ``name`` is an identifier.
>
> The value of such a named expression is the same as the incorporated
> expression, with the additional side-effect that the target is assigned
> that value::
>
> # Handle a matched regex
> if (match := pattern.search(data)) is not None:
> ...
>
> # A more explicit alternative to the 2-arg form of iter() invocation
> while (value := read_next_item()) is not None:
> ...
>
> # Share a subexpression between a comprehension filter clause and its
> output
> filtered_data = [y for x in data if (y := f(x)) is not None]
>
>
> Differences from regular assignment statements
> ----------------------------------------------
>
> Most importantly, since ``:=`` is an expression, it can be used in contexts
> where statements are illegal, including lambda functions and
> comprehensions.
>
> An assignment statement can assign to multiple targets, left-to-right::
>
> x = y = z = 0
>
> The equivalent assignment expression is parsed as separate binary
> operators,
> and is therefore processed right-to-left, as if it were spelled thus::
>
> assert 0 == (x := (y := (z := 0)))
>
> Statement assignment can include annotations. This would be syntactically
> noisy in expressions, and is of minor importance. An annotation can be
> given separately from the assignment if needed::
>
> x:str = "" # works
> (x:str := "") # SyntaxError
> x:str # possibly before a loop
> (x := "") # fine
>
> Augmented assignment is not supported in expression form::
>
> >>> x +:= 1
> File "<stdin>", line 1
> x +:= 1
> ^
> SyntaxError: invalid syntax
>
> Statement assignment is able to set attributes and subscripts, but
> expression assignment is restricted to names. (This restriction may be
> relaxed in a future version of Python.)
>
> Otherwise, the semantics of assignment are identical in statement and
> expression forms.
>
>
> Alterations to comprehensions
> -----------------------------
>
> The current behaviour of list/set/dict comprehensions and generator
> expressions has some edge cases that would behave strangely if an
> assignment
> expression were to be used. Therefore the proposed semantics are changed,
> removing the current edge cases, and instead altering their behaviour
> *only*
> in a class scope.
>
> As of Python 3.7, the outermost iterable of any comprehension is evaluated
> in the surrounding context, and then passed as an argument to the implicit
> function that evaluates the comprehension.
>
> Under this proposal, the entire body of the comprehension is evaluated in
> its implicit function. Names not assigned to within the comprehension are
> located in the surrounding scopes, as with normal lookups. As one special
> case, a comprehension at class scope will **eagerly bind** any name which
> is already defined in the class scope.
>
> A list comprehension can be unrolled into an equivalent function. With
> Python 3.7 semantics::
>
> numbers = [x + y for x in range(3) for y in range(4)]
> # Is approximately equivalent to
> def <listcomp>(iterator):
> result = []
> for x in iterator:
> for y in range(4):
> result.append(x + y)
> return result
> numbers = <listcomp>(iter(range(3)))
>
> Under the new semantics, this would instead be equivalent to::
>
> def <listcomp>():
> result = []
> for x in range(3):
> for y in range(4):
> result.append(x + y)
> return result
> numbers = <listcomp>()
>
> When a class scope is involved, a naive transformation into a function
> would
> prevent name lookups (as the function would behave like a method)::
>
> class X:
> names = ["Fred", "Barney", "Joe"]
> prefix = "> "
> prefixed_names = [prefix + name for name in names]
>
> With Python 3.7 semantics, this will evaluate the outermost iterable at
> class
> scope, which will succeed; but it will evaluate everything else in a
> function::
>
> class X:
> names = ["Fred", "Barney", "Joe"]
> prefix = "> "
> def <listcomp>(iterator):
> result = []
> for name in iterator:
> result.append(prefix + name)
> return result
> prefixed_names = <listcomp>(iter(names))
>
> The name ``prefix`` is thus searched for at global scope, ignoring the
> class
> name. Under the proposed semantics, this name will be eagerly bound; and
> the
> same early binding then handles the outermost iterable as well. The list
> comprehension is thus approximately equivalent to::
>
> class X:
> names = ["Fred", "Barney", "Joe"]
> prefix = "> "
> def <listcomp>(names=names, prefix=prefix):
> result = []
> for name in names:
> result.append(prefix + name)
> return result
> prefixed_names = <listcomp>()
>
> With list comprehensions, this is unlikely to cause any confusion. With
> generator expressions, this has the potential to affect behaviour, as the
> eager binding means that the name could be rebound between the creation of
> the genexp and the first call to ``next()``. It is, however, more closely
> aligned to normal expectations. The effect is ONLY seen with names that
> are looked up from class scope; global names (eg ``range()``) will still
> be late-bound as usual.
>
> One consequence of this change is that certain bugs in genexps will not
> be detected until the first call to ``next()``, where today they would be
> caught upon creation of the generator.
>
>
> Recommended use-cases
> =====================
>
> Simplifying list comprehensions
> -------------------------------
>
> A list comprehension can map and filter efficiently by capturing
> the condition::
>
> results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0]
>
> Similarly, a subexpression can be reused within the main expression, by
> giving it a name on first use::
>
> stuff = [[y := f(x), x/y] for x in range(5)]
>
> # There are a number of less obvious ways to spell this in current
> # versions of Python, such as:
>
> # Inline helper function
> stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)]
>
> # Extra 'for' loop - potentially could be optimized internally
> stuff = [[y, x/y] for x in range(5) for y in [f(x)]]
>
> # Using a mutable cache object (various forms possible)
> c = {}
> stuff = [[c.update(y=f(x)) or c['y'], x/c['y']] for x in range(5)]
>
> In all cases, the name is local to the comprehension; like iteration
> variables,
> it cannot leak out into the surrounding context.
>
>
> Capturing condition values
> --------------------------
>
> Assignment expressions can be used to good effect in the header of
> an ``if`` or ``while`` statement::
>
> # Proposed syntax
> while (command := input("> ")) != "quit":
> print("You entered:", command)
>
> # Capturing regular expression match objects
> # See, for instance, Lib/pydoc.py, which uses a multiline spelling
> # of this effect
> if match := re.search(pat, text):
> print("Found:", match.group(0))
>
> # Reading socket data until an empty string is returned
> while data := sock.read():
> print("Received data:", data)
>
> # Equivalent in current Python, not caring about function return value
> while input("> ") != "quit":
> print("You entered a command.")
>
> # To capture the return value in current Python demands a four-line
> # loop header.
> while True:
> command = input("> ");
> if command == "quit":
> break
> print("You entered:", command)
>
> Particularly with the ``while`` loop, this can remove the need to have an
> infinite loop, an assignment, and a condition. It also creates a smooth
> parallel between a loop which simply uses a function call as its condition,
> and one which uses that as its condition but also uses the actual value.
>
>
> Rejected alternative proposals
> ==============================
>
> Proposals broadly similar to this one have come up frequently on
> python-ideas.
> Below are a number of alternative syntaxes, some of them specific to
> comprehensions, which have been rejected in favour of the one given above.
>
>
> Alternative spellings
> ---------------------
>
> Broadly the same semantics as the current proposal, but spelled
> differently.
>
> 1. ``EXPR as NAME``::
>
> stuff = [[f(x) as y, x/y] for x in range(5)]
>
> Since ``EXPR as NAME`` already has meaning in ``except`` and ``with``
> statements (with different semantics), this would create unnecessary
> confusion or require special-casing (eg to forbid assignment within the
> headers of these statements).
>
> 2. ``EXPR -> NAME``::
>
> stuff = [[f(x) -> y, x/y] for x in range(5)]
>
> This syntax is inspired by languages such as R and Haskell, and some
> programmable calculators. (Note that a left-facing arrow ``y <- f(x)``
> is
> not possible in Python, as it would be interpreted as less-than and
> unary
> minus.) This syntax has a slight advantage over 'as' in that it does not
> conflict with ``with`` and ``except`` statements, but otherwise is
> equivalent.
>
> 3. Adorning statement-local names with a leading dot::
>
> stuff = [[(f(x) as .y), x/.y] for x in range(5)] # with "as"
> stuff = [[(.y := f(x)), x/.y] for x in range(5)] # with ":="
>
> This has the advantage that leaked usage can be readily detected,
> removing
> some forms of syntactic ambiguity. However, this would be the only
> place
> in Python where a variable's scope is encoded into its name, making
> refactoring harder.
>
> 4. Adding a ``where:`` to any statement to create local name bindings::
>
> value = x**2 + 2*x where:
> x = spam(1, 4, 7, q)
>
> Execution order is inverted (the indented body is performed first,
> followed
> by the "header"). This requires a new keyword, unless an existing
> keyword
> is repurposed (most likely ``with:``). See PEP 3150 for prior
> discussion
> on this subject (with the proposed keyword being ``given:``).
>
> 5. ``TARGET from EXPR``::
>
> stuff = [[y from f(x), x/y] for x in range(5)]
>
> This syntax has fewer conflicts than ``as`` does (conflicting only with
> the
> ``raise Exc from Exc`` notation), but is otherwise comparable to it.
> Instead
> of paralleling ``with expr as target:`` (which can be useful but can
> also be
> confusing), this has no parallels, but is evocative.
>
>
> Special-casing conditional statements
> -------------------------------------
>
> One of the most popular use-cases is ``if`` and ``while`` statements.
> Instead
> of a more general solution, this proposal enhances the syntax of these two
> statements to add a means of capturing the compared value::
>
> if re.search(pat, text) as match:
> print("Found:", match.group(0))
>
> This works beautifully if and ONLY if the desired condition is based on the
> truthiness of the captured value. It is thus effective for specific
> use-cases (regex matches, socket reads that return `''` when done), and
> completely useless in more complicated cases (eg where the condition is
> ``f(x) < 0`` and you want to capture the value of ``f(x)``). It also has
> no benefit to list comprehensions.
>
> Advantages: No syntactic ambiguities. Disadvantages: Answers only a
> fraction
> of possible use-cases, even in ``if``/``while`` statements.
>
>
> Special-casing comprehensions
> -----------------------------
>
> Another common use-case is comprehensions (list/set/dict, and genexps). As
> above, proposals have been made for comprehension-specific solutions.
>
> 1. ``where``, ``let``, or ``given``::
>
> stuff = [(y, x/y) where y = f(x) for x in range(5)]
> stuff = [(y, x/y) let y = f(x) for x in range(5)]
> stuff = [(y, x/y) given y = f(x) for x in range(5)]
>
> This brings the subexpression to a location in between the 'for' loop
> and
> the expression. It introduces an additional language keyword, which
> creates
> conflicts. Of the three, ``where`` reads the most cleanly, but also has
> the
> greatest potential for conflict (eg SQLAlchemy and numpy have ``where``
> methods, as does ``tkinter.dnd.Icon`` in the standard library).
>
> 2. ``with NAME = EXPR``::
>
> stuff = [(y, x/y) with y = f(x) for x in range(5)]
>
> As above, but reusing the `with` keyword. Doesn't read too badly, and
> needs
> no additional language keyword. Is restricted to comprehensions, though,
> and cannot as easily be transformed into "longhand" for-loop syntax. Has
> the C problem that an equals sign in an expression can now create a name
> binding, rather than performing a comparison. Would raise the question
> of
> why "with NAME = EXPR:" cannot be used as a statement on its own.
>
> 3. ``with EXPR as NAME``::
>
> stuff = [(y, x/y) with f(x) as y for x in range(5)]
>
> As per option 2, but using ``as`` rather than an equals sign. Aligns
> syntactically with other uses of ``as`` for name binding, but a simple
> transformation to for-loop longhand would create drastically different
> semantics; the meaning of ``with`` inside a comprehension would be
> completely different from the meaning as a stand-alone statement, while
> retaining identical syntax.
>
> Regardless of the spelling chosen, this introduces a stark difference
> between
> comprehensions and the equivalent unrolled long-hand form of the loop. It
> is
> no longer possible to unwrap the loop into statement form without reworking
> any name bindings. The only keyword that can be repurposed to this task is
> ``with``, thus giving it sneakily different semantics in a comprehension
> than
> in a statement; alternatively, a new keyword is needed, with all the costs
> therein.
>
>
> Lowering operator precedence
> ----------------------------
>
> There are two logical precedences for the ``:=`` operator. Either it should
> bind as loosely as possible, as does statement-assignment; or it should
> bind
> more tightly than comparison operators. Placing its precedence between the
> comparison and arithmetic operators (to be precise: just lower than bitwise
> OR) allows most uses inside ``while`` and ``if`` conditions to be spelled
> without parentheses, as it is most likely that you wish to capture the
> value
> of something, then perform a comparison on it::
>
> pos = -1
> while pos := buffer.find(search_term, pos + 1) >= 0:
> ...
>
> Once find() returns -1, the loop terminates. If ``:=`` binds as loosely as
> ``=`` does, this would capture the result of the comparison (generally
> either
> ``True`` or ``False``), which is less useful.
>
> While this behaviour would be convenient in many situations, it is also
> harder
> to explain than "the := operator behaves just like the assignment
> statement",
> and as such, the precedence for ``:=`` has been made as close as possible
> to
> that of ``=``.
>
>
> Migration path
> ==============
>
> The semantic changes to list/set/dict comprehensions, and more so to
> generator
> expressions, may potentially require migration of code. In many cases, the
> changes simply make legal what used to raise an exception, but there are
> some
> edge cases that were previously legal and now are not, and a few corner
> cases
> with altered semantics.
>
>
> The Outermost Iterable
> ----------------------
>
> As of Python 3.7, the outermost iterable in a comprehension is special: it
> is
> evaluated in the surrounding context, instead of inside the comprehension.
> Thus it is permitted to contain a ``yield`` expression, to use a name also
> used elsewhere, and to reference names from class scope. Also, in a genexp,
> the outermost iterable is pre-evaluated, but the rest of the code is not
> touched until the genexp is first iterated over. Class scope is now handled
> more generally (see above), but if other changes require the old behaviour,
> the iterable must be explicitly elevated from the comprehension::
>
> # Python 3.7
> def f(x):
> return [x for x in x if x]
> def g():
> return [x for x in [(yield 1)]]
> # With PEP 572
> def f(x):
> return [y for y in x if y]
> def g():
> sent_item = (yield 1)
> return [x for x in [sent_item]]
>
> This more clearly shows that it is g(), not the comprehension, which is
> able
> to yield values (and is thus a generator function). The entire
> comprehension
> is consistently in a single scope.
>
> The following expressions would, in Python 3.7, raise exceptions
> immediately.
> With the removal of the outermost iterable's special casing, they are now
> equivalent to the most obvious longhand form::
>
> gen = (x for x in rage(10)) # NameError
> gen = (x for x in 10) # TypeError (not iterable)
> gen = (x for x in range(1/0)) # ZeroDivisionError
>
> def <genexp>():
> for x in rage(10):
> yield x
> gen = <genexp>() # No exception yet
> tng = next(gen) # NameError
>
>
> Open questions
> ==============
>
> Importing names into comprehensions
> -----------------------------------
>
> A list comprehension can use and update local names, and they will retain
> their values from one iteration to another. It would be convenient to use
> this feature to create rolling or self-effecting data streams::
>
> progressive_sums = [total := total + value for value in data]
>
> This will fail with UnboundLocalError due to ``total`` not being
> initalized.
> Simply initializing it outside of the comprehension is insufficient -
> unless
> the comprehension is in class scope::
>
> class X:
> total = 0
> progressive_sums = [total := total + value for value in data]
>
> At other scopes, it may be beneficial to have a way to fetch a value from
> the
> surrounding scope. Should this be automatic? Should it be controlled with a
> keyword? Hypothetically (and using no new keywords), this could be
> written::
>
> total = 0
> progressive_sums = [total := total + value
> import nonlocal total
> for value in data]
>
> Translated into longhand, this would become::
>
> total = 0
> def <listcomp>(total=total):
> result = []
> for value in data:
> result.append(total := total + value)
> return result
> progressive_sums = <listcomp>()
>
> ie utilizing the same early-binding technique that is used at class scope.
>
>
> Frequently Raised Objections
> ============================
>
> Why not just turn existing assignment into an expression?
> ---------------------------------------------------------
>
> C and its derivatives define the ``=`` operator as an expression, rather
> than
> a statement as is Python's way. This allows assignments in more contexts,
> including contexts where comparisons are more common. The syntactic
> similarity
> between ``if (x == y)`` and ``if (x = y)`` belies their drastically
> different
> semantics. Thus this proposal uses ``:=`` to clarify the distinction.
>
>
> This could be used to create ugly code!
> ---------------------------------------
>
> So can anything else. This is a tool, and it is up to the programmer to
> use it
> where it makes sense, and not use it where superior constructs can be used.
>
>
> With assignment expressions, why bother with assignment statements?
> -------------------------------------------------------------------
>
> The two forms have different flexibilities. The ``:=`` operator can be
> used
> inside a larger expression; the ``=`` statement can be augmented to ``+=``
> and
> its friends, can be chained, and can assign to attributes and subscripts.
>
>
> Why not use a sublocal scope and prevent namespace pollution?
> -------------------------------------------------------------
>
> Previous revisions of this proposal involved sublocal scope (restricted to
> a
> single statement), preventing name leakage and namespace pollution. While
> a
> definite advantage in a number of situations, this increases complexity in
> many others, and the costs are not justified by the benefits. In the
> interests
> of language simplicity, the name bindings created here are exactly
> equivalent
> to any other name bindings, including that usage at class or module scope
> will
> create externally-visible names. This is no different from ``for`` loops
> or
> other constructs, and can be solved the same way: ``del`` the name once it
> is
> no longer needed, or prefix it with an underscore.
>
> Names bound within a comprehension are local to that comprehension, even in
> the outermost iterable, and can thus be used freely without polluting the
> surrounding namespace.
>
> (The author wishes to thank Guido van Rossum and Christoph Groth for their
> suggestions to move the proposal in this direction. [2]_)
>
>
> Style guide recommendations
> ===========================
>
> As expression assignments can sometimes be used equivalently to statement
> assignments, the question of which should be preferred will arise. For the
> benefit of style guides such as PEP 8, two recommendations are suggested.
>
> 1. If either assignment statements or assignment expressions can be
> used, prefer statements; they are a clear declaration of intent.
>
> 2. If using assignment expressions would lead to ambiguity about
> execution order, restructure it to use statements instead.
>
>
> Acknowledgements
> ================
>
> The author wishes to thank Guido van Rossum and Nick Coghlan for their
> considerable contributions to this proposal, and to members of the
> core-mentorship mailing list for assistance with implementation.
>
>
> References
> ==========
>
> .. [1] Proof of concept / reference implementation
> (https://github.com/Rosuav/cpython/tree/assignment-expressions)
> .. [2] Pivotal post regarding inline assignment semantics
> (https://mail.python.org/pipermail/python-ideas/2018-March/049409.html)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> ..
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> coding: utf-8
> End:
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180426/1fe8baf4/attachment-0001.html>
More information about the Python-Dev
mailing list