PEP 640: Unused variable syntax.
One of the problems I have with the Pattern Matching proposal (PEP 622 originally, now PEPs 634, 635, 636) is the special-casing of '_' to not actually assign to the name, which is a subtle but meaningful divergence from the rest of Python. In discussions with the authors I proposed using '?' instead *and* extending that to existing unpacking syntax. The PEP authors were understandably a little hesitant to tack that onto their already quite extensive proposal, so I suggested making it a separate proposal. That proposal is at https://www.python.org/dev/peps/pep-0640 (unless that hasn't updated yet, in which case it's at https://github.com/python/peps/blob/master/pep-0640.rst), and also included below. This proposal doesn't necessarily require pattern matching to be accepted -- the new syntax stands well enough on its own -- but I'm recommending this *not* be accepted if pattern matching using the same syntax is not also accepted. The benefit without pattern matching is real but small, and in my opinion it's not worth the added complexity. The PEP: PEP: 640 Title: Unused variable syntax Author: Thomas Wouters <thomas@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 04-Oct-2020 Python-Version: 3.10 Post-History: Abstract ======== This PEP proposes new syntax for *unused variables*, providing a pseudo-name that can be assigned to but not otherwise used. The assignment doesn't actually happen, and the value is discarded instead. Motivation ========== In Python it is somewhat common to need to do an assignment without actually needing the result. Conventionally, people use either ``"_"`` or a name such as ``"unused"`` (or with ``"unused"`` as a prefix) for this. It's most common in *unpacking assignments*:: x, unused, z = range(3) x, *unused, z = range(10) It's also used in ``for`` loops and comprehensions:: for unused in range(10): ... [ SpamObject() for unused in range(10) ] The use of ``"_"`` in these cases is probably the most common, but it potentially conflicts with the use of ``"_"`` in internationalization, where a call like gettext.gettext() is bound to ``"_"`` and used to mark strings for translation. In the proposal to add Pattern Matching to Python (originally PEP 622, now split into PEP 634, PEP 635 and PEP 636), ``"_"`` has an *additional* special meaning. It is a wildcard pattern, used in places where variables could be assigned to, to indicate anything should be matched but not assigned to anything. The choice of ``"_"`` there matches the use of ``"_"`` in other languages, but the semantic difference with ``"_"`` elsewhere in Python is significant. This PEP proposes to allow a special token, ``?``, to be used instead. This has most of the benefits of ``"_"`` without affecting other uses of that otherwise regular variable. Allowing the use of the same wildcard pattern would make pattern matching and unpacking assignment more consistent with each other. Rationale ========= Marking certain variables as unused is a useful tool, as it helps clarity of purpose of the code. It makes it obvious to readers of the code as well as automated linters, that a particular variable is *intentionally* unused. However, despite the convention, ``"_"`` is not a special variable. The value is still assigned to, the object it refers to is still kept alive until the end of the scope, and it can still be used. Nor is the use of ``"_"`` for unused variables entirely ubiquitous, since it conflicts with conventional internationalization, it isn't obvious that it is a regular variable, and it isn't as obviously unused like a variable named ``"unused"``. In the Pattern Matching proposal, the use of ``"_"`` for wildcard patterns side-steps the problems of ``"_"`` for unused variables by virtue of it being in a separate scope. The only conflict it has with internationalization is one of potential confusion, it will not actually interact with uses of a global variable called ``"_"``. However, the special-casing of ``"_"`` for this wildcard pattern purpose is still problematic: the different semantics *and meaning* of ``"_"`` inside pattern matching and outside of it means a break in consistency in Python. Introducing ``?`` as special syntax for unused variables *both inside and outside pattern matching* allows us to retain that consistency. It avoids the conflict with internationalization *or any other uses of _ as a variable*. It makes unpacking assignment align more closely with pattern matching, making it easier to explain pattern matching as an extension of unpacking assignment. In terms of code readability, using a special token makes it easier to find out what it means (``"what does question mark in Python do"`` versus ``"why is my _ variable not getting assigned to"``), and makes it more obvious that the actual intent is for the value to be unused -- since it is entirely impossible to use it. Specification ============= A new token is introduced, ``"?"``, or ``token.QMARK``. The grammar is modified to allow ``"?"`` in assignment contexts (``star_atom`` and ``t_atom`` in the current grammar), creating a ``Name`` AST node with identifier set to NULL. The AST is modified to allow the ``Name`` expression's identifier to be optional (it is currently required). The identifier being empty would only be allowed in a ``STORE`` context. In CPython, the bytecode compiler is modified to emit ``POP_TOP`` instead of ``STORE_NAME`` for ``Name`` nodes with no identifier. Other uses of the ``Name`` node are updated to handle the identifier being empty, as appropriate. The uses of the modified grammar nodes encompass at least the following forms of assignment:: ? = ... x, ?, z = ... x, *?, z = ... for ? in range(3): ... # including comprehension forms for x, ?, z in matrix: ... # including comprehension forms with open(f) as ?: ... with func() as (x, ?, z): ... The use of a single ``"?"``, not in an unpacking context, is allowed in normal assignment and the ``with`` statement. It doesn't really make sense on its own, and it is possible to disallow those specific cases. However, ``for ? in range(3)`` clearly has its uses, so for consistency reasons if nothing else it seems more sensible to allow the use of the single ``"?"`` in other cases. Using ``"?"`` in augmented assignment (``? *= 2``) is not allowed, since ``"?"`` can only be used for assignment. Backwards Compatibility ======================= Introducing a new token means there are no backward compatibility concerns. No valid syntax changes meaning. "?" is not considered an identifier, so str.isidentifier() does not change. The AST does change in an incompatible way, as the identifier of a Name token can now be empty. Code using the AST will have to be adjusted accordingly. How to Teach This ================= ``?`` can be introduced along with unpacking assignment, explaining it is special syntax for 'unused' and mentioning that it can also be used in other places. Alternatively, it could be introduced as part of an explanation on assignment in ``for`` loops, showing an example where the loop variable is unused. PEP 636 discusses how to teach ``"_"``, and can simply replace ``"_"`` with ``?``, perhaps noting that ``?`` is similarly usable in other contexts. Reference Implementation ======================== A prototype implementation exists at <https://github.com/Yhg1s/cpython/tree/nonassign>. Rejected Ideas ============== Open Issues =========== Should ``"?"`` be allowed in the following contexts:: # imports done for side-effect only. import os as ? from os import path as ? # Function defined for side-effects only (e.g. decorators) @register_my_func def ?(...): ... # Class defined for side-effects only (e.g. decorators, __init_subclass__) class ?(...): ... # Parameters defined for unused positional-only arguments: def f(a, ?, ?): ... lambda a, ?, ?: ... # Unused variables with type annotations: ?: int = f() # Exception handling: try: ... except Exception as ?: ... # With blocks: with open(f) as ?: ... Some of these may seem to make sense from a consistency point of view, but practical uses are limited and dubious. Type annotations on ``"?"`` and using it with ``except`` and ``with`` do not seem to make any sense. In the reference implementation, ``except`` is not supported (the existing syntax only allows a name) but ``with`` is (by virtue of the existing syntax supporting unpacking assignment). Should this PEP be accepted even if pattern matching is rejected? Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
I can't say that I like the look of pseudo-assignment to question mark: for ? in range(20): ... but I could probably learn to live with it. But one of your rationalisations:
and makes it more obvious that the actual intent is for the value to be unused -- since it is entirely impossible to use it.
is actually an anti-feature, in my opinion. I think that people might like the idea of not actually binding a value in situations like this: a, *?, b = expression until you end up with something unexpected in a and b and need to debug what it going on, either in a debugger or with print: a, *?, b = expression print(?) # wait this doesn't work In my opinion, having a convention to treat certain variables as "unused" is great (I'm partial to `__` myself, to avoid clobbering the special variable `_` in the REPL). But having that be a pseudo-variable which is *actually* unused and unuseable strikes me as being an attractive nuisance. -- Steve
On Tue, Oct 20, 2020 at 11:26 AM Steven D'Aprano <steve@pearwood.info> wrote:
I can't say that I like the look of pseudo-assignment to question mark:
for ? in range(20): ...
but I could probably learn to live with it. But one of your rationalisations:
and makes it more obvious that the actual intent is for the value to be unused -- since it is entirely impossible to use it.
is actually an anti-feature, in my opinion.
I think that people might like the idea of not actually binding a value in situations like this:
a, *?, b = expression
until you end up with something unexpected in a and b and need to debug what it going on, either in a debugger or with print:
a, *?, b = expression print(?) # wait this doesn't work;
I'm not sure how this is different than, say, a, _, _ = range(3) print(_) or a = range(3)[0] print(<whatever the second value is>) That is to say, some things are impossible; if you want to print the value, don't assign to '?'.
In my opinion, having a convention to treat certain variables as "unused" is great (I'm partial to `__` myself, to avoid clobbering the special variable `_` in the REPL). But having that be a pseudo-variable which is *actually* unused and unuseable strikes me as being an attractive nuisance.
And yet that's exactly what is being proposed in pattern matching. https://www.python.org/dev/peps/pep-0634/#id3
-- Steve _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OPUUJWBC... Code of Conduct: http://python.org/psf/codeofconduct/
-- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On Tue, Oct 20, 2020 at 01:41:02PM +0200, Thomas Wouters wrote:
a, *?, b = expression print(?) # wait this doesn't work;
I'm not sure how this is different than, say,
a, _, _ = range(3) print(_)
For starters, that will actually print something, not fail with a SyntaxError or runtime exception. You are correct that if I use the same `_` variable in multiple places: a, _, _, b, _ = something and then need to see the "ignore" values for debugging, I can't. I will only see the value in the final `_` unless I rename them. But with your proposal, I can't even see a *single* `?` (non-)target. So that's even more inconvenient than `_` because it affects more cases. - `_` is inconvenient for debugging if you have two or more in one assignment statement; - `?` is inconvenient for debugging if you have two or more in one assignment statement; - PLUS if you have only a single one in the statement. So strictly more inconvenient. Each time I use a `?` target, I expect that eventually I will need to debug that piece of code. When I need to debug it, I will need to change the `?` into a real variable. I don't plan to change it back: why churn the code unnecessarily? So there's this ratchett effect, where over time every `?` I use will eventually be converted into a real variable. So why not just use real variables in the first place? Unless there is a seriously large performance gain from `?` I wouldn't bother using it in the first place. I don't hate the idea, but I don't think it's very useful either.
That is to say, some things are impossible; if you want to print the value, don't assign to '?'.
Indeed, but the problem is that when I name the variable *today*, I might not know that *tomorrow* I will need to see its value.
In my opinion, having a convention to treat certain variables as "unused" is great (I'm partial to `__` myself, to avoid clobbering the special variable `_` in the REPL). But having that be a pseudo-variable which is *actually* unused and unuseable strikes me as being an attractive nuisance.
And yet that's exactly what is being proposed in pattern matching. https://www.python.org/dev/peps/pep-0634/#id3
Quote: "A wildcard pattern always succeeds. It binds no name." That says nothing about variables. It defines a *pattern* which always matches anything. Patterns are a different kind of thing to variables, just as keywords are not variables. A better argument for your position woud be this quote from a different PEP, 635: "The wildcard pattern is a special case of a 'capture' pattern: it accepts any value, but does not bind it to a variable." https://www.python.org/dev/peps/pep-0635/#id2 But I think that description makes a conceptual error. I think it is a mistake to think of the wildcard pattern as a non-binding capture pattern, since that's a contradiction: if it never captures any value, how can it be a capture pattern? I think that it is important to recognise that the wildcard pattern is *distinct* from capture patterns, literal patterns etc. Like literal patterns, it captures nothing. Unlike literal and other patterns, it matches everything. There's a certainly similarity to capture patterns No, that's not "exactly what is being proposed". The *pattern* "_" is not an assignment target. Capture patterns are similar to assigment targets, but patterns in general are not: case [] does not attempt to use [] as an assignment target. Likewise for value patterns such as `case HttpStatus.OK`. Value patterns like HttpStatus.OK are perfectly legal assignment targets, but that's not what they mean inside a case statement. Symbols having multiple meanings are not so unusual: - inside a float, a . is part of the float; - outside of a float, a . is an attribute delimiter; - the chars 'None' have special meaning on their own; - but otherwise inside an identifier, or a string, they're just letters; - round brackets (parentheses) can mean grouping or calling; - square brackets create lists, or subscripting; etc. Why are we so hung up over the fact that inside a case statement, the underscore means something different from outside of a match statement? It seems like a very minor issue to worry about. If we can accept that `None` and other keywords are syntactically legal identifiers, but not actually usable as identifiers as they are reserved for other purposes, then we ought to be able to accept that `_` is syntactically a legal assignment target, but not usable as a capture pattern as it is reserved for another purpose (wildcard pattern). -- Steve
On 20Oct2020 1021, Steven D'Aprano wrote:
In my opinion, having a convention to treat certain variables as "unused" is great (I'm partial to `__` myself, to avoid clobbering the special variable `_` in the REPL). But having that be a pseudo-variable which is *actually* unused and unuseable strikes me as being an attractive nuisance.
I agree entirely. The convention is fine, and the workaround for when you don't want to overwrite a legitimate `_` variable is also fine. Making `_` enough of an official convention (but how?) that linting tools stop warning about it (e.g. type checkers might warn about multiple conflicting assignments) seems like an overall happier path, that neither makes existing code forwards-incompatible nor future code backwards-incompatible. I don't think we have to codify every emergent coding pattern in syntax. Cheers, Steve
On Mon, Oct 19, 2020 at 3:11 PM Thomas Wouters <thomas@python.org> wrote:
PEP: 640 Title: Unused variable syntax Author: Thomas Wouters <thomas@python.org>
...
In Python it is somewhat common to need to do an assignment without actually needing the result. Conventionally, people use either ``"_"`` or a name such as ``"unused"`` (or with ``"unused"`` as a prefix) for this. It's most common in *unpacking assignments*::
Many times I'm not using an assignment target, I still like to give a descriptive name. The reason is that it lets me see what value I'm not using. It helps to document and confirm my understanding of the value being unpacked. It also lets you toggle easily between using and not using a value if you're working on the code. To illustrate, I might do this-- scheme, _netloc, _path, params, query, fragment = urlparse(url) instead of this-- scheme, _, _, params, query, fragment = urlparse(url) So I'd prefer if the scheme would allow including a name (either by prefixing or some other method), or at least not preclude such an extension in the future. --Chris
On Tue, Oct 20, 2020 at 2:02 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Mon, Oct 19, 2020 at 3:11 PM Thomas Wouters <thomas@python.org> wrote:
PEP: 640 Title: Unused variable syntax Author: Thomas Wouters <thomas@python.org>
...
In Python it is somewhat common to need to do an assignment without actually needing the result. Conventionally, people use either ``"_"`` or a name such as ``"unused"`` (or with ``"unused"`` as a prefix) for this. It's most common in *unpacking assignments*::
Many times I'm not using an assignment target, I still like to give a descriptive name. The reason is that it lets me see what value I'm not using. It helps to document and confirm my understanding of the value being unpacked. It also lets you toggle easily between using and not using a value if you're working on the code.
To illustrate, I might do this--
scheme, _netloc, _path, params, query, fragment = urlparse(url)
instead of this--
scheme, _, _, params, query, fragment = urlparse(url)
So I'd prefer if the scheme would allow including a name (either by prefixing or some other method), or at least not preclude such an extension in the future.
It does not preclude it -- ?somename is not valid syntax, so it could be added later -- but please note that the pattern matching proposal also does not allow this. Using names instead of ? is still an option -- both in regular unpacking and in pattern matching -- it just does something subtly different. The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching.
--Chris
-- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On Tue, 20 Oct 2020 at 13:13, Thomas Wouters <thomas@python.org> wrote:
The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching.
That's something that should be addressed or debated in the pattern matching PEP. I'm -1 on this PEP being *solely* to patch over a wart in the pattern matching PEP, and the other justifications for the PEP as a standalone proposal don't seem to be convincing people (they don't convince me either, FWIW). Paul
On Tue, Oct 20, 2020 at 2:22 PM Paul Moore <p.f.moore@gmail.com> wrote:
The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in
On Tue, 20 Oct 2020 at 13:13, Thomas Wouters <thomas@python.org> wrote: pattern matching.
That's something that should be addressed or debated in the pattern matching PEP. I'm -1 on this PEP being *solely* to patch over a wart in the pattern matching PEP, and the other justifications for the PEP as a standalone proposal don't seem to be convincing people (they don't convince me either, FWIW).
I did say, in the original email: This proposal doesn't necessarily require pattern matching to be accepted -- the new syntax stands well enough on its own -- but I'm recommending this *not* be accepted if pattern matching using the same syntax is not also accepted. The benefit without pattern matching is real but small, and in my opinion it's not worth the added complexity. -- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On Tue, 20 Oct 2020 at 13:25, Thomas Wouters <thomas@python.org> wrote:
On Tue, Oct 20, 2020 at 2:22 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Tue, 20 Oct 2020 at 13:13, Thomas Wouters <thomas@python.org> wrote:
The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching.
That's something that should be addressed or debated in the pattern matching PEP. I'm -1 on this PEP being *solely* to patch over a wart in the pattern matching PEP, and the other justifications for the PEP as a standalone proposal don't seem to be convincing people (they don't convince me either, FWIW).
I did say, in the original email:
This proposal doesn't necessarily require pattern matching to be accepted -- the new syntax stands well enough on its own -- but I'm recommending this *not* be accepted if pattern matching using the same syntax is not also accepted. The benefit without pattern matching is real but small, and in my opinion it's not worth the added complexity.
Understood. But unless I'm missing something, the pattern matching PEP(s) is/are in limbo at the moment, there's a lot going on in github but nothing has been posted here. So I'm not clear what there is to discuss here at the moment, if the proposal is only relevant if pattern matching includes it, but no published pattern matching PEP has suggested it... (Sorry if the above sounds a little disgruntled, it feels like there's a lot going in "in private" with the pattern matching PEP and I sort of feel like a bit more transparency would be good. Maybe I'm mistaken...) Paul
On Tue, Oct 20, 2020 at 2:44 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Tue, 20 Oct 2020 at 13:25, Thomas Wouters <thomas@python.org> wrote:
On Tue, Oct 20, 2020 at 2:22 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Tue, 20 Oct 2020 at 13:13, Thomas Wouters <thomas@python.org> wrote:
The reason for this PEP is that pattern matching will make '_' (but
not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching.
That's something that should be addressed or debated in the pattern matching PEP. I'm -1 on this PEP being *solely* to patch over a wart in the pattern matching PEP, and the other justifications for the PEP as a standalone proposal don't seem to be convincing people (they don't convince me either, FWIW).
I did say, in the original email:
This proposal doesn't necessarily require pattern matching to be accepted -- the new syntax stands well enough on its own -- but I'm recommending this *not* be accepted if pattern matching using the same syntax is not also accepted. The benefit without pattern matching is real but small, and in my opinion it's not worth the added complexity.
Understood. But unless I'm missing something, the pattern matching PEP(s) is/are in limbo at the moment, there's a lot going on in github but nothing has been posted here. So I'm not clear what there is to discuss here at the moment, if the proposal is only relevant if pattern matching includes it, but no published pattern matching PEP has suggested it...
They are not in limbo. They are actively being worked on. (At the sprints Brandt mentioned they expect to post updated PEPs later this week.) The Steering Council had a conversation with the PEP authors a while back, discussing various objections and alternatives, including using something else instead of '_'. At that time they were already talking about splitting the PEP up into three parts (which they've since done, but not posted about yet). I'm not sure how to put it differently than I have in the PEP or the email: I proposed they use ? instead of _ and also apply that to regular unpacking (because it is very easy to see pattern matching as an extension of unpacking assignment), and (besides other disagreements) they were uncomfortable including non-pattern-matching proposals in their PEP. This PEP covers the non-pattern-matching uses of '?'. (Sorry if the above sounds a little disgruntled, it feels like there's
a lot going in "in private" with the pattern matching PEP and I sort of feel like a bit more transparency would be good. Maybe I'm mistaken...)
It's not so much 'private' as 'in separate groups', and they're really still processing all the feedback they've received about PEP 622. There's a #pattern-matching channel on the discord server used for the core dev sprints right now (that all sprinters have access to), and the work on PEPs 634, 635 and 636 is happening on the peps repo. -- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On Tue, 20 Oct 2020 at 14:26, Thomas Wouters <thomas@python.org> wrote:
I'm not sure how to put it differently than I have in the PEP or the email: I proposed they use ? instead of _ and also apply that to regular unpacking (because it is very easy to see pattern matching as an extension of unpacking assignment), and (besides other disagreements) they were uncomfortable including non-pattern-matching proposals in their PEP. This PEP covers the non-pattern-matching uses of '?'.
OK, fair. In which case, I'll probably wait for the revised pattern matching PEP, but I expect to be -1 on using ? there as well. I was fine with _ and the special rule TBH. Paul
On 20Oct2020 1309, Thomas Wouters wrote:
The reason for this PEP is that pattern matching will make '_' (but not any other names) have the behaviour suggested in this PEP, but *only* in pattern matching.
Then why is this PEP proposing a different syntax? At the very least, wait for pattern matching to get in before proposing an expansion, so then you won't be caught out suggesting the wrong thing :) Cheers, Steve
Hello, On Tue, 20 Oct 2020 00:00:49 +0200 Thomas Wouters <thomas@python.org> wrote:
One of the problems I have with the Pattern Matching proposal (PEP 622 originally, now PEPs 634, 635, 636) is the special-casing of '_' to not actually assign to the name, which is a subtle but meaningful divergence from the rest of Python. In discussions with the authors I proposed using '?' instead *and* extending that to existing unpacking
One problem with this PEP, which I didn't see mentioned in the other replies, is that it tries to grab "?" character, which is already sought-for by another pending PEP: "PEP 505 -- None-aware operators", https://www.python.org/dev/peps/pep-0505/ . Use of "?" in PEP640 can be disambiguated enough from PEP505's "??", "?.", "?[" from a compiler token perspective perspective, but what about confusion/clarity to humans: ?, ?, c = a d = b?.c ?, c = b ?? (None, 2) e = b?[i] (And PEP 505 would need to be addressed sooner or later, now that it's part of other mainstream languages. In Python's conceptual debt tower, non-aware operators definitely precede pattern matching). [] -- Best regards, Paul mailto:pmiscml@gmail.com
On 10/20/20 10:45 PM, Paul Sokolovsky wrote:
One problem with this PEP, which I didn't see mentioned in the other replies, is that it tries to grab "?" character, which is already sought-for by another pending PEP: "PEP 505 -- None-aware operators", https://www.python.org/dev/peps/pep-0505/ .
I don't think PEP 505 is much of a concern. It's from 2015, and it's marked "Deferred". I suspect if it was going to happen it would have happened by now. //arry/
participants (7)
-
Chris Jerdonek -
Larry Hastings -
Paul Moore -
Paul Sokolovsky -
Steve Dower -
Steven D'Aprano -
Thomas Wouters