Time to relax some restrictions on the walrus operator?
I don't have an opinion one way or the other, but there is a discussion on Discourse about the walrus operator: https://discuss.python.org/t/walrus-fails-with/15606/1 Just a quick straw poll, how would people feel about relaxing the restriction on the walrus operator so that iterable unpacking is allowed? # Currently a syntax error. results = (1, 2, (a, b) := (3, 4), 5) which would create the following bindings: results = (1, 2, (3, 4), 5) a = 3 b = 4 A more complex example: expression = "uvwxyz" results = (1, 2, ([a, *b, c] := expression), 5) giving: results = (1, 2, "uvwxyz", 5) a = "u" b = ("v", "w", "x", "y") c = "z" Thoughts? -- Steve
A while ago there was a discussion about allowing "match" patterns for the walrus operator. This would cover iterable unpacking as you suggested along with all the patterns allowed in match statements. if [x, y, z] := re.match(r"...").groups(): print(x, y, z) The walrus expression would evaluate to None if the pattern on the left can't be matched. print(x := 42) # 42 print(1 := 42) # None This would make it really useful in if statements and list comprehensions. Here are a couple motivating examples: # Buy every pizza on the menu cost_for_all_pizzas = sum( price for food in menu if ({"type": "pizza", "price": price} := food) ) # Monitor service health while Response(status=200, json={"stats": stats}) := health_check(): print(stats) time.sleep(5)
On Sun, 8 May 2022 at 22:09, Valentin Berlier <berlier.v@gmail.com> wrote:
What if it does match, though? That's a little less clear. I like the idea, but I'm not entirely sure what this should do, for instance: print( {"x": x} := some_dict ) Clearly it should assign x to some_dict["x"] if possible, but then what should get printed out? A small dict with one key/value pair? The original dict? There would need to be a guarantee that the return value is truthy, otherwise it'll create bizarre situations where you try to match and it looks as if the match failed. By the way: With any proposal to further generalize assignment expressions, it'd be good to keep in mind the "implicit nonlocal" effect that they currently have with comprehensions:
My best guess is that this effect should be extended to any simple names that could _potentially_ be assigned to, even if (as in the case of some match expressions) they might not all actually be assigned to. ChrisA
What if it does match, though?
The walrus operator returns the value on the right side so this wouldn't change. In your example the original dict would get printed. some_dict = {"x": 1, "y": 2} print({"x": x} := some_dict) # {"x": 1, "y": 2} The only pattern where it's not possible to know if the pattern was matched by looking at the return value is the None pattern: print(None := x) If the pattern matches this will print the value of x, None, and if the pattern doesn't match this will also print None because the walrus operator evaluates to None instead of the value on the right side when the pattern doesn't match. This is not a problem since testing for None is normally done with the is operator: x is None The only patterns where it's not possible to know if the pattern was matched by looking at the truthiness of the return values are the following: print(None := x) print(False := x) print(0 := x) print(0.0 := x) print("" := x) print([] := x) print({} := x) This is not a problem since in all of these cases testing for truthiness is normally done with bool() or by testing the value directly in an if statement. In my opinion the behavior is fairly unsurprising: return the right side if the pattern matches or None otherwise. We can even explain the current restricted behavior in terms of pattern matching: the name on the left side is an "irrefutable" pattern which will always match and therefore always return the right side of the expression.
On Mon, 9 May 2022 at 00:04, Valentin Berlier <berlier.v@gmail.com> wrote:
Yes, but what if you're testing for something that could *potentially* match one of these empty objects? There's a reason that re.match always returns a match object, NOT the matched string, because if a regex matches an empty string, "if re.match(...)" should still count it as true.
In my opinion the behavior is fairly unsurprising: return the right side if the pattern matches or None otherwise. We can even explain the current restricted behavior in terms of pattern matching: the name on the left side is an "irrefutable" pattern which will always match and therefore always return the right side of the expression.
There will always be edge cases, though. Maybe pathological ones, but edge cases nonetheless. I'm worried that this will end up being a bug magnet, or conversely, that people will have to work around it with "if ([x] := foo()) is not None:", which is way too clunky. But I would like to see something like this happen. A match expression is also something people have wanted from time to time. ChrisA
Yes, but what if you're testing for something that could *potentially* match one of these empty objects?
The right side can absolutely be falsy but to be able to conflate the falsy return value with the None emitted when the pattern doesn't match, the left side has to be one of the dubious patterns I listed.
I'm worried that this will end up being a bug magnet, or conversely, that people will have to work around it with "if ([x] := foo()) is not None:", which is way too clunky.
In your example "if ([x] := foo()) is not None:" there is no possible value returned by foo() that could be both falsy and match the [x] pattern at the same time. All patterns besides the ones I listed can only be matched by truthy values so the work-around would only be needed for those dubious patterns. I think I'll experiment with a prototype when I have more time.
On Mon, 9 May 2022 at 00:58, Valentin Berlier <berlier.v@gmail.com> wrote:
In your example "if ([x] := foo()) is not None:" there is no possible value returned by foo() that could be both falsy and match the [x] pattern at the same time. All patterns besides the ones I listed can only be matched by truthy values so the work-around would only be needed for those dubious patterns.
Do you count pathological examples? class FakeEmpty(list): def __bool__(self): return False borked = FakeEmpty(("zero",)) match borked: case [x]: if borked: print("It's iterable.", x) else: print("Wut") case _: print("It's not [x].") "No possible value" ought to exclude anything that can be created with a couple of lines of Python like this. I could accept "no non-pathological values" for the [x] case, but are there any match types that could unexpectedly match a falsy value in normal circumstances? I'm *really* not a fan of having to put "if (....) is not None" just to work around a rare possibility, but I'm even less a fan of lurking bugs waiting to strike (think about how datetime.time() used to be false for midnight). Boilerplate to dodge bugs is C's job, not Python's. ChrisA
Yeah you can technically craft such pathological edge cases but this is already heavily discouraged. Libraries that change the usual semantics of python's object model are rare. The only exception I can think of would be numpy which disallows truthiness checks because of the ambiguity of arrays being both scalars and containers. However, this means that the "if x := get_some_array():" construct is already ill-formed so nothing would change by generalizing the left side to arbitrary patterns. And since numpy arrays aren't proper Sequence types you can't use them with the current implementation of pattern matching anyway. So besides carefully crafted pathological edge-cases the "if (...) is not None" construct would be completely redundant.
On Mon, 9 May 2022 at 01:07, MRAB <python@mrabarnett.plus.com> wrote:
It doesn't actually even need to be a different operator. There is currently no semantic meaning for "pattern := expr" save in the strict case where the pattern is a simple name, so semantics can be granted according to what's most useful. ChrisA
On Sun, 8 May 2022 at 19:38, Ethan Furman <ethan@stoneleaf.us> wrote:
And what is wrong with cost_for_all_pizzas = 0 for food in menu: match food: case {"type": "pizza", "price": price}: cost_for_all_pizzas += price Seriously, people are just getting used to match statements (I had to look the syntax up for the above) so IMO it's a bit early to be trying to cram them into dubious one-liners. Paul.
On Mon, 9 May 2022 at 07:36, Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:
The restrictions were never technical. They were always social - too many people were scared that assignment expressions would lead to a plethora of problems. (Which is not unreasonable. Guido made assignment a statement very deliberately, back when Python was first created. Changing that decision, or even modifying it slightly, should indeed be thoroughly thought out.) Requiring that the target be a simple name only was one way to minimize the risk. ChrisA
On Mon, 9 May 2022 at 08:35, Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:
I've always thought of relaxing the restrictions. It's actually easy to implement (even including attribute assignment/subscript assignment) and I had to modify only two files (plus regenerate the parser).
I think attribute assignment needs care to be mixed with structural pattern matching, since obj.attr can also be considered as a value pattern. Note attribute assignment with the walrus operator is not absolutely necessary since the purpose can also be achieved with a version of the function setattr which returns not None but the set value. Best regards, Takuo Matsuoka
A while ago there was a discussion about allowing "match" patterns for the walrus operator. This would cover iterable unpacking as you suggested along with all the patterns allowed in match statements. if [x, y, z] := re.match(r"...").groups(): print(x, y, z) The walrus expression would evaluate to None if the pattern on the left can't be matched. print(x := 42) # 42 print(1 := 42) # None This would make it really useful in if statements and list comprehensions. Here are a couple motivating examples: # Buy every pizza on the menu cost_for_all_pizzas = sum( price for food in menu if ({"type": "pizza", "price": price} := food) ) # Monitor service health while Response(status=200, json={"stats": stats}) := health_check(): print(stats) time.sleep(5)
On Sun, 8 May 2022 at 22:09, Valentin Berlier <berlier.v@gmail.com> wrote:
What if it does match, though? That's a little less clear. I like the idea, but I'm not entirely sure what this should do, for instance: print( {"x": x} := some_dict ) Clearly it should assign x to some_dict["x"] if possible, but then what should get printed out? A small dict with one key/value pair? The original dict? There would need to be a guarantee that the return value is truthy, otherwise it'll create bizarre situations where you try to match and it looks as if the match failed. By the way: With any proposal to further generalize assignment expressions, it'd be good to keep in mind the "implicit nonlocal" effect that they currently have with comprehensions:
My best guess is that this effect should be extended to any simple names that could _potentially_ be assigned to, even if (as in the case of some match expressions) they might not all actually be assigned to. ChrisA
What if it does match, though?
The walrus operator returns the value on the right side so this wouldn't change. In your example the original dict would get printed. some_dict = {"x": 1, "y": 2} print({"x": x} := some_dict) # {"x": 1, "y": 2} The only pattern where it's not possible to know if the pattern was matched by looking at the return value is the None pattern: print(None := x) If the pattern matches this will print the value of x, None, and if the pattern doesn't match this will also print None because the walrus operator evaluates to None instead of the value on the right side when the pattern doesn't match. This is not a problem since testing for None is normally done with the is operator: x is None The only patterns where it's not possible to know if the pattern was matched by looking at the truthiness of the return values are the following: print(None := x) print(False := x) print(0 := x) print(0.0 := x) print("" := x) print([] := x) print({} := x) This is not a problem since in all of these cases testing for truthiness is normally done with bool() or by testing the value directly in an if statement. In my opinion the behavior is fairly unsurprising: return the right side if the pattern matches or None otherwise. We can even explain the current restricted behavior in terms of pattern matching: the name on the left side is an "irrefutable" pattern which will always match and therefore always return the right side of the expression.
On Mon, 9 May 2022 at 00:04, Valentin Berlier <berlier.v@gmail.com> wrote:
Yes, but what if you're testing for something that could *potentially* match one of these empty objects? There's a reason that re.match always returns a match object, NOT the matched string, because if a regex matches an empty string, "if re.match(...)" should still count it as true.
In my opinion the behavior is fairly unsurprising: return the right side if the pattern matches or None otherwise. We can even explain the current restricted behavior in terms of pattern matching: the name on the left side is an "irrefutable" pattern which will always match and therefore always return the right side of the expression.
There will always be edge cases, though. Maybe pathological ones, but edge cases nonetheless. I'm worried that this will end up being a bug magnet, or conversely, that people will have to work around it with "if ([x] := foo()) is not None:", which is way too clunky. But I would like to see something like this happen. A match expression is also something people have wanted from time to time. ChrisA
Yes, but what if you're testing for something that could *potentially* match one of these empty objects?
The right side can absolutely be falsy but to be able to conflate the falsy return value with the None emitted when the pattern doesn't match, the left side has to be one of the dubious patterns I listed.
I'm worried that this will end up being a bug magnet, or conversely, that people will have to work around it with "if ([x] := foo()) is not None:", which is way too clunky.
In your example "if ([x] := foo()) is not None:" there is no possible value returned by foo() that could be both falsy and match the [x] pattern at the same time. All patterns besides the ones I listed can only be matched by truthy values so the work-around would only be needed for those dubious patterns. I think I'll experiment with a prototype when I have more time.
On Mon, 9 May 2022 at 00:58, Valentin Berlier <berlier.v@gmail.com> wrote:
In your example "if ([x] := foo()) is not None:" there is no possible value returned by foo() that could be both falsy and match the [x] pattern at the same time. All patterns besides the ones I listed can only be matched by truthy values so the work-around would only be needed for those dubious patterns.
Do you count pathological examples? class FakeEmpty(list): def __bool__(self): return False borked = FakeEmpty(("zero",)) match borked: case [x]: if borked: print("It's iterable.", x) else: print("Wut") case _: print("It's not [x].") "No possible value" ought to exclude anything that can be created with a couple of lines of Python like this. I could accept "no non-pathological values" for the [x] case, but are there any match types that could unexpectedly match a falsy value in normal circumstances? I'm *really* not a fan of having to put "if (....) is not None" just to work around a rare possibility, but I'm even less a fan of lurking bugs waiting to strike (think about how datetime.time() used to be false for midnight). Boilerplate to dodge bugs is C's job, not Python's. ChrisA
Yeah you can technically craft such pathological edge cases but this is already heavily discouraged. Libraries that change the usual semantics of python's object model are rare. The only exception I can think of would be numpy which disallows truthiness checks because of the ambiguity of arrays being both scalars and containers. However, this means that the "if x := get_some_array():" construct is already ill-formed so nothing would change by generalizing the left side to arbitrary patterns. And since numpy arrays aren't proper Sequence types you can't use them with the current implementation of pattern matching anyway. So besides carefully crafted pathological edge-cases the "if (...) is not None" construct would be completely redundant.
On Mon, 9 May 2022 at 01:07, MRAB <python@mrabarnett.plus.com> wrote:
It doesn't actually even need to be a different operator. There is currently no semantic meaning for "pattern := expr" save in the strict case where the pattern is a simple name, so semantics can be granted according to what's most useful. ChrisA
On Sun, 8 May 2022 at 19:38, Ethan Furman <ethan@stoneleaf.us> wrote:
And what is wrong with cost_for_all_pizzas = 0 for food in menu: match food: case {"type": "pizza", "price": price}: cost_for_all_pizzas += price Seriously, people are just getting used to match statements (I had to look the syntax up for the above) so IMO it's a bit early to be trying to cram them into dubious one-liners. Paul.
On Mon, 9 May 2022 at 07:36, Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:
The restrictions were never technical. They were always social - too many people were scared that assignment expressions would lead to a plethora of problems. (Which is not unreasonable. Guido made assignment a statement very deliberately, back when Python was first created. Changing that decision, or even modifying it slightly, should indeed be thoroughly thought out.) Requiring that the target be a simple name only was one way to minimize the risk. ChrisA
On Mon, 9 May 2022 at 08:35, Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:
I've always thought of relaxing the restrictions. It's actually easy to implement (even including attribute assignment/subscript assignment) and I had to modify only two files (plus regenerate the parser).
I think attribute assignment needs care to be mixed with structural pattern matching, since obj.attr can also be considered as a value pattern. Note attribute assignment with the walrus operator is not absolutely necessary since the purpose can also be achieved with a version of the function setattr which returns not None but the set value. Best regards, Takuo Matsuoka
participants (9)
-
Chris Angelico -
Ethan Furman -
Jeremiah Vivian -
Matsuoka Takuo -
MRAB -
Paul Moore -
Rob Cliffe -
Steven D'Aprano -
Valentin Berlier