Dict unpacking assignment
I would like to propose dict (or mapping) unpacking assignment. This is inspired in part by the Python-Ideas thread "f-strings as assignment targets", dict unpacking in function calls, and iterable unpacking assignment. Comments welcome. Background ---------- Iterable unpacking assignment: values = (1, 2, 3) a, b, c = *values is a very successful and powerful technique in Python. Likewise dict unpacking in function calls and dict displays: kwargs = {'a': 1, 'b': 2} func(**kwargs) d = {**kwargs, 'c': 3} There have been various requests for allowing dict unpacking on the right-hand side of an assignment [citation required] but in my opinion none of them have had a good justification. Motivated by the idea of scanning text strings with a scanf-style function, I propose the following behaviour for dict unpacking assignment: items = {'eggs': 2, 'cheese': 3, 'spam': 1} spam, eggs, cheese = **items assert spam == 1 and eggs == 2 and cheese == 3 Syntax ------ target_list [, **target] = **expression `target_list` is a comma-separated list of targets. Targets may be: - simple names, e.g. `spam` and `eggs` - dotted names, e.g. `spam.eggs` - numbered subscripts, e.g. `spam[1]` but is not required to support arbitrary complex targets such as: spam(*args).eggs(2*x + y)[1].cheese # not supported Likewise only int literals are supported for subscripts. (These restrictions may be lifted.) This is similar to the limited range of fields acceptabled by the string format mini-language. The same restrictions apply to `**target`. Each target must be unique. `expression` must evaluate to a dict or other mapping. Assignment proceeds by matching up targets from the left to keys on the right: 1. Every target must be matched exactly by a key. If there is a target without a corresponding key, that is an error. 2. Any key which does not match up to a target is an error, unless a `**target` is given. 3. If `**target` is given, it will collect any excess key:value pairs remaining into a dict. 4. If the targets and keys match up, then the bindings are applied from left to right, binding the target to the value associated with that key. Examples: # Targets are not unique a, b, a = **items => SyntaxError # Too many targets a, b, c = **{'a': 1, 'b': 2} => raises a runtime exception # Too few targets a = **{'a': 1, 'b': 2} => raises a runtime exception a, **extras = **{'a': 1, 'b': 2} assert a == 1 assert extras == {'b': 2} # Equal targets and keys a, b, **extras = **{'a': 1, 'b': 2} assert a == 1 assert b == 2 assert extras == {} # Dotted names from types import SimpleNamespace obj = SimpleNamespace() obj.spam = **{'obj.spam': 1} assert obj.spam == 1 # Subscripts arr = [None]*5 arr[1], arr[3] = **{'arr[3]': 33, 'arr[1]': 11} assert arr == [None, 11, None, 33, None] Assignments to dotted names or subscripts may fail, in which case the assignment may only partially succeed: spam = 'something' eggs = None spam, eggs.attr = {'spam': 1, 'eggs.attr': 2} # raises AttributeError: 'NoneType' object has no attribute 'attr' # but `spam` may have already been bound to 1 (I think that this is undesirable but unavoidable.) Motivating use-cases -------------------- The motivation comes from the discussion for scanf-like functionality. The addition of dict unpacking assignment would allow something like this: pattern = "I'll have {main} and {extra} with {colour} coffee." string = "I'll have spam and eggs with black coffee." main, extra, colour = **scanf(pattern, string) assert main == 'spam' assert extra == 'eggs' assert colour == 'black' But the possibilities are not restricted to string scanning. This will allow functions that return multiple values to choose between returning them by position or by name: height, width = get_dimensions(window) # returns a tuple height, width = **get_dimensions(window) # returns a mapping Developers can choose whichever model best suits their API. Another use-case is dealing with kwargs inside functions and methods: def method(self, **kwargs): spam, eggs, **kw = **kwargs process(spam, eggs) super().method(**kw) -- Steve
Oh, that's quite different than mapping patterns in PEP 634. :-( On Thu, Oct 22, 2020 at 8:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Thu, Oct 22, 2020 at 09:11:57PM -0700, Guido van Rossum wrote:
Oh, that's quite different than mapping patterns in PEP 634. :-(
That wasn't intentional, and to be honest I hadn't noticed the mapping patterns in 634 as yet. (There's a lot in that PEP.) Having read that now, I think I see the differences: (1) PEP 634 pattern matches on both the key and/or the value, where keys must be literals: case {"sleep": duration}: # From PEP 636, the tutorial. matches on a key "sleep" and captures the value in `duration`. My suggestion uses the key as an assignment target, and captures the value. Bringing them into alignment: 'sleep': duration = **items would look up key "sleep" and bind it to variable "duration". (2) PEP 634 ignores extra keys by default. My suggestion doesn't. (3) PEP 634 mandates that key:value pairs are looked up with the `get` method. I didn't specify the method, but I expected it would probably involves `keys()` and `__getitem__`. (4) PEP 634 declares that duplicate keys would raise ValueError (at runtime?). My suggestion would raise SyntaxError at compile-time. I gave this suggestion earlier: pattern = "I'll have {main} and {extra} with {colour} coffee." string = "I'll have spam and eggs with black coffee." main, extra, colour = **scanf(pattern, string) The assumption is that scanf would return a dict: {'main': 'spam', 'extra': 'eggs', 'colour': 'black'} Using match...case syntax, we would write: # correct me if I have this wrong match scanf(pattern, string): case {'main': main, 'extra': extra, 'colour': colour}: ... To bring my suggestion into alignment with PEP 634, I could write: {'main': main, 'extra': extra, 'colour': colour} = **scanf(pattern, string) My other example was: def method(self, **kwargs): spam, eggs, **kw = **kwargs process(spam, eggs) super().method(**kw) Using match...case syntax, we would write: def method(self, **kwargs): match kwargs: case {'spam': spam, 'eggs': eggs, **kw}: process(spam, eggs) super().method(**kw) Using PEP 634 syntax, I could write: def method(self, **kwargs): {'spam': spam, 'eggs': eggs, **kw} = **kwargs process(spam, eggs) super().method(**kw) -- Steve
On Fri, 23 Oct 2020 at 10:18, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
Comparing it to the pattern matching version Steven showed just before this in his post, I don't see enough additional benefit over the pattern matching version to justify needing a second way to do it. So I'd say this is covered just fine by pattern matching, and there's no need for a dict unpacking syntax as well. (Yes, I know we'll have list patterns and list unpacking, and I don't advocate removing list unpacking because there's a pattern matching variant. But we've lived without dict unpacking for a long time, so unlike lists I think the pattern matching version is enough). Paul
On Fri, Oct 23, 2020 at 5:27 AM Steven D'Aprano <steve@pearwood.info> wrote:
Your proposed syntax seems to rest on being similar to this syntax for iterable unpacking. But that asterisk isn't valid syntax, so I'm confused. This is valid syntax: a, b, c, *rest = values but that doesn't make it make sense to write `... = **values` as you suggest. And this is valid: a, b, c = [*values] but that asterisk has nothing to do with assignment.
On Fri, Oct 23, 2020 at 10:19:07AM +0200, Alex Hall wrote:
Oops, you are absolutely right, I confabulated that from the similar unpacking that does work: py> [1, 2, *"abc", 3] [1, 2, 'a', 'b', 'c', 3] and in function calls. Sorry for the confusion. I swear, no matter how well I know Python, the moment I don't test something in the REPL, I'm sure to get it wrong *wink*
but that doesn't make it make sense to write `... = **values` as you suggest.
Iterator unpacking on a dict already works: py> d = {'a': 10, 'b': 20} py> spam, eggs = d py> spam, eggs ('a', 'b') so we need to distinguish the iterator unpacking case from the dict unpacking case. To me it makes sense to use the same double star used in dict unpacking inside dict displays and function calls. -- Steve
On Fri, Oct 23, 2020 at 11:10 AM Steven D'Aprano <steve@pearwood.info> wrote:
I understand that, I just don't think this particular method of distinguishing is sufficiently justified. (Heretical question: do we *really* need to distinguish it in syntax? Iterator unpacking a dict seems like a dumb idea, I wouldn't be sad if we broke compatibility there) To me it makes sense to use the same double star used in
dict unpacking inside dict displays and function calls.
It makes some sense, but overall it's still quite different to anything existing. Typically the mechanics of assignment are defined by symbols that come before the =. This applies to iterable unpacking, setting attributes and mapping items, and augmented assignment. Everything after = just a normal expression. The most obvious syntax is to just assign to a dict display: {'spam': spam, 'eggs': eggs, **kw} = kwargs # not **kwargs The meaning seems intuitive and obvious at a glance. And it's flexible if the variable names don't always match the keys. But it's verbose and repetitive in the common case where the names match. I think it would be great if we had a syntax for abbreviating normal dicts with 'same name' keys. We discussed a lot of options earlier this year, e.g: {**, spam, eggs} {:spam, :eggs} {{spam, eggs}} Then using the same syntax in both dict unpacking and dict displays as expressions would be intuitive and obvious. This would be valid, although redundant: {**, spam, eggs} = {**, spam, eggs} and it would still be easy to add cases where names don't match: {'sleep': duration, **, spam, eggs} = kwargs Also, unpacking nested dicts follows naturally, whether we have an abbreviated syntax or not: {'spam': {'eggs': eggs, 'foo': foo}, 'bar': bar} = {'spam': {'eggs': eggs, 'foo': foo}, 'bar': bar} As does unpacking in a loop: for {**, spam, eggs} in list_of_dicts: whereas I'm not sure what would be done in your proposal. Something like this? for spam, eggs in **list_of_dicts:
Fwiw, although I see how PEP 634 has the potential to be incredibly powerful and I'm not opposed to it, I've tried to read and understand it twice and it is so overwhelming I still find the syntax inscrutable (I'm sure I'll get it eventually). I think Steven's proposed idea has a lot of merit even in a post PEP 634 universe because the syntax is just so easy to understand right away. At least for me. Pattern matching seems like it will fall under a learn python course section titled "advanced python topics". Dict unpacking is more of a moderate python topic. Additionally, Alex Hall's suggestions for how to use the various proposals for shortened dict display syntax, bringing Steven's proposal in line with pattern matching syntax but not having to repeat key names, is a really nice reconciliation of the too syntax ideas. And it would also allow something like this: {spam, eggs, 'def': def_, 'class': class_, **kwargs} = kwargs ...fixing the name collision problem. On Fri, Oct 23, 2020, 5:41 AM Alex Hall <alex.mojaki@gmail.com> wrote:
On Fri, Oct 23, 2020 at 5:43 PM David Mertz <mertz@gnosis.cx> wrote:
What I mean is that maybe `a, b = foo` could first test if `foo` is a mapping, use mapping unpacking if it is, otherwise fall back to regular iterable unpacking. So dicts would remain iterable, but that wouldn't feature in unpacking. Then there wouldn't be a change in syntax, or in other kinds of iteration, just a change in what happens if you unpack a dict.
For what it's worth, I was just writing the following today, and Steven's proposal came to mind. If we ignore the set of questionable (but realistic) decisions that led to me having to do the following, and the discussion on how the unpacking syntax would look: ---[ wot I wrote ] --- if request.method == 'POST': results = {} for item in request.json: thing = item['thing'] if item['kind'] == 'ignore': results[thing] = ('ignore', None) else: results[thing] = (item['kind'], item['alternate']) --- [dict unpacking version] --- if request.method == 'POST': results = {} for item in request.json: thing, kind, alternate = **item results[thing] = (kind_, None if kind == 'ignore' else alternate) I'm not using an inline-if expression in the first example, but the hypothetical second one seems simple enough to support it. Note: One thing that slightly bothers me about this is the potential for naming collisions with built-ins (I.e, I nearly used 'type' instead of 'kind' int the above data generated from javascript) Steve On Fri, Oct 23, 2020 at 9:20 AM Alex Hall <alex.mojaki@gmail.com> wrote:
I think that instead of dict unpacking specifically, what we need is to come up with a way to use the pattern-matching proposed in PEP 634 outside of match statements. This would make it possible to unpack any pattern. My opinion is that the walrus operator is practically waiting to support pattern-matching: if Response(status=200, json={"title": title}) := get('/posts/42'): print(title) I wrote a few more examples here: - https://mail.python.org/archives/list/python-ideas@python.org/thread/MJ7JHYK... -- Valentin
I really like this style of programming, especially after fooling around with Rust a lot, where pattern matching and its variations, like "if let" are working really well. I also like the idea of using the walrus operator in this use case, while it is probably up for a lot of discussing if the walrus operator should/can be reimplemented like this. With this pattern matching movement (PEP 622, 634, 636) this idea fits right in. A downside IMO is the readability. In the patter matching proposed in PEP 636, we introduce verbosity, while preserving readability. Every case is clearly readable. In your example I had to look twice that the 'Response' is the pattern which is matched from the 'get' result. But in the end this argument also applies to the already implemented walrus operator, so there is that. Patrick
Steven D'Aprano wrote:
Currently in Python `arr[1]` is the same as `arr[ 1 ]` (notice the added spaces). How is it taken into account in you proposal, does one match and the other doesn't ? Are those line equivalent or not : arr[1], arr[3] = **{'arr[3]': 33, 'arr[1]': 11} arr[ 1 ], arr[ 3 ] = **{'arr[3]': 33, 'arr[1]': 11} arr[1], arr[3] = **{'arr[ 3 ]': 33, 'arr[ 1 ]': 11} If not that would mean that the style of writing changes the execution of the program, which was never the case before AFAIK. Joseph
What if the mapping assignment were more harmonious with the pattern matching PEP? Something like this: items = {'eggs': 2, 'cheese': 3, 'spam': 1} {'eggs': eggs, 'spam': i_dont_need_to_name_this_spam, **rest} = items assert i_dont_need_to_name_this_spam == 1 assert eggs == 2 and cheese == 3 assert rest == {'cheese': 3} The keys here could be arbitrary hashables and the "values" could be arbitrary assignment targets (assigned all-or-nothing). This wouldn't need the right-hand-side double-star, and I think it more closely resembles the sequence unpacking assignment syntax. You can assign to a (thing that looks like a) tuple or to a (thing that looks like a) list or to a sequence subscript or object attribute, why not be able to assign to a (thing that looks like a) dictionary? This also avoids polluting the local namespace in case one of your keys is the string "range" or something. It also feels less magical to me, albeit more verbose. Calls to a hypothetical parse/sscanf function could closely mirror some str.format() calls: text = "{a}, {b}, {c}".format(**{'a': a0, 'b': b0, 'c': c0}) {'a': a1, 'b': b1, 'c': c1} = "{a}, {b}, {c}".parse(text) assert (a1, b1, c1) == (a0, b0, c0) Alternative positional parsing would be useful as well, as in: text = "{}, {}, {}".format(a0, b0, c0) a1, b1, c1 = "{}, {}, {}".parse(text) assert (a1, b1, c1) == (a0, b0, c0) This way, pattern.format() and pattern.parse() would be trying to be inverses of each other (as much as is reasonable, probably limited to parsing strings, floats and ints). Then maybe people could get used to a format-string-like mini-language for parsing, and eventually, the f-string assignment might be better received, and we could propose something like text = f"{a0}, {b0}, {c0}" f"{a1}, {b1}, {c1}" = text assert (a1, b1, c1) == (a0, b0, c0) as well, where we lose some of the flexibility but gain better D.R.Y. and more visual locality, useful in the simple cases. I see the potential for a strong analogy: positional format() : keyword-based format() : fstrings :: positional parse() : keyword-based parse(): assignment to fstrings
Nice! On Sun, Oct 25, 2020 at 9:59 PM Dennis Sweeney <sweeney.dennis650@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Tue, Oct 27, 2020 at 5:01 AM Dennis Sweeney <sweeney.dennis650@gmail.com> wrote:
They would unambiguously be strings (there's nothing else that that pattern could logically imply), so the only consideration is whether the input has the commas, which is a normal concern of all parsing. ChrisA
On Mon, Oct 26, 2020 at 04:56:24AM -0000, Dennis Sweeney wrote:
I see Guido likes this. I guess I could learn to live with it, but it seems a bit verbose to my taste. On the other hand, it does make the capture target independent of the key, and supports arbitrary keys: items = {'key': 'A', None: 'B', 5: 'C'} {5: spam, None: eggs, 'key': aardvark} = items assert spam == 'C' assert eggs == 'B' assert aardvark == 'A' So I think on balance I would give this a +1.
The keys here could be arbitrary hashables and the "values" could be arbitrary assignment targets (assigned all-or-nothing).
"All or nothing" is a stronger promise than iterable unpacking provides. py> a = [None]*5 py> a[0], a[1], a[9999], a[2] = "abcd" Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: list assignment index out of range py> a ['a', 'b', None, None, None] I think we can promise that: * if the mapping on the right has missing keys, no assignments occur; * if the mapping on the right has extra keys, and there is no double star target to capture the extras, then no assignments occur; * otherwise, assignments occur from left to right. -- Steve
Why must I always handle all keys in the dictionary? I can think of cases where I want to pull more than 1 key out of a dictionary but do not care about the rest. I have a dict containing config items. I just need a couple of the keys not all of them. I have a JSON response and the code is only interested in some of the values returned. For the pattern of passing **kwds down the __init__ chain I would clearly want to use the form: {'myitem': self.myitem, **other_kwds} = kwds
* otherwise, assignments occur from left to right.
Barry
On Sat, Oct 31, 2020 at 08:24:04AM +0000, Barry Scott wrote:
You don't. Use a double-star target to capture the excess, then ignore it. This is similar to sequence unpacking: spam, eggs, *who_cares = sequence {'spam': spam, 'eggs', eggs, **who_cares} = mapping will collect any excess items into `who_cares`. In the first case, it will be a list; in the second, it will be a dict. Barry:
{'myitem': self.myitem, **other_kwds} = kwds
Indeed. -- Steve
Steven D'Aprano writes:
{'spam': spam, 'eggs', eggs, **who_cares} = mapping
Shouldn't that be {'spam': spam, 'eggs', eggs, **_} = mapping 8-D Actually, I kinda like that, it looks like the side-eye emoji! Or a flounder, but that's a different kettle of fish. -- Yet Another Steve
On Sat, Oct 31, 2020 at 8:46 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Hm, for PEP 622/634 we looked at this and ended up making it so that this is the default -- you only have to write ``` {'spam': spam, 'eggs': eggs} = mapping ``` and any extra keys are ignored. This is because for the common use case here we want to ignore extra keys, not insist there aren't any. (We wrote it up a little better in the Mapping Patterns section of PEP 635.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sat, Oct 31, 2020 at 09:05:43PM -0700, Guido van Rossum wrote:
Regardless of which is commoner than the other, what about the use-case where you do want to insist that the mapping is an exact match? Having matched the two keys how do I say that there are no more keys? In the case of the `match` statement, I think that "ignore extra keys" by default is risky. Consider something like this: match mapping: case {'spam': spam}: print(spam) case {'spam': spam, 'eggs': eggs}: print('this will never be called') There's no case that will match the second that isn't already captured by the first. -- Steve
On Sat, Oct 31, 2020 at 21:47 Steven D'Aprano <steve@pearwood.info> wrote:
You add ‘**rest’ and a guard ‘if not rest’.
In the case of the `match` statement, I think that "ignore extra keys"
That falls under the general heading of “put more specific cases first”, so you should swap the cases. (Similar rules apply to class patterns and Are familiar from except clauses.)
-- --Guido (mobile)
I do not see why you would force the who_cares dict to be created when the only thing that my use case will do with it is delete it. I'd like to think that python had the freedom to optimise this construct and forcing the creating of who_cares seems like it would limit optimisation choices. I do not think being the same as sequences is required for dict.
Barry
On Sat, Oct 24, 2020 at 09:26:23PM -0000, Joseph Martinot-Lagarde wrote:
Good question! On the left hand side of the assignment, the target follows the normal Python rules so spaces within the subscript disappear when the code is compiled: >>> dis.dis('arr[1]') 1 0 LOAD_NAME 0 (arr) 2 LOAD_CONST 0 (1) 4 BINARY_SUBSCR 6 RETURN_VALUE >>> dis.dis('arr[ 1 \t ]') 1 0 LOAD_NAME 0 (arr) 2 LOAD_CONST 0 (1) 4 BINARY_SUBSCR 6 RETURN_VALUE On the right hand side of the assignment, the situation is a little more complicated because the keys are strings. The key matching would have to use the same rules as the interpreter, so that all of these keys would be treated identically: 'arr[1]' 'arr[ 1 ]' 'arr [ 1 ]' 'arr[(1)]' # but not 'arr[(1,)]' as that would be a tuple subscript etc. This adds some complexity to the process, not quite as simple as a naive string comparison, but shouldn't be too difficult. -- Steve
Oh, that's quite different than mapping patterns in PEP 634. :-( On Thu, Oct 22, 2020 at 8:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Thu, Oct 22, 2020 at 09:11:57PM -0700, Guido van Rossum wrote:
Oh, that's quite different than mapping patterns in PEP 634. :-(
That wasn't intentional, and to be honest I hadn't noticed the mapping patterns in 634 as yet. (There's a lot in that PEP.) Having read that now, I think I see the differences: (1) PEP 634 pattern matches on both the key and/or the value, where keys must be literals: case {"sleep": duration}: # From PEP 636, the tutorial. matches on a key "sleep" and captures the value in `duration`. My suggestion uses the key as an assignment target, and captures the value. Bringing them into alignment: 'sleep': duration = **items would look up key "sleep" and bind it to variable "duration". (2) PEP 634 ignores extra keys by default. My suggestion doesn't. (3) PEP 634 mandates that key:value pairs are looked up with the `get` method. I didn't specify the method, but I expected it would probably involves `keys()` and `__getitem__`. (4) PEP 634 declares that duplicate keys would raise ValueError (at runtime?). My suggestion would raise SyntaxError at compile-time. I gave this suggestion earlier: pattern = "I'll have {main} and {extra} with {colour} coffee." string = "I'll have spam and eggs with black coffee." main, extra, colour = **scanf(pattern, string) The assumption is that scanf would return a dict: {'main': 'spam', 'extra': 'eggs', 'colour': 'black'} Using match...case syntax, we would write: # correct me if I have this wrong match scanf(pattern, string): case {'main': main, 'extra': extra, 'colour': colour}: ... To bring my suggestion into alignment with PEP 634, I could write: {'main': main, 'extra': extra, 'colour': colour} = **scanf(pattern, string) My other example was: def method(self, **kwargs): spam, eggs, **kw = **kwargs process(spam, eggs) super().method(**kw) Using match...case syntax, we would write: def method(self, **kwargs): match kwargs: case {'spam': spam, 'eggs': eggs, **kw}: process(spam, eggs) super().method(**kw) Using PEP 634 syntax, I could write: def method(self, **kwargs): {'spam': spam, 'eggs': eggs, **kw} = **kwargs process(spam, eggs) super().method(**kw) -- Steve
On Fri, 23 Oct 2020 at 10:18, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
Comparing it to the pattern matching version Steven showed just before this in his post, I don't see enough additional benefit over the pattern matching version to justify needing a second way to do it. So I'd say this is covered just fine by pattern matching, and there's no need for a dict unpacking syntax as well. (Yes, I know we'll have list patterns and list unpacking, and I don't advocate removing list unpacking because there's a pattern matching variant. But we've lived without dict unpacking for a long time, so unlike lists I think the pattern matching version is enough). Paul
On Fri, Oct 23, 2020 at 5:27 AM Steven D'Aprano <steve@pearwood.info> wrote:
Your proposed syntax seems to rest on being similar to this syntax for iterable unpacking. But that asterisk isn't valid syntax, so I'm confused. This is valid syntax: a, b, c, *rest = values but that doesn't make it make sense to write `... = **values` as you suggest. And this is valid: a, b, c = [*values] but that asterisk has nothing to do with assignment.
On Fri, Oct 23, 2020 at 10:19:07AM +0200, Alex Hall wrote:
Oops, you are absolutely right, I confabulated that from the similar unpacking that does work: py> [1, 2, *"abc", 3] [1, 2, 'a', 'b', 'c', 3] and in function calls. Sorry for the confusion. I swear, no matter how well I know Python, the moment I don't test something in the REPL, I'm sure to get it wrong *wink*
but that doesn't make it make sense to write `... = **values` as you suggest.
Iterator unpacking on a dict already works: py> d = {'a': 10, 'b': 20} py> spam, eggs = d py> spam, eggs ('a', 'b') so we need to distinguish the iterator unpacking case from the dict unpacking case. To me it makes sense to use the same double star used in dict unpacking inside dict displays and function calls. -- Steve
On Fri, Oct 23, 2020 at 11:10 AM Steven D'Aprano <steve@pearwood.info> wrote:
I understand that, I just don't think this particular method of distinguishing is sufficiently justified. (Heretical question: do we *really* need to distinguish it in syntax? Iterator unpacking a dict seems like a dumb idea, I wouldn't be sad if we broke compatibility there) To me it makes sense to use the same double star used in
dict unpacking inside dict displays and function calls.
It makes some sense, but overall it's still quite different to anything existing. Typically the mechanics of assignment are defined by symbols that come before the =. This applies to iterable unpacking, setting attributes and mapping items, and augmented assignment. Everything after = just a normal expression. The most obvious syntax is to just assign to a dict display: {'spam': spam, 'eggs': eggs, **kw} = kwargs # not **kwargs The meaning seems intuitive and obvious at a glance. And it's flexible if the variable names don't always match the keys. But it's verbose and repetitive in the common case where the names match. I think it would be great if we had a syntax for abbreviating normal dicts with 'same name' keys. We discussed a lot of options earlier this year, e.g: {**, spam, eggs} {:spam, :eggs} {{spam, eggs}} Then using the same syntax in both dict unpacking and dict displays as expressions would be intuitive and obvious. This would be valid, although redundant: {**, spam, eggs} = {**, spam, eggs} and it would still be easy to add cases where names don't match: {'sleep': duration, **, spam, eggs} = kwargs Also, unpacking nested dicts follows naturally, whether we have an abbreviated syntax or not: {'spam': {'eggs': eggs, 'foo': foo}, 'bar': bar} = {'spam': {'eggs': eggs, 'foo': foo}, 'bar': bar} As does unpacking in a loop: for {**, spam, eggs} in list_of_dicts: whereas I'm not sure what would be done in your proposal. Something like this? for spam, eggs in **list_of_dicts:
Fwiw, although I see how PEP 634 has the potential to be incredibly powerful and I'm not opposed to it, I've tried to read and understand it twice and it is so overwhelming I still find the syntax inscrutable (I'm sure I'll get it eventually). I think Steven's proposed idea has a lot of merit even in a post PEP 634 universe because the syntax is just so easy to understand right away. At least for me. Pattern matching seems like it will fall under a learn python course section titled "advanced python topics". Dict unpacking is more of a moderate python topic. Additionally, Alex Hall's suggestions for how to use the various proposals for shortened dict display syntax, bringing Steven's proposal in line with pattern matching syntax but not having to repeat key names, is a really nice reconciliation of the too syntax ideas. And it would also allow something like this: {spam, eggs, 'def': def_, 'class': class_, **kwargs} = kwargs ...fixing the name collision problem. On Fri, Oct 23, 2020, 5:41 AM Alex Hall <alex.mojaki@gmail.com> wrote:
On Fri, Oct 23, 2020 at 5:43 PM David Mertz <mertz@gnosis.cx> wrote:
What I mean is that maybe `a, b = foo` could first test if `foo` is a mapping, use mapping unpacking if it is, otherwise fall back to regular iterable unpacking. So dicts would remain iterable, but that wouldn't feature in unpacking. Then there wouldn't be a change in syntax, or in other kinds of iteration, just a change in what happens if you unpack a dict.
For what it's worth, I was just writing the following today, and Steven's proposal came to mind. If we ignore the set of questionable (but realistic) decisions that led to me having to do the following, and the discussion on how the unpacking syntax would look: ---[ wot I wrote ] --- if request.method == 'POST': results = {} for item in request.json: thing = item['thing'] if item['kind'] == 'ignore': results[thing] = ('ignore', None) else: results[thing] = (item['kind'], item['alternate']) --- [dict unpacking version] --- if request.method == 'POST': results = {} for item in request.json: thing, kind, alternate = **item results[thing] = (kind_, None if kind == 'ignore' else alternate) I'm not using an inline-if expression in the first example, but the hypothetical second one seems simple enough to support it. Note: One thing that slightly bothers me about this is the potential for naming collisions with built-ins (I.e, I nearly used 'type' instead of 'kind' int the above data generated from javascript) Steve On Fri, Oct 23, 2020 at 9:20 AM Alex Hall <alex.mojaki@gmail.com> wrote:
I think that instead of dict unpacking specifically, what we need is to come up with a way to use the pattern-matching proposed in PEP 634 outside of match statements. This would make it possible to unpack any pattern. My opinion is that the walrus operator is practically waiting to support pattern-matching: if Response(status=200, json={"title": title}) := get('/posts/42'): print(title) I wrote a few more examples here: - https://mail.python.org/archives/list/python-ideas@python.org/thread/MJ7JHYK... -- Valentin
I really like this style of programming, especially after fooling around with Rust a lot, where pattern matching and its variations, like "if let" are working really well. I also like the idea of using the walrus operator in this use case, while it is probably up for a lot of discussing if the walrus operator should/can be reimplemented like this. With this pattern matching movement (PEP 622, 634, 636) this idea fits right in. A downside IMO is the readability. In the patter matching proposed in PEP 636, we introduce verbosity, while preserving readability. Every case is clearly readable. In your example I had to look twice that the 'Response' is the pattern which is matched from the 'get' result. But in the end this argument also applies to the already implemented walrus operator, so there is that. Patrick
Steven D'Aprano wrote:
Currently in Python `arr[1]` is the same as `arr[ 1 ]` (notice the added spaces). How is it taken into account in you proposal, does one match and the other doesn't ? Are those line equivalent or not : arr[1], arr[3] = **{'arr[3]': 33, 'arr[1]': 11} arr[ 1 ], arr[ 3 ] = **{'arr[3]': 33, 'arr[1]': 11} arr[1], arr[3] = **{'arr[ 3 ]': 33, 'arr[ 1 ]': 11} If not that would mean that the style of writing changes the execution of the program, which was never the case before AFAIK. Joseph
What if the mapping assignment were more harmonious with the pattern matching PEP? Something like this: items = {'eggs': 2, 'cheese': 3, 'spam': 1} {'eggs': eggs, 'spam': i_dont_need_to_name_this_spam, **rest} = items assert i_dont_need_to_name_this_spam == 1 assert eggs == 2 and cheese == 3 assert rest == {'cheese': 3} The keys here could be arbitrary hashables and the "values" could be arbitrary assignment targets (assigned all-or-nothing). This wouldn't need the right-hand-side double-star, and I think it more closely resembles the sequence unpacking assignment syntax. You can assign to a (thing that looks like a) tuple or to a (thing that looks like a) list or to a sequence subscript or object attribute, why not be able to assign to a (thing that looks like a) dictionary? This also avoids polluting the local namespace in case one of your keys is the string "range" or something. It also feels less magical to me, albeit more verbose. Calls to a hypothetical parse/sscanf function could closely mirror some str.format() calls: text = "{a}, {b}, {c}".format(**{'a': a0, 'b': b0, 'c': c0}) {'a': a1, 'b': b1, 'c': c1} = "{a}, {b}, {c}".parse(text) assert (a1, b1, c1) == (a0, b0, c0) Alternative positional parsing would be useful as well, as in: text = "{}, {}, {}".format(a0, b0, c0) a1, b1, c1 = "{}, {}, {}".parse(text) assert (a1, b1, c1) == (a0, b0, c0) This way, pattern.format() and pattern.parse() would be trying to be inverses of each other (as much as is reasonable, probably limited to parsing strings, floats and ints). Then maybe people could get used to a format-string-like mini-language for parsing, and eventually, the f-string assignment might be better received, and we could propose something like text = f"{a0}, {b0}, {c0}" f"{a1}, {b1}, {c1}" = text assert (a1, b1, c1) == (a0, b0, c0) as well, where we lose some of the flexibility but gain better D.R.Y. and more visual locality, useful in the simple cases. I see the potential for a strong analogy: positional format() : keyword-based format() : fstrings :: positional parse() : keyword-based parse(): assignment to fstrings
Nice! On Sun, Oct 25, 2020 at 9:59 PM Dennis Sweeney <sweeney.dennis650@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Tue, Oct 27, 2020 at 5:01 AM Dennis Sweeney <sweeney.dennis650@gmail.com> wrote:
They would unambiguously be strings (there's nothing else that that pattern could logically imply), so the only consideration is whether the input has the commas, which is a normal concern of all parsing. ChrisA
On Mon, Oct 26, 2020 at 04:56:24AM -0000, Dennis Sweeney wrote:
I see Guido likes this. I guess I could learn to live with it, but it seems a bit verbose to my taste. On the other hand, it does make the capture target independent of the key, and supports arbitrary keys: items = {'key': 'A', None: 'B', 5: 'C'} {5: spam, None: eggs, 'key': aardvark} = items assert spam == 'C' assert eggs == 'B' assert aardvark == 'A' So I think on balance I would give this a +1.
The keys here could be arbitrary hashables and the "values" could be arbitrary assignment targets (assigned all-or-nothing).
"All or nothing" is a stronger promise than iterable unpacking provides. py> a = [None]*5 py> a[0], a[1], a[9999], a[2] = "abcd" Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: list assignment index out of range py> a ['a', 'b', None, None, None] I think we can promise that: * if the mapping on the right has missing keys, no assignments occur; * if the mapping on the right has extra keys, and there is no double star target to capture the extras, then no assignments occur; * otherwise, assignments occur from left to right. -- Steve
Why must I always handle all keys in the dictionary? I can think of cases where I want to pull more than 1 key out of a dictionary but do not care about the rest. I have a dict containing config items. I just need a couple of the keys not all of them. I have a JSON response and the code is only interested in some of the values returned. For the pattern of passing **kwds down the __init__ chain I would clearly want to use the form: {'myitem': self.myitem, **other_kwds} = kwds
* otherwise, assignments occur from left to right.
Barry
On Sat, Oct 31, 2020 at 08:24:04AM +0000, Barry Scott wrote:
You don't. Use a double-star target to capture the excess, then ignore it. This is similar to sequence unpacking: spam, eggs, *who_cares = sequence {'spam': spam, 'eggs', eggs, **who_cares} = mapping will collect any excess items into `who_cares`. In the first case, it will be a list; in the second, it will be a dict. Barry:
{'myitem': self.myitem, **other_kwds} = kwds
Indeed. -- Steve
Steven D'Aprano writes:
{'spam': spam, 'eggs', eggs, **who_cares} = mapping
Shouldn't that be {'spam': spam, 'eggs', eggs, **_} = mapping 8-D Actually, I kinda like that, it looks like the side-eye emoji! Or a flounder, but that's a different kettle of fish. -- Yet Another Steve
On Sat, Oct 31, 2020 at 8:46 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Hm, for PEP 622/634 we looked at this and ended up making it so that this is the default -- you only have to write ``` {'spam': spam, 'eggs': eggs} = mapping ``` and any extra keys are ignored. This is because for the common use case here we want to ignore extra keys, not insist there aren't any. (We wrote it up a little better in the Mapping Patterns section of PEP 635.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sat, Oct 31, 2020 at 09:05:43PM -0700, Guido van Rossum wrote:
Regardless of which is commoner than the other, what about the use-case where you do want to insist that the mapping is an exact match? Having matched the two keys how do I say that there are no more keys? In the case of the `match` statement, I think that "ignore extra keys" by default is risky. Consider something like this: match mapping: case {'spam': spam}: print(spam) case {'spam': spam, 'eggs': eggs}: print('this will never be called') There's no case that will match the second that isn't already captured by the first. -- Steve
On Sat, Oct 31, 2020 at 21:47 Steven D'Aprano <steve@pearwood.info> wrote:
You add ‘**rest’ and a guard ‘if not rest’.
In the case of the `match` statement, I think that "ignore extra keys"
That falls under the general heading of “put more specific cases first”, so you should swap the cases. (Similar rules apply to class patterns and Are familiar from except clauses.)
-- --Guido (mobile)
I do not see why you would force the who_cares dict to be created when the only thing that my use case will do with it is delete it. I'd like to think that python had the freedom to optimise this construct and forcing the creating of who_cares seems like it would limit optimisation choices. I do not think being the same as sequences is required for dict.
Barry
On Sat, Oct 24, 2020 at 09:26:23PM -0000, Joseph Martinot-Lagarde wrote:
Good question! On the left hand side of the assignment, the target follows the normal Python rules so spaces within the subscript disappear when the code is compiled: >>> dis.dis('arr[1]') 1 0 LOAD_NAME 0 (arr) 2 LOAD_CONST 0 (1) 4 BINARY_SUBSCR 6 RETURN_VALUE >>> dis.dis('arr[ 1 \t ]') 1 0 LOAD_NAME 0 (arr) 2 LOAD_CONST 0 (1) 4 BINARY_SUBSCR 6 RETURN_VALUE On the right hand side of the assignment, the situation is a little more complicated because the keys are strings. The key matching would have to use the same rules as the interpreter, so that all of these keys would be treated identically: 'arr[1]' 'arr[ 1 ]' 'arr [ 1 ]' 'arr[(1)]' # but not 'arr[(1,)]' as that would be a tuple subscript etc. This adds some complexity to the process, not quite as simple as a naive string comparison, but shouldn't be too difficult. -- Steve
participants (17)
-
Alex Hall -
Barry Scott -
Chris Angelico -
Daniel Moisset -
David Mertz -
Dennis Sweeney -
Guido van Rossum -
Joseph Martinot-Lagarde -
Marco Sulla -
MRAB -
patrickhaller40＠googlemail.com -
Paul Moore -
Ricky Teachey -
Stephen J. Turnbull -
Stestagg -
Steven D'Aprano -
Valentin Berlier