If the keys are not strings, it currently works in CPython, but it may not work with other implementations, or future versions of CPython[2].

I don't think so. https://bugs.python.org/issue35105 and https://mail.python.org/pipermail/python-dev/2018-October/155435.html are about kwargs. I think non string keys are allowed for {**d1, **d2} by language. -- INADA Naoki

Hi, Steven. I can help you with it. I added it as PEP 584. I had to add the PEP headers, but didn't do any other editing. I'm going to be out of town for the next 2 weeks, so I might be slow in responding. Eric On 3/1/2019 11:26 AM, Steven D'Aprano wrote:

Attached is a draft PEP on adding + and - operators to dict for discussion.

This should probably go here:

https://github.com/python/peps

but due to technical difficulties at my end, I'm very limited in what I can do on Github (at least for now). If there's anyone who would like to co-author and/or help with the process, that will be appreciated.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Thanks -- FYI I renamed the file to .rst (per convention for PEPs in ReST format) and folded long text lines. On Fri, Mar 1, 2019 at 8:53 AM Eric V. Smith wrote:

Hi, Steven.

I can help you with it. I added it as PEP 584. I had to add the PEP headers, but didn't do any other editing.

I'm going to be out of town for the next 2 weeks, so I might be slow in responding.

Eric

On 3/1/2019 11:26 AM, Steven D'Aprano wrote:

...

Attached is a draft PEP on adding + and - operators to dict for discussion.

This should probably go here:

https://github.com/python/peps

but due to technical difficulties at my end, I'm very limited in what I can do on Github (at least for now). If there's anyone who would like to co-author and/or help with the process, that will be appreciated.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

While working through my implementation, I've come across a couple of inconsistencies with the current proposal:

The merge operator will have the same relationship to the dict.update method as the list concatenation operator has to list.extend, with dict difference being defined analogously.

I like this premise. += for lists *behaves* like extend, and += for dicts *behaves* like update. However, later in the PEP it says:

Augmented assignment will just call the update method. This is analogous to the way list += calls the extend method, which accepts any iterable, not just lists.

In your Python implementation samples from the PEP, dict subclasses will behave differently from how list subclasses do. List subclasses, without overrides, return *list* objects for bare "+" operations (and "+=" won't call an overridden "extend" method). So a more analogous pseudo-implementation (if that's what we seek) would look like: def __add__(self, other): if isinstance(other, dict): new = dict.copy(self) dict.update(new, other) return new return NotImplemented def __radd__(self, other): if isinstance(other, dict): new = dict.copy(other) dict.update(other, self) return new return NotImplemented def __iadd__(self, other): if isinstance(other, dict): dict.update(self, other) return self return NotImplemented This is what my C looks like right now. We can choose to update these semantics to be "nicer" to subclasses, but I don't see any precedent for it (lists, sets, strings, etc.). Brandt On Fri, Mar 1, 2019 at 11:41 AM Brett Cannon wrote:

On Fri, Mar 1, 2019 at 8:50 AM Brandt Bucher wrote:

...

I’ve never been part of this process before, but I’m interested in learning and helping any way I can.

Thanks!

...

My addition implementation is attached to the bpo, and I’m working today on bringing it in line with the PEP in its current form (specifically, subtraction operations).

https://github.com/python/cpython/pull/12088

When your proposed patch is complete, Brandt, just ask Steven to update the PEP to mention that there's a proposed implementation attached to the issue tracking the idea.

-Brett

...

Brandt

On Mar 1, 2019, at 08:26, Steven D'Aprano wrote:

Attached is a draft PEP on adding + and - operators to dict for discussion.

This should probably go here:

https://github.com/python/peps

but due to technical difficulties at my end, I'm very limited in what I can do on Github (at least for now). If there's anyone who would like to co-author and/or help with the process, that will be appreciated.

-- Steven

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Executive summary: - I'm going to argue for subclass-preserving behaviour; - I'm not wedded to the idea that dict += should actually call the update method, so long as it has the same behaviour; - __iadd__ has no need to return NotImplemented or type-check its argument. Details below. On Fri, Mar 01, 2019 at 04:10:44PM -0800, Brandt Bucher wrote: [...]

In your Python implementation samples from the PEP, dict subclasses will behave differently from how list subclasses do. List subclasses, without overrides, return *list* objects for bare "+" operations

Right -- and I think they are wrong to do so, for reasons I explained here: https://mail.python.org/pipermail/python-ideas/2019-March/055547.html I think the standard handling of subclasses in Python builtins is wrong, and I don't wish to emulate that wrong behaviour without a really good reason. Or at least a better reason than "other methods break subclassing unless explicitly overloaded, so this should do so too". Or at least not without a fight :-)

(and "+=" won't call an overridden "extend" method).

I'm slightly less opinionated about that. Looking more closely into the docs, I see that they don't actually say that += calls list.extend: s.extend(t) extends s with the contents of t (for or s += t the most part the same as s[len(s):len(s)] = t) https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types only that they have the same effect. So the wording re lists calling extend certainly needs to be changed. But that doesn't mean that we must change the implementation. We have a choice: - regardless of what lists do, we define += for dicts as literally calling dict.update; the more I think about it, the less I like this. - Or we say that += behaves similarly to update, without actually calling the method. I think I prefer this. (The second implies either that += either contains a duplicate of the update logic, or that += and update both delegate to a private, C-level function that does most of the work.) I think that the second approach (define += as having the equivalent semantics of update but without actually calling the update method) is probably better. That decouples the two methods, allows subclasses to change one without necessarily changing the other.

So a more analogous pseudo-implementation (if that's what we seek) would look like:

def __add__(self, other): if isinstance(other, dict): new = dict.copy(self) dict.update(new, other) return new return NotImplemented

We should not require the copy method. The PEP should be more explicit that the approximate implementation does not imply the copy() and update() methods are actually called.

def __iadd__(self, other): if isinstance(other, dict): dict.update(self, other) return self return NotImplemented

I don't agree with that implementation. According to PEP 203, which introduced augmented assignment, the sequence of calls in ``d += e`` is: 1. Try to call ``d.__iadd__(e)``. 2. If __iadd__ is not present, try ``d.__add__(e)``. 3. If __add__ is missing too, try ``e.__radd__(d)``. but my tests suggest this is inaccurate. I think the correct behaviour is this: 1. Try to call ``d.__iadd__(e)``. 2. If __iadd__ is not present, or if it returns NotImplemented, try ``d.__add__(e)``. 3. If __add__ is missing too, or if it returns NotImplemented, fail with TypeError. In other words, e.__radd__ is not used. We don't want dict.__iadd__ to try calling __add__, since the later is more restrictive and less efficient than the in-place merge. So there is no need for __iadd__ to return NotImplemented. It should either succeed on its own, or fail hard: def __iadd__(self, other): self.update(other) return self Except that the actual C implementation won't call the update method itself, but will follow the same semantics. See the docstring for dict.update for details of what is accepted by update. -- Steven

I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown. This way, d1 + d2 isn’t just another obvious way to do {**d1, **d2}. The second syntax makes it clear that a new dictionary is being constructed and that d2 overrides keys from d1. One can reasonably expect or imagine a situation where a section of code that expects to merge two dictionaries with non-conflicting keys commits a semantic error if it merges two dictionaries with conflicting keys. To better explain, imagine a program where options is a global variable storing parsed values from the command line. def verbose_options(): if options.quiet return {'verbose': True} def quiet_options(): if options.quiet: return {'verbose': False} If we were to define an options() function, return {**quiet_options(), **verbose_options()} implies that verbose overrules quiet; whereas return quiet_options() + verbose_options() implies that verbose and quiet cannot be used simultaneously. I am not aware of another easy way in Python to merge dictionaries while checking for non-conflicting keys. Compare: def settings(): return {**quiet_options(), **verbose_options()} def settings(): try: return quiet_options() + verbose_options() except KeyError: print('conflicting options used', sys.stderr') sys.exit(1) *** This is a simple scenario, but you can imagine more complex ones as well. Does —quiet-stage-1 loosen —verbose? Does —quiet-stage-1 conflict with —verbose-stage-1?Does —verbosity=5 override —verbosity=4 or cause an error? Having {**, **} and + do different things provides a convenient and Pythonic way to model such relationships in code. Indeed, you can even combine the two syntaxes in the same expression to show a mix of overriding and exclusionary behavior. Anyways, I think it’s a good idea to have this semantic difference in behavior so Python developers have a good way to communicate what is expected of the two dictionaries being merged inside the language. This is like an assertion without Again, I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown, because such “non-conflicting merge” behavior would be useful in Python. It gives clarifying power to the + sign. The + and the {**, **} should serve different roles. In other words, explicit + is better than implicit {**, **#, unless explicitly suppressed. Here + is explicit whereas {**, **} is implicitly allowing inclusive keys, and the KeyError is expressed suppressed by virtue of not using the {**, **} syntax. People expect the + operator to be commutative, while the {**, **} syntax prompts further examination by virtue of its “weird” syntax.

James Lu schrieb am 04.03.19 um 03:28:

I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown.

Please, no. That would be really annoying. If you need that feature, it can become a new method on dicts. Stefan

Jimmy Girardet schrieb am 04.03.19 um 10:12:

I'm not old on this list but every time there is a proposal, the answer is "what are you trying to solve ?".

Since

|z ={**x,**y} and z.update(y) Exists, I can"t find the answer.

I think the main intentions is to close a gap in the language. [1,2,3] + [4,5,6] works for lists and tuples, {1,2,3} | {4,5,6} works for sets, but joining two dicts isn't simply {1:2, 3:4} + {5:6} but requires either some obscure syntax or a statement instead of a simple expression. The proposal is to enable the obvious syntax for something that should be obvious. Stefan

but requires either some obscure syntax or a statement instead of a simple expression.

The proposal is to enable the obvious syntax for something that should be obvious.

Stefan

The discussions on this list show that the behavior of `+` operator with dict will never be obvious (first wins or second wins or add results or raise Exception). So the user will always have to look at the doc or test it to know the intended behavior. That said, [1,2] + [3] equals  [1,2,3]  but not[1,2, [3]] and that was not obvious to me, and I survived.

INADA Naoki schrieb am 04.03.19 um 11:15:

Why statement is not enough?

I'm not sure I understand why you're asking this, but a statement is "not enough" because it's a statement and not an expression. It does not replace the convenience of an expression. Stefan

On Mar 4, 2019, at 10:02 AM, Stefan Behnel wrote:

INADA Naoki schrieb am 04.03.19 um 11:15:

...

Why statement is not enough?

I'm not sure I understand why you're asking this, but a statement is "not enough" because it's a statement and not an expression. It does not replace the convenience of an expression.

Stefan There is already an expression for key-overriding merge. Why do we need a new one?

On Mon, Mar 04, 2019 at 10:09:32AM -0500, James Lu wrote:

How many situations would you need to make a copy of a dictionary and then update that copy and override old keys from a new dictionary?

Very frequently. That's why we have a dict.update method, which if I remember correctly, was introduced in Python 1.5 because people were frequently re-inventing the same wheel: def update(d1, d2): for key in d2.keys(): d1[key] in d2[key] You should have a look at how many times it is used in the standard library: [steve@ando cpython]$ cd Lib/ [steve@ando Lib]$ grep -U "\.update[(]" *.py */*.py | wc -l 373 Now some of those are false positives (docstrings, comments, non-dicts, etc) but that still leaves a lot of examples of wanting to override old keys. This is a very common need. Wanting an exception if the key already exists is, as far as I can tell, very rare. It is true that many of the examples in the std lib involve updating an existing dict, not creating a new one. But that's only to be expected: since Python didn't provide an obvious functional version of update, only an in-place version, naturally people get used to writing in-place code. (Think about how long we made do without sorted(). I don't know about other people, but I now find sorted indispensible, and probably use it ten or twenty times more often than the in-place version.) [...]

The KeyError of my proposal is a feature, a sign that something is wrong, a sign an invariant is being violated.

Why is "keys are unique" an invariant? The PEP gives a good example of when this "invariant" would be unnecessarily restrictive: For example, updating default configuration values with user-supplied values would most often fail under the requirement that keys are unique:: prefs = site_defaults + user_defaults + document_prefs Another example would be when reading command line options, where the most common convention is for "last option seen" to win: [steve@ando Lib]$ grep --color=always --color=never "zero" f*.py fileinput.py: numbers are zero; nextfile() has no effect. fractions.py: # the same way for any finite a, so treat a as zero. functools.py: # prevent their ref counts from going to zero during and the output is printed without colour. (I've slightly edited the above output so it will fit in the email without wrapping.) The very name "update" should tell us that the most useful behaviour is the one the devs decided on back in 1.5: have the last seen value win. How can you update values if the operation raises an error if the key already exists? If this behaviour is ever useful, I would expect that it will be very rare. An update or merge is effectively just running through a loop setting the value of a key. See the pre-Python 1.5 function above. Having update raise an exception if the key already exists would be about as useful as having ``d[key] = value`` raise an exception if the key already exists. Unless someone can demonstrate that the design of dict.update() was a mistake, and the "require unique keys" behaviour is more common, then I maintain that for the very rare cases you want an exception, you can subclass dict and overload the __add__ method: # Intentionally simplified version. def __add__(self, other): if self.keys() & other.keys(): raise KeyError return super().__add__(self, other)

The ugliness of the syntax makes one pause and think and ask: “Why is it important that the keys from this dictionary override the ones from another dictionary?”

Because that is the most common and useful behaviour. That's what it means to *update* a dict or database, and this proposal is for an update operator. The ugliness of the existing syntax is not a feature, it is a barrier. -- Steven

On Mon, Mar 04, 2019 at 10:01:23AM -0500, James Lu wrote:

If you want to merge it without a KeyError, learn and use the more explicit {**d1, **d2} syntax.

On Mar 4, 2019, at 10:25 AM, Steven D'Aprano wrote:

In your previous email, you said the {**d ...} syntax was implicit:

In other words, explicit + is better than implicit {**, **#, unless explicitly suppressed. Here + is explicit whereas {**, **} is implicitly allowing inclusive keys, and the KeyError is expressed suppressed by virtue of not using the {**, **} syntax.

It is difficult to take your "explicit/implicit" argument seriously when you cannot even decided which is which. I misspoke. In your previous email, you said the {**d ...} syntax was implicit:

In other words, explicit + is better than implicit {**, **#, unless explicitly suppressed. Here + is explicit whereas {**, **} is implicitly allowing inclusive keys, and the KeyError is expressed suppressed by virtue of not using the {**, **} syntax.

It is difficult to take your "explicit/implicit" argument seriously when you cannot even decided which is which.

Yes, + is explicit. {**, **} is implicit. My argument: We should set the standard that + is for non-conflicting merge and {**, **} is for overriding merge. That standard should be so that + explicitly asserts that the keys will not conflict whereas {**d1, **d2} is ambiguous on why d2 is overriding d1.^ ^Presumably you’re making a copy of d1 so why should d3 have d2 take priority? The syntax deserves a comment, perhaps explaining that items from d2 are newer in time or that the items in d1 are always nonces. The + acts as an implicit assertion and an opportunity to catch an invariant violation or data input error. Give me an example of a situation where you need a third dictionary from two existing dictionaries and having conflict where a key has a different value in both is desirable behavior. The situation where non-conflicting merge is what’s desired is more common and in that case throwing an exception in the case of a conflicting value is a good thing, a way to catch code smell.

On Mon, Mar 4, 2019 at 3:31 PM Del Gan wrote:

...

the augmented assignment version allows anything the ``update`` method allows, such as iterables of key/value pairs

I am a little surprised by this choice.

First, this means that "a += b" would not be equivalent to "a = a + b". Is there other built-in types which act differently if called with the operator or augmented assignment version?

Yes. The same happens for lists. [1] + 'a' is a TypeError, but a += 'a' works:

...

...

a = [1] a + 'a' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can only concatenate list (not "str") to list a += 'a' a [1, 'a']

Secondly, that would imply I would no longer be able to infer the type of "a" while reading "a += [('foo', 'bar')]". Is it a list? A dict?

Real code more likely looks like "a += b" and there you already don't have much of a clue -- the author of the code should probably communicate this using naming conventions or type annotations.

Those two points make me uncomfortable with "+=" strictly behaving like ".update()".

And yet that's how it works for lists. (Note that dict.update() still has capabilities beyond +=, since you can also invoke it with keyword args.) -- --Guido van Rossum (python.org/~guido)

On Mar 4, 2019, at 11:25 AM, Steven D'Aprano wrote:

The PEP gives a good example of when this "invariant" would be unnecessarily restrictive:

For example, updating default configuration values with user-supplied values would most often fail under the requirement that keys are unique::

prefs = site_defaults + user_defaults + document_prefs

Another example would be when reading command line options, where the most common convention is for "last option seen" to win:

[steve@ando Lib]$ grep --color=always --color=never "zero" f*.py fileinput.py: numbers are zero; nextfile() has no effect. fractions.py: # the same way for any finite a, so treat a as zero. functools.py: # prevent their ref counts from going to zero during

Indeed, in this case you would want to use {**, **} syntax.

and the output is printed without colour.

(I've slightly edited the above output so it will fit in the email without wrapping.)

The very name "update" should tell us that the most useful behaviour is the one the devs decided on back in 1.5: have the last seen value win. How can you update values if the operation raises an error if the key already exists? If this behaviour is ever useful, I would expect that it will be very rare.

An update or merge is effectively just running through a loop setting the value of a key. See the pre-Python 1.5 function above. Having update raise an exception if the key already exists would be about as useful as having ``d[key] = value`` raise an exception if the key already exists.

Unless someone can demonstrate that the design of dict.update() was a mistake You’re making a logical mistake here. + isn’t supposed to have .update’s behavior and it never was supposed to.

, and the "require unique keys" behaviour is more common, I just have. 99% of the time you want to have keys from one dict override another, you’d be better off doing it in-place and so would be using .update() anyways.

then I maintain that for the very rare cases you want an exception, you can subclass dict and overload the __add__ method: Well, yes, the whole point is to define the best default behavior.

I have seen a ton of discussion about what dict addition should do, but have seen almost no mention of dict difference. This lack of discussion interest combined with me not recalling having needed the proposed subtraction semantics personally makes me wonder if we should hold off on locking in subtraction semantics just yet. Perhaps we could just scope the proposal to dictionary addition only for now? If I *were* to define dict difference, my intuition suggests supporting a second operand that is any iterable of keys and not just dicts. (Augmented dict subtraction is already proposed to accept such a broader second argument.) David Foster | Seattle, WA, USA On 3/1/19 8:26 AM, Steven D'Aprano wrote:

Attached is a draft PEP on adding + and - operators to dict for discussion.

This should probably go here:

https://github.com/python/peps

but due to technical difficulties at my end, I'm very limited in what I can do on Github (at least for now). If there's anyone who would like to co-author and/or help with the process, that will be appreciated.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Mar 5, 2019 at 12:02 AM Stefan Behnel wrote:

INADA Naoki schrieb am 04.03.19 um 11:15:

...

Why statement is not enough?

I'm not sure I understand why you're asking this, but a statement is "not enough" because it's a statement and not an expression. It does not replace the convenience of an expression.

Stefan

It seems tautology and say nothing. What is "convenience of an expression"? Is it needed to make Python more readable language? Anyway, If "there is expression" is the main reason for this proposal, symbolic operator is not necessary. `new = d1.updated(d2)` or `new = dict.merge(d1, d2)` are enough. Python preferred name over symbol in general. Symbols are readable and understandable only when it has good math metaphor. Sets has symbol operator because it is well known in set in math, not because set is frequently used. In case of dict, there is no simple metaphor in math. It just cryptic and hard to Google. -- INADA Naoki

On Sat, Mar 02, 2019 at 01:47:37AM +0900, INADA Naoki wrote:

...

If the keys are not strings, it currently works in CPython, but it may not work with other implementations, or future versions of CPython[2].

I don't think so. https://bugs.python.org/issue35105 and https://mail.python.org/pipermail/python-dev/2018-October/155435.html are about kwargs. I think non string keys are allowed for {**d1, **d2} by language.

Is this documented somewhere? Or is there a pronouncement somewhere that it is definitely expected to work in any language calling itself Python? Thanks, -- Steven

On Mar 4, 2019, at 4:51 AM, Stefan Behnel wrote:

I think the main intentions is to close a gap in the language.

[1,2,3] + [4,5,6]

works for lists and tuples,

{1,2,3} | {4,5,6}

works for sets, but joining two dicts isn't simply

{1:2, 3:4} + {5:6}

but requires either some obscure syntax or a statement instead of a simple expression.

The proposal is to enable the obvious syntax for something that should be obvious.

I would challenge that this dictionary merging is something that is obvious. The existing sequences are simple collections of values where a dictionary is a mapping of values. The difference between the two is akin to the difference between a mathematical array or set and a unary mapping function. There is a clear and obvious way to combine arrays and sets -- concatenation for arrays and union for sets. Combining mapping functions is less than obvious. "Putting Metaclasses to Work" (ISBN-13 978-0201433050) presents a more mathematical view of programming language types that includes two distinct operations for combining dictionaries -- merge and recursive merge. For two input dictionaries D1 & D2 and the output dictionary O D1 merge D2 O is D1 with the of those keys of D2 that do not have keys in D1 D1 recursive-merge D2 For all keys k, O[k] = D1[k] recursive merge D2[k] if both D1[k] and D2[k] are dictionaries, otherwise O[k] = (D1 merge D2)[k]. Note that neither of the cases is the same as:

...

...

O = D1.copy() O.update(D2)

So that gives us three different ways to combine dictionaries that are each sensible. The following example uses dictionaries from "Putting Metaclasses to Work":

...

...

d1 = { ... 'title': 'Structured Programming', ... 'authors': 'Dahl, Dijkstra, and Hoare', ... 'locations': { ... 'Dahl': 'University of Oslo', ... 'Dijkstra': 'University of Texas', ... 'Hoare': 'Oxford University', ... }, ... }

d2 = { ... 'publisher': 'Academic Press', ... 'locations': { ... 'North America': 'New York', ... 'Europe': 'London', ... }, ... }

o = d1.copy() o.update(d2) o {'publisher': 'Academic Press', 'title': 'Structured Programming', 'locations': {'North America': 'New York', 'Europe': 'London'}, 'authors': 'Dahl, Dijkstra, and Hoare'}

merge(d1, d2) {'publisher': 'Academic Press', 'title': 'Structured Programming', 'locations': {'Dijkstra': 'University of Texas', 'Hoare': 'Oxford University', 'Dahl': 'University of Oslo'}, 'authors': 'Dahl, Dijkstra, and Hoare'}

recursive_merge(d1, d2) {'publisher': 'Academic Press', 'title': 'Structured Programming', 'locations': {'North America': 'New York', 'Europe': 'London', 'Dijkstra': 'University of Texas', 'Hoare': 'Oxford University', 'Dahl': 'University of Oslo'}, 'authors': 'Dahl, Dijkstra, and Hoare'}

https://repl.it/@dave_shawley/PuttingMetaclassesToWork https://repl.it/@dave_shawley/PuttingMetaclassesToWork

IMO, having more than one obvious outcome means that we should refuse the temptation to guess. If we do, then the result is only obvious to a subset of users and will be a surprise to the others. It's also useful to note that I am having trouble coming up with another programming language that supports a "+" operator for map types. Does anyone have an example of another programming language that allows for addition of dictionaries/mappings? If so, what is the behavior there? - dave -- Any linter or project that treats PEP 8 as mandatory has *already* failed, as PEP 8 itself states that the rules can be broken as needed. - Paul Moore.

On Tue, Mar 05, 2019 at 08:11:29AM -0500, David Shawley wrote:

"Putting Metaclasses to Work" (ISBN-13 978-0201433050) presents a more mathematical view of programming language types that includes two distinct operations for combining dictionaries -- merge and recursive merge.

For two input dictionaries D1 & D2 and the output dictionary O

D1 merge D2 O is D1 with the of those keys of D2 that do not have keys in D1

D1 recursive-merge D2 For all keys k, O[k] = D1[k] recursive merge D2[k] if both D1[k] and D2[k] are dictionaries, otherwise O[k] = (D1 merge D2)[k].

I'm afraid I cannot understand either of those algorithms as written. I suspect that you've left at least one word out of the first. Fortunately your example below is extremely clear, thank you. [...]

The following example uses dictionaries from "Putting Metaclasses to Work":

...

...

...

d1 = { ... 'title': 'Structured Programming', ... 'authors': 'Dahl, Dijkstra, and Hoare', ... 'locations': { ... 'Dahl': 'University of Oslo', ... 'Dijkstra': 'University of Texas', ... 'Hoare': 'Oxford University', ... }, ... }

d2 = { ... 'publisher': 'Academic Press', ... 'locations': { ... 'North America': 'New York', ... 'Europe': 'London', ... }, ... }

o = d1.copy() o.update(d2) o {'publisher': 'Academic Press', 'title': 'Structured Programming', 'locations': {'North America': 'New York', 'Europe': 'London'}, 'authors': 'Dahl, Dijkstra, and Hoare'}

Yes, that's the classic "update with last seen wins". That's what the PEP proposes as that seems to be the most frequently requested behaviour. It is also the only behaviour which has been deemed useful enough in nearly 30 years of Python's history to be added to dict as a method.

...

...

...

merge(d1, d2) {'publisher': 'Academic Press', 'title': 'Structured Programming', 'locations': {'Dijkstra': 'University of Texas', 'Hoare': 'Oxford University', 'Dahl': 'University of Oslo'}, 'authors': 'Dahl, Dijkstra, and Hoare'}

That seems to be "update with first seen wins", which is easily done using ChainMap or the proposed dict difference operator: dict( ChainMap(d1, d2) ) # or d1 + (d2 - d1) or simply by swapping the order of the operands: d2 + d1 (These are not *identical* in effect, there are small differences with respect to key:value identity, and order of keys. But they ought to give *equal* results.) Personally, I don't think that behaviour is as useful as the first, but it is certainly a legitimate kind of merge. As far as I know, this has never been requested before. Perhaps it is too niche?

...

...

...

recursive_merge(d1, d2) {'publisher': 'Academic Press', 'title': 'Structured Programming', 'locations': {'North America': 'New York', 'Europe': 'London', 'Dijkstra': 'University of Texas', 'Hoare': 'Oxford University', 'Dahl': 'University of Oslo'}, 'authors': 'Dahl, Dijkstra, and Hoare'}

That's an interesting one. I'd write it something like this: def merge(a, b): new = a.copy() for key, value in b: if key not in a: # Add new keys. new[key] = value else: v = new[key] if isinstance(value, dict) and isinstance(v, dict): # If both values are dicts, merge them. new[key] = merge(v, value) else: # What to do if only one is a dict? # Or if neither is a dict? return new I've seen variants of this where duplicate keys are handled by building a list of the values: def merge(a, b): new = a.copy() for key, value in b: if key in a: v = new[key] if isinstance(v, list): v.append(value) else: new[key] = [v, value] ... or by concatenating values, or adding them (as Counter does), etc. We have subclasses and operator overloading, so you can implement whatever behaviour you like. The question is, is this behaviour useful enough and common enough to be built into dict itself?

IMO, having more than one obvious outcome means that we should refuse the temptation to guess.

We're not *guessing*. We're *chosing* which behaviour we want. Nobody says: When I print some strings, I can seperate them with spaces, or dots, or newlines, and print a newline at the end, or suppress the newline. Since all of these behaviours might be useful for somebody, we should not "guess" what the user wants. Therefore we should not have a print() function at all. The behaviour of print() is not a guess as to what the user wants. We offer a specific behaviour, and if the user is happy with that, then they can use print(), and if not, they can write their own. The same applies here: we're offering one specific behaviour that we think is the most important, and anyone who wants another can write their own. If people don't like my choice of what I think is the most important (copy-and-update, with last seen wins), they can argue for whichever alternative they like. If they make a convincing enough case, the PEP can change :-) James Lu has already tried to argue that the "raise on non-unique keys" is the best behaviour. I have disagreed with that, but if James makes a strong enough case for his idea, and it gains sufficient support, I could be persuaded to change my position. Or he can write a competing PEP and the Steering Council can decide between the two ideas.

If we do, then the result is only obvious to a subset of users and will be a surprise to the others.

Its only a surprise to those users who don't read the docs and make assumptions about behaviour based on their own wild guesses. We should get away from the idea that the only behaviours we can provide are those which are "obvious" (intuitive?) to people who guess what it means without reading the docs. It's great when a function's meaning can be guessed or inferred from a basic understanding of English: len(string) # assuming len is an abbreviation for length but that sets the bar impossibly high. We can't guess what these do, not with any precision: print(spam, eggs) # prints spaces between arguments or not? spam is eggs # that's another way of spelling == right? zip(spam, eggs) # what does it do if args aren't the same length? and who can guess what these do without reading the docs? property, classmethod, slice, enumerate, iter I don't think that Python is a worse language for having specified a meaning for these rather than leaving them out. The Zen's prohibition against guessing in the face of ambiguity does not mean that we must not add a feature to the language that requires the user to learn what it does first.

It's also useful to note that I am having trouble coming up with another programming language that supports a "+" operator for map types.

Does anyone have an example of another programming language that allows for addition of dictionaries/mappings?

If so, what is the behavior there?

An excellent example, but my browser just crashed and it's after 3am here so I'm going to take this opportunity to go to bed :-) -- Steven

I'd like to remove all doubt: {**d1} needs to work regardless of the key type, as long as it's hashable (d1 could be some mapping implemented without hashing, e.g. using a balanced tree, so that it could support unhashable keys).

If there's doubt about this anywhere, we could add an example to the docs and to the PEP.

On a related note: **kwargs, should they support arbitrary strings as keys? I depend on this behavior in production code and all python implementations handle it. / Anders

There was a thread for the topic. https://mail.python.org/pipermail/python-dev/2018-October/155435.html 2019年3月6日(水) 2:02 Anders Hovmöller :

On a related note: **kwargs, should they support arbitrary strings as keys? I depend on this behavior in production code and all python implementations handle it.

/ Anders

Steven D'Aprano wrote:

The question is, is [recursive merge] behaviour useful enough and common enough to be built into dict itself?

I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values. -- Greg

Should our __sub__ behavior be the same...

Sorry, our "__isub__" behavior. Long day... On Tue, Mar 5, 2019 at 2:47 PM Brandt Bucher wrote:

These semantics are intended to match those of update as closely as

...

possible. For the dict built-in itself, calling keys is redundant as iteration over a dict iterates over its keys; but for subclasses or other mappings, update prefers to use the keys method.

The above paragraph may be inaccurate. Although the dict docstring states that keys will be called if it exists, this does not seem to be the case for dict subclasses. Bug or feature?

...

...

...

print(dict.update.__doc__) D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

It's actually pretty interesting... and misleading/wrongish. It never says that keys is *called*... in reality, it just checks for the "keys" method before deciding whether to proceed with PyDict_Merge or PyDict _MergeFromSeq2. It should really read more like:

D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present, has a .keys() method, and is a subclass of dict, then does: for k in E: D[k] = E[k] If E is present, has a .keys() method, and is not a subclass of dict, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

Should our __sub__ behavior be the same (i.e., iterate for dict subclasses and objects without "keys()", otherwise call "keys()" and iterate over that)? __iadd__ calls into this logic already. It seems to be the most "natural" solution here, if we desire behavior analogous to "update".

Brandt

On Fri, Mar 1, 2019 at 8:26 AM Steven D'Aprano wrote:

...

Attached is a draft PEP on adding + and - operators to dict for discussion.

This should probably go here:

https://github.com/python/peps

but due to technical difficulties at my end, I'm very limited in what I can do on Github (at least for now). If there's anyone who would like to co-author and/or help with the process, that will be appreciated.

-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:

I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown.

This proposal is for a simple, operator-based equivalent to dict.update() which returns a new dict. dict.update has existed since Python 1.5 (something like a quarter of a century!) and never grown a "unique keys" version. I don't recall even seeing a request for such a feature. If such a unique keys version is useful, I don't expect it will be useful often.

This way, d1 + d2 isn’t just another obvious way to do {**d1, **d2}.

One of the reasons for preferring + is that it is an obvious way to do something very common, while {**d1, **d2} is as far from obvious as you can get without becoming APL or Perl :-) If I needed such a unique key version of update, I'd use a subclass: class StrictDict(dict): def __add__(self, other): if isinstance(other, dict) and (self.keys() & other.keys()): raise KeyError('non-unique keys') return super().__add__(self, other) # and similar for __radd__. rather than burden the entire language, and every user of it, with having to learn the subtle difference between the obvious + operator and the error-prone and unobvious trick of {*d1, *d2}. ( Did you see what I did there? *wink* )

The second syntax makes it clear that a new dictionary is being constructed and that d2 overrides keys from d1.

Only because you have learned the rule that {**d, **e) means to construct a new dict by merging, with the rule that in the event of duplicate keys, the last key seen wins. If you hadn't learned that rule, there is nothing in the syntax which would tell you the behaviour. We could have chosen any rule we liked: - raise an exception, like you get a TypeError if you pass the same keyword argument to a function twice: spam(foo=1, foo=2); - first value seen wins; - last value seen wins; - random value wins; - anything else we liked! There is nothing "clear" about the syntax which makes it obvious which behaviour is implemented. We have to learn it.

One can reasonably expect or imagine a situation where a section of code that expects to merge two dictionaries with non-conflicting keys commits a semantic error if it merges two dictionaries with conflicting keys.

I can imagine it, but I don't think I've ever needed it, and I can't imagine wanting it often enough to wish it was not just a built-in function or method, but actual syntax. Do you have some real examples of wanting an error when trying to update a dict if keys match?

To better explain, imagine a program where options is a global variable storing parsed values from the command line.

def verbose_options(): if options.quiet return {'verbose': True}

def quiet_options(): if options.quiet: return {'verbose': False}

That seems very artifical to me. Why not use a single function: def verbose_options(): # There's more than one? return {'verbose': not options.quiet} The way you have written those functions seems weird to me. You already have a nice options object, with named fields like "options.quiet", why are you turning it into not one but *two* different dicts, both reporting the same field? And its buggy: if options.quiet is True, then the key 'quiet' should be True, not the 'verbose' key. Do you have *two* functions for every preference setting that takes a true/false flag? What do you do for preference settings that take multiple values? Create a vast number of specialised functions, one for each possible value? def A4_page_options(): if options.page_size == 'A4': return {'page_size': 'A4'} def US_Letter_page_options(): if options.page_size == 'US Letter': return {'page_size': 'US Letter'} page_size = ( A4_page_options() + A3_page_options() + A5_page_options() + Foolscape_page_options + Tabloid_page_options() + US_Letter_page_options() + US_Legal_page_options() # and about a dozen more... ) The point is, although I might be wrong, I don't think that this example is a practical, realistic use-case for a unique keys version of update. To me, your approach seems so complicated and artificial that it seems like it was invented specifically to justify this "unique key" operator, not something that we would want to write in real life. But even if it real code, the question is not whether it is EVER useful for a dict update to raise an exception on matching keys. The question is whether this is so often useful that this is the behaviour we want to make the default for dicts. [...]

Again, I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown, because such “non-conflicting merge” behavior would be useful in Python.

I don't think it would be, at least not often. If it were common enough to justify a built-in operator to do this, we would have had many requests for a dict.unique_update or similar by now, and I don't think we have.

It gives clarifying power to the + sign. The + and the {**, **} should serve different roles.

In other words, explicit + is better than implicit {**, **#, unless explicitly suppressed. Here + is explicit whereas {**, **} is implicitly allowing inclusive keys,

If I had a cent for every time people misused "explicit" to mean "the proposal that I like", I'd be rich. In what way is the "+" operator *explicit* about raising an exception on duplicate keys? These are both explicit: merge_but_raise_exception_if_any_duplicates(d1, d2) merge(d1, d2, raise_if_duplicates=True) and these are both equally implicit: d1 + d2 {**d1, **d2} since the behaviour on duplicates is not explicitly stated in clear and obvious language, but implied by the rules of the language. [...]

People expect the + operator to be commutative

THey are wrong to expect that, because the + operator is already not commutative for: str bytes bytearray list tuple array.array collections.deque collections.Counter and possibly others. -- Steven

On Tue, Mar 5, 2019 at 11:16 PM Steven D'Aprano wrote:

On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:

...

I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown.

This proposal is for a simple, operator-based equivalent to dict.update() which returns a new dict. dict.update has existed since Python 1.5 (something like a quarter of a century!) and never grown a "unique keys" version.

I don't recall even seeing a request for such a feature. If such a unique keys version is useful, I don't expect it will be useful often.

I have one argument in favor of such a feature: It preserves concatenation semantics. + means one of two things in all code I've ever seen (Python or otherwise): 1. Numeric addition (including element-wise numeric addition as in Counter and numpy arrays) 2. Concatenation (where the result preserves all elements, in order, including, among other guarantees, that len(seq1) + len(seq2) == len(seq1 + seq2)) dict addition that didn't reject non-unique keys wouldn't fit *either* pattern; the main proposal (making it equivalent to left.copy(), followed by .update(right)) would have the left hand side would win on ordering, the right hand side on values, and wouldn't preserve the length invariant of concatenation. At least when repeated keys are rejected, most concatenation invariants are preserved; order is all of the left elements followed by all of the right, and no elements are lost.

...

This way, d1 + d2 isn’t just another obvious way to do {**d1, **d2}.

One of the reasons for preferring + is that it is an obvious way to do something very common, while {**d1, **d2} is as far from obvious as you can get without becoming APL or Perl :-)

From the moment PEP 448 published, I've been using unpacking as a more

composable/efficient form of concatenation, merging, etc. I'm sorry you don't find it obvious, but a couple e-mails back you said: "The Zen's prohibition against guessing in the face of ambiguity does not mean that we must not add a feature to the language that requires the user to learn what it does first." Learning to use the unpacking syntax in the case of function calls is necessary for tons of stuff (writing general function decorators, handling initialization in class hierarchies, etc.), and as PEP 448 is titled, this is just a generalization combining the features of unpacking arguments with collection literals.

The second syntax makes it clear that a new dictionary is being

...

constructed and that d2 overrides keys from d1.

Only because you have learned the rule that {**d, **e) means to construct a new dict by merging, with the rule that in the event of duplicate keys, the last key seen wins. If you hadn't learned that rule, there is nothing in the syntax which would tell you the behaviour. We could have chosen any rule we liked:

No, because we learned the general rule for dict literals that {'a': 1, 'a': 2} produces {'a': 2}; the unpacking generalizations were very good about adhering to the existing rules, so it was basically zero learning curve if you already knew dict literal rules and less general unpacking rules. The only part to "learn" is that when there is a conflict between dict literal rules and function call rules, dict literal rules win. To be clear: I'm not supporting + as raising error on non-unique keys. Even if it makes dict + dict adhere to the rules of concatenation, I don't think it's a common or useful functionality. My order of preferences is roughly: 1. Do nothing (even if you don't like {**d1, **d2}, .copy() followed by .update() is obvious, and we don't need more than one way to do it) 2. Add a new method to dict, e.g. dict.merge (whether it's a class method or an instance method is irrelevant to me) 3. Use | (because dicts are *far* more like sets than they are like sequences, and the semi-lossy rules of unioning make more sense there); it would also make - make sense, since + is only matched by - in numeric contexts; on collections, | and - are paired. And I consider the - functionality the most useful part of this whole proposal (because I *have* wanted to drop a collection of known blacklisted keys from a dict and while it's obvious you can do it by looping, I always wanted to be able to do something like d1.keys() -= badkeys, and remain disappointed nothing like it is available) -Josh Rosenberg

On Tue, Mar 5, 2019 at 3:50 PM Josh Rosenberg < shadowranger+pythonideas@gmail.com> wrote:

...

On Tue, Mar 5, 2019 at 11:16 PM Steven D'Aprano wrote:

...

On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:

...

I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown.

This proposal is for a simple, operator-based equivalent to dict.update() which returns a new dict. dict.update has existed since Python 1.5 (something like a quarter of a century!) and never grown a "unique keys" version.

I don't recall even seeing a request for such a feature. If such a unique keys version is useful, I don't expect it will be useful often.

I have one argument in favor of such a feature: It preserves concatenation semantics. + means one of two things in all code I've ever seen (Python or otherwise):

1. Numeric addition (including element-wise numeric addition as in Counter and numpy arrays) 2. Concatenation (where the result preserves all elements, in order, including, among other guarantees, that len(seq1) + len(seq2) == len(seq1 + seq2))

dict addition that didn't reject non-unique keys wouldn't fit *either* pattern; the main proposal (making it equivalent to left.copy(), followed by .update(right)) would have the left hand side would win on ordering, the right hand side on values, and wouldn't preserve the length invariant of concatenation. At least when repeated keys are rejected, most concatenation invariants are preserved; order is all of the left elements followed by all of the right, and no elements are lost.

I must by now have seen dozens of post complaining about this aspect of the proposal. I think this is just making up rules (e.g. "+ never loses information") to deal with an aspect of the design where a *choice* must be made. This may reflect the Zen of Python's "In the face of ambiguity, refuse the temptation to guess." But really, that's a pretty silly rule (truly, they aren't all winners). Good interface design constantly makes choices in ambiguous situations, because the alternative is constantly asking, and that's just annoying.

We have a plethora of examples (in fact, almost all alternatives considered) of situations related to dict merging where a choice is made between conflicting values for a key, and it's always the value further to the right that wins: from d[k] = v (which overrides the value when k is already in the dict) to d1.update(d2) (which lets the values in d2 win), including the much lauded {**d1, **d2} and even plain {'a': 1, 'a': 2} has a well-defined meaning where the latter value wins.

Yeah. And I'm fine with the behavior for update because the name itself is descriptive; we're spelling out, in English, that we're update-ing the

On Wed, Mar 6, 2019 at 12:08 AM Guido van Rossum wrote: thing it's called on, so it makes sense to have the thing we're sourcing for updates take precedence. Similarly, for dict literals (and by extension, unpacking), it's following an existing Python convention which doesn't contradict anything else. Overloading + lacks the clear descriptive aspect of update that describes the goal of the operation, and contradicts conventions (in Python and elsewhere) about how + works (addition or concatenation, and a lot of people don't even like it doing the latter, though I'm not that pedantic). A couple "rules" from C++ on overloading are "*Whenever the meaning of an operator is not obviously clear and undisputed, it should not be overloaded.* *Instead, provide a function with a well-chosen name.*" and "*Always stick to the operator’s well-known semantics".* (Source: https://stackoverflow.com/a/4421708/364696 , though the principle is restated in many other places). Obviously the C++ community isn't perfect on this (see iostream and <> operators), but they're otherwise pretty consistent. + means addition, and in many languages including C++ strings, concatenation, but I don't know of any languages outside the "esoteric" category that use it for things that are neither addition nor concatenation. You've said you don't want the whole plethora of set-like behaviors on dicts, but dicts are syntactically and semantically much more like sets than sequences, and if you add + (with semantics differing from both sets and sequences), the language becomes less consistent. I'm not against making it easier to merge dictionaries. But people seem to be arguing that {**d1, **d2} is bad because of magic punctuation that obscures meaning, when IMO: d3 = d1 + d2 is obscuring meaning by adding yet a third rule for what + means, inconsistent with both existing rules (from both Python and the majority of languages I've had cause to use). A named method (class or instance) or top-level function (a la sorted) is more explicit, easier to look up (after all, the major complaint about ** syntax is the difficulty of finding the documentation on it). It's also easier to make it do the right thing; d1 + d2 + d3 + ... dN is inefficient (makes many unnecessary temporaries), {**d1, **d2, **d3, ..., **dN} is efficient but obscure (and not subclass friendly), but a varargs method like dict.combine(d1, d2, d3, ..., dN) (or merge, or whatever; I'm not trying to bikeshed) is correct, efficient, and most importantly, easy to look up documentation for. I occasionally find it frustrating that concatenation exists given the wealth of Schlemiel the Painter's algorithms it encourages, and the "correct" solution for combining sequences (itertools.chain for general cases, str.join/bytes.join for special cases) being less obvious means my students invariably use the "wrong" tool out of convenience (and it's not really wrong in 90% of code where the lengths are always short, but then they use it where lengths are often huge and suffer for it). If we're going to make dict merging more convenient, I'd prefer we make the obvious, convenient solution also the one that doesn't encourage non-scalable anti-patterns. As to why raising is worse: First, none of the other situations I listed

above raises for conflicts. Second, there's the experience of str+unicode in Python 2, which raises if the str argument contains any non-ASCII bytes. In fact, we disliked it so much that we changed the language incompatibly to deal with it.

Agreed, I don't like raising. It's consistent with + (the only argument in favor of it really), but it's a bad idea, for all the reasons you mention. - Josh Rosenberg

Le 05/03/2019 à 23:40, Greg Ewing a écrit :

Steven D'Aprano wrote:

...

The question is, is [recursive merge] behaviour useful enough and common enough to be built into dict itself?

I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values.

That's what this conversation led me to. I'm not against the addition for the most general usage (and current PEP's describes the behaviour I would expect before reading the doc), but for all other more specific usages, where we intend any special or not-so-common behaviour, I'd go with modifying Dict.update like this: foo.update(bar, on_collision=updator)  # Although I'm not a fan of the keyword I used `updator` being a simple function like this one: def updator(updated, updator, key) -> Any:     if key == "related":         return updated[key].update(updator[key])     if key == "tags":         return updated[key] + updator[key]     if key in ["a", "b", "c"]:  # Those         return updated[key]     return updator[key] There's nothing here that couldn't be made today by using a custom update function, but leaving the burden of checking for values that are in both and actually inserting the new values to Python's language, and keeping on our side only the parts that are specific to our use case, makes in my opinion the code more readable, with fewer possible bugs and possibly better optimization.

Le 6 mars 2019 à 10:26:15, Brice Parent (contact@brice.xyz(mailto:contact@brice.xyz)) a écrit:

Le 05/03/2019 à 23:40, Greg Ewing a écrit :

...

Steven D'Aprano wrote:

...

The question is, is [recursive merge] behaviour useful enough and common enough to be built into dict itself?

I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values.

That's what this conversation led me to. I'm not against the addition for the most general usage (and current PEP's describes the behaviour I would expect before reading the doc), but for all other more specific usages, where we intend any special or not-so-common behaviour, I'd go with modifying Dict.update like this:

foo.update(bar, on_collision=updator) # Although I'm not a fan of the keyword I used

Le 6 mars 2019 à 10:26:15, Brice Parent (contact@brice.xyz(mailto:contact@brice.xyz)) a écrit:

Le 05/03/2019 à 23:40, Greg Ewing a écrit :

...

Steven D'Aprano wrote:

...

The question is, is [recursive merge] behaviour useful enough and common enough to be built into dict itself?

I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values.

That's what this conversation led me to. I'm not against the addition for the most general usage (and current PEP's describes the behaviour I would expect before reading the doc), but for all other more specific usages, where we intend any special or not-so-common behaviour, I'd go with modifying Dict.update like this:

foo.update(bar, on_collision=updator) # Although I'm not a fan of the keyword I used

This won’t be possible update() already takes keyword arguments:

...

...

foo = {} bar = {'a': 1} foo.update(bar, on_collision=lambda e: e) foo {'a': 1, 'on_collision': }

`updator` being a simple function like this one:

def updator(updated, updator, key) -> Any: if key == "related": return updated[key].update(updator[key])

if key == "tags": return updated[key] + updator[key]

if key in ["a", "b", "c"]: # Those return updated[key]

return updator[key]

There's nothing here that couldn't be made today by using a custom update function, but leaving the burden of checking for values that are in both and actually inserting the new values to Python's language, and keeping on our side only the parts that are specific to our use case, makes in my opinion the code more readable, with fewer possible bugs and possibly better optimization.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 06/03/2019 10:29, Ka-Ping Yee wrote:

len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense.

I'm sorry, but you're going to have to justify why this identity is important. Making assumptions about length where any dictionary manipulations are concerned seems unwise to me, which makes a nonsense of your claim that this is nonsense :-) -- Rhodri James *-* Kynesim Ltd

Ka-Ping Yee wrote:

len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense.

len(dict1 + dict2) cannot even be computed by any expression involving +. Using len() to test the semantics of the operation is not arbitrary; the fact that the sizes do not add is a defining quality of a merge. This is a merge, not an addition. The proper analogy is to sets, not lists.

For me, this comment is excellent. It neatly expresses the central concern about this proposal. I think most us will agree that the proposal is to use '+' to express a merge operation, namely update. (There are other merge operations, when there are two values to combine, such as taking the min or max of the two values.) Certainly, many of the posts quite naturally use the word merge. Indeed PEP 584 writes "This PEP suggests adding merge '+' and difference '-' operators to the built-in dict class." We would all agree that it would be obviously wrong to suggest adding merge '-' and difference '+' operators. (Note: I've swapped '+' and '-'.) And why? Because it is obviously wrong to use '-' to denote merge, etc. Some of us are also upset by the use of '+' to denote merge. By the way, there is already a widespread symbol for merge. It appears on many road signs. It looks like an upside down 'Y'. It even has merge left and merge right versions. Python already has operator symbols '+', '-', '*', '/' and so on. See https://docs.python.org/3/reference/lexical_analysis.html#operators Perhaps we should add a merge or update symbol to this list, so that we don't overload to breaking point the humble '+' operator. Although that would make Python a bit more like APL. By the way, Pandas already has a merge operation, called merge, that takes many parameters. I've only glanced at it. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.... -- Jonathan

Rhodri James wrote:

Making assumptions about length where any dictionary manipulations are concerned seems unwise to me

I think you're a bit hasty here. Some assumptions are sensible. Suppose a = len(d1) b = len(d2) c = len(d1 + d2) # Using the suggested syntax. Then we know max(a, b) <= c <= a + b And this is, in broad terms, characteristic of merge operations. -- Jonathan

On Wed, Mar 6, 2019 at 11:18 PM Brice Parent wrote:

The major implication to such a modification of the Dict.update method, is that when you're using it with keyword arguments (by opposition to passing another dict/iterable as positional), you're making a small non-backward compatible change in that if in some code, someone was already using the keyword that would be chosing (here "on_collision"), their code would be broken by the new feature. Anyway, if the keyword is slected wisely, the collision case will almost never happen, and be quite easy to correct if it ever happened.

You can make it unlikely, yes, but I'd dispute "easy to correct". Let's suppose that someone had indeed used the chosen keyword (and remember, the more descriptive the argument name, the more likely that it'll be useful elsewhere and therefore have a collision). How would they discover this? If they're really lucky, there MIGHT be an exception (if on_collision accepts only a handful of keywords, and the collision isn't one of them), but if your new feature is sufficiently flexible, that might not happen. There'll just be incorrect behaviour. As APIs go, using specific keyword args at the same time as **kw is a bit odd. Consider: button_options.update(button_info, on_click=frobnicate, style="KDE", on_collision="replace") It's definitely not obvious which of those will end up in the dictionary and which won't. Big -1 from me on that change. ChrisA

I strongly agree with Ka-Ping. '+' is intuitively concatenation not merging. The behavior is overwhelmingly more similar to the '|' operator in sets (whether or not a user happens to know the historical implementation overlap).

I think the behavior proposed in the PEP makes sense whether you think of "+" as meaning "concatenation" or "merging". If your instinct is to assume "+" means "concatenation", then it would be natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs into a new dict. But of course, you can't have duplicate keys in Python. So, you would either recall or look up how duplicate keys are handled when constructing a dict and learn that the rule is that the right-most key wins. So the natural conclusion is that "+" would follow this existing rule -- and you end up with exactly the behavior described in the PEP. This also makes explaining the behavior of "d1 + d2" slightly easier than explaining "d1 | d2". For the former, you can just say "d1 + d2 means we concat the two dicts together" and stop there. You almost don't need to explain the merging/right-most key wins behavior at all, since that behavior is the only one consistent with the existing language rules. In contrast, you *would* need to explain this with "d1 | d2": I would mentally translate this expression to mean "take the union of these two dicts" and there's no real way to deduce which key-value pair ends up in the final dict given that framing. Why is it that key-value pairs in d2 win over pairs in d1 here? That choice seems pretty arbitrary when you think of this operation in terms of unions, rather than either concat or merge. Using "|" would also violate an important existing property of unions: the invariant "d1 | d2 == d2 | d1" is no longer true. As far as I'm aware, the union operation is always taken to be commutative in math, and so I think it's important that we preserve that property in Python. At the very least, I think it's far more important to preserve commutativity of unions then it is to preserve some of the invariants I've seen proposed above, like "len(d1 + d2) == len(d1) + len(d2)". Personally, I don't really have a strong opinion on this PEP, or the other one I've seen proposed where we add a "d1.merge(d2, d3, ...)". But I do know that I'm a strong -1 on adding set operations to dicts: it's not possible to preserve the existing semantics of union (and intersection) with dict and think expressions like "d1 | d2" and "d1 & d2" would just be confusing and misleading to encounter in the wild. -- Michael On Wed, Mar 6, 2019 at 4:53 AM David Mertz wrote:

I strongly agree with Ka-Ping. '+' is intuitively concatenation not merging. The behavior is overwhelmingly more similar to the '|' operator in sets (whether or not a user happens to know the historical implementation overlap).

I think growing the full collection of set operations world be a pleasant addition to dicts. I think shoe-horning in plus would always be jarring to me.

On Wed, Mar 6, 2019, 5:30 AM Ka-Ping Yee wrote:

...

len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense.

len(dict1 + dict2) cannot even be computed by any expression involving +. Using len() to test the semantics of the operation is not arbitrary; the fact that the sizes do not add is a defining quality of a merge. This is a merge, not an addition. The proper analogy is to sets, not lists.

The operators should be |, &, and -, exactly as for sets, and the behaviour defined with just three rules:

1. The keys of dict1 [op] dict2 are the elements of dict1.keys() [op] dict2.keys().

2. The values of dict2 take priority over the values of dict1.

3. When either operand is a set, it is treated as a dict whose values are None.

This yields many useful operations and, most importantly, is simple to explain. "sets and dicts can |, &, -" takes up less space in your brain than "sets can |, &, - but dicts can only + and -, where dict + is like set |".

merge and update some items:

{'a': 1, 'b': 2} | {'b': 3, 'c': 4} => {'a': 1, 'b': 3, 'c': 4}

pick some items:

{'a': 1, 'b': 2} & {'b': 3, 'c': 4} => {'b': 3}

remove some items:

{'a': 1, 'b': 2} - {'b': 3, 'c': 4} => {'a': 1}

reset values of some keys:

{'a': 1, 'b': 2} | {'b', 'c'} => {'a': 1, 'b': None, 'c': None}

ensure certain keys are present:

{'b', 'c'} | {'a': 1, 'b': 2} => {'a': 1, 'b': 2, 'c': None}

pick some items:

{'b', 'c'} | {'a': 1, 'b': 2} => {'b': 2}

remove some items:

{'a': 1, 'b': 2} - {'b', 'c'} => {'a': 1}

On Wed, Mar 6, 2019 at 1:51 AM Rémi Lapeyre wrote:

...

Le 6 mars 2019 à 10:26:15, Brice Parent (contact@brice.xyz(mailto:contact@brice.xyz)) a écrit:

...

Le 05/03/2019 à 23:40, Greg Ewing a écrit :

...

Steven D'Aprano wrote:

...

The question is, is [recursive merge] behaviour useful enough and common enough to be built into dict itself?

I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values.

That's what this conversation led me to. I'm not against the addition for the most general usage (and current PEP's describes the behaviour I would expect before reading the doc), but for all other more specific usages, where we intend any special or not-so-common behaviour, I'd go with modifying Dict.update like this:

foo.update(bar, on_collision=updator) # Although I'm not a fan of the keyword I used

Le 6 mars 2019 à 10:26:15, Brice Parent (contact@brice.xyz(mailto:contact@brice.xyz)) a écrit:

...

Le 05/03/2019 à 23:40, Greg Ewing a écrit :

...

Steven D'Aprano wrote:

...

The question is, is [recursive merge] behaviour useful enough and common enough to be built into dict itself?

I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values.

That's what this conversation led me to. I'm not against the addition for the most general usage (and current PEP's describes the behaviour I would expect before reading the doc), but for all other more specific usages, where we intend any special or not-so-common behaviour, I'd go with modifying Dict.update like this:

foo.update(bar, on_collision=updator) # Although I'm not a fan of the keyword I used

This won’t be possible update() already takes keyword arguments:

...

...

...

foo = {} bar = {'a': 1} foo.update(bar, on_collision=lambda e: e) foo {'a': 1, 'on_collision': }

...

`updator` being a simple function like this one:

def updator(updated, updator, key) -> Any: if key == "related": return updated[key].update(updator[key])

if key == "tags": return updated[key] + updator[key]

if key in ["a", "b", "c"]: # Those return updated[key]

return updator[key]

There's nothing here that couldn't be made today by using a custom update function, but leaving the burden of checking for values that are in both and actually inserting the new values to Python's language, and keeping on our side only the parts that are specific to our use case, makes in my opinion the code more readable, with fewer possible bugs and possibly better optimization.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Michael Lee wrote:

If your instinct is to assume "+" means "concatenation", then it would be natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs into a new dict.

But of course, you can't have duplicate keys in Python. So, you would either recall or look up how duplicate keys are handled when constructing a dict and learn that the rule is that the right-most key wins. So the natural conclusion is that "+" would follow this existing rule -- and you end up with exactly the behavior described in the PEP.

This is a nice argument. And well presented. And it gave me surprise, that taught me something. Here goes: >>> {'a': 0} {'a': 0} >>> {'a': 0, 'a': 0} {'a': 0} >>> {'a': 0, 'a': 1} {'a': 1} >>> {'a': 1, 'a': 0} {'a': 0} This surprised me quite a bit. I was expecting to get an exception. However >>> dict(a=0) {'a': 0} >>> dict(a=0, a=0) SyntaxError: keyword argument repeated does give an exception. I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested) anywhere? I didn't find it in these URLs: https://docs.python.org/3/library/stdtypes.html#mapping-types-dict https://docs.python.org/3/tutorial/datastructures.html#dictionaries I think this behaviour might give rise to gotchas. For example, if we define inverse_f by >>> inverse_f = { f(a): a, f(b): b } then is the next statement always true (assuming a <> b)? >>> inverse_f[ f(a) ] == a Well, it's not true with these values >>> a, b = 1, 2 >>> def f(n): pass # There's a bug here, f(n) should be a bijection. A quick check that len(inverse) == 2 would provide a sanity check. Or perhaps better, len(inverse_f) == len(set(a, b)). (I don't have an example of this bug appearing 'in the wild'.) Once again, I thank Michael for his nice, instructive and well-presented example. -- Jonathan

SUMMARY: The outcome of a search for: python dict literal duplicate keys. No conclusions (so far). BACKGROUND In the thread "PEP: Dict addition and subtraction" I wrote

>>> {'a': 0, 'a': 1} {'a': 1}

I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested) anywhere? I didn't find it in these URLs: https://docs.python.org/3/library/stdtypes.html#mapping-types-dict https://docs.python.org/3/tutorial/datastructures.html#dictionaries

LINKS I've since found some relevant URLs. [1] https://stackoverflow.com/questions/34539772/is-a-dict-literal-containing-re... [2] https://help.semmle.com/wiki/display/PYTHON/Duplicate+key+in+dict+literal [3] https://bugs.python.org/issue26910 [4] https://bugs.python.org/issue16385 [5] https://realpython.com/python-dicts/ ANALYSIS [1] gives a reference to [6], which correctly states the behaviour of {'a':0, 'a':1}, although without giving an example. (Aside: Sometimes one example is worth 50 or more words.) [2] is from Semmle, who provide an automated code review tool, called LGTM. The page [2] appears to be part of the documentation for LGTM. This page provides a useful link to [7]. [3] is a re-opening of [4]. It was rapidly closed by David Murray, who recommended reopening the discussion on python-ideas. [4] was raised by Albert Ferras, based on his real-world experience. In particular, a configuration file that contains a long dict literal. This was closed by Benjamin Peterson, who said that raising an error was "out of the question for compatibility isssues". Given few use case and little support on python-ideas,Terry Ready supported the closure. Raymond Hettinger supported the closure. [5] is from RealPython, who provide online tutorials. This page contains the statement "a given key can appear in a dictionary only once. Duplicate keys are not allowed." Note that {'a': 0, 'a': 1} can reasonably be thought of as a dictionary with duplicate keys. NOTE As I recall SGML (this shows my age) allows multiple entity declarations, as in <!ENTITY key "original"> <!ENTITY key "updated"> And as I recall, in SGML the first value "original" is the one that is in effect. This is what happens with the LaTeX command \providecommand. FURTHER LINKS [6] https://docs.python.org/3/reference/expressions.html#dictionary-displays [7] https://cwe.mitre.org/data/definitions/561.html # CWE-561: Dead Code -- Jonathan

SUMMARY: Off-thread-topic comment on examples and words in documentation. Inada Naoki quoted (from doc.python ref [6] in my original post):

...

If a comma-separated sequence of key/datum pairs is given, they are evaluated from left to right to define the entries of the dictionary: each key object is used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary’s value for that key will be the last one given.

Indeed. Although off-topic, I think

...

...

{'a': 0, 'a': 1} == {'a': 1} True

is much better than "This means that you can specify the same key multiple times in the key/datum list, and the final dictionary’s value for that key will be the last one given." By the way, today I think we'd say key/value pairs. And I've read https://www.theguardian.com/guardian-observer-style-guide-d data takes a singular verb (like agenda), though strictly a plural; you come across datum, the singular of data, about as often as you hear about an agendum Oh, and "the final dictionary's value" should I think be "the dictionary's final value" or perhaps just "the dictionary's value" But now we're far from the thread topic. I'm happy to join in on a thread on improving documentation (by using simpler language and good examples). -- Jonathan

On 06/03/2019 18:12, Rhodri James wrote:

On 06/03/2019 17:43, Jonathan Fine wrote:

...

Indeed. Although off-topic, I think

...

...

...

{'a': 0, 'a': 1} == {'a': 1} True

is much better than "This means that you can specify the same key multiple times in the key/datum list, and the final dictionary’s value for that key will be the last one given."

I disagree.  An example is an excellent thing, but the words are definitive and must be there.

Sigh. I hit SEND before I finished changing the title. Sorry, folks. -- Rhodri James *-* Kynesim Ltd

Would it shut down this particular subthread if (as the language's designer, if not its BDFL) I declared that this was an explicit design decision that I made nearly 30 years ago? I should perhaps blog about the background of this decision, but it was quite a conscious one. There really is no point in thinking that this is an accident of implementation or could be changed. On Wed, Mar 6, 2019 at 9:10 AM Jonathan Fine wrote:

SUMMARY: The outcome of a search for: python dict literal duplicate keys. No conclusions (so far).

BACKGROUND In the thread "PEP: Dict addition and subtraction" I wrote

...

>>> {'a': 0, 'a': 1} {'a': 1}

...

I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested) anywhere? I didn't find it in these URLs: https://docs.python.org/3/library/stdtypes.html#mapping-types-dict https://docs.python.org/3/tutorial/datastructures.html#dictionaries

LINKS I've since found some relevant URLs.

[1] https://stackoverflow.com/questions/34539772/is-a-dict-literal-containing-re... [2] https://help.semmle.com/wiki/display/PYTHON/Duplicate+key+in+dict+literal [3] https://bugs.python.org/issue26910 [4] https://bugs.python.org/issue16385 [5] https://realpython.com/python-dicts/

ANALYSIS [1] gives a reference to [6], which correctly states the behaviour of {'a':0, 'a':1}, although without giving an example. (Aside: Sometimes one example is worth 50 or more words.)

[2] is from Semmle, who provide an automated code review tool, called LGTM. The page [2] appears to be part of the documentation for LGTM. This page provides a useful link to [7].

[3] is a re-opening of [4]. It was rapidly closed by David Murray, who recommended reopening the discussion on python-ideas. [4] was raised by Albert Ferras, based on his real-world experience. In particular, a configuration file that contains a long dict literal. This was closed by Benjamin Peterson, who said that raising an error was "out of the question for compatibility isssues". Given few use case and little support on python-ideas,Terry Ready supported the closure. Raymond Hettinger supported the closure.

[5] is from RealPython, who provide online tutorials. This page contains the statement "a given key can appear in a dictionary only once. Duplicate keys are not allowed." Note that {'a': 0, 'a': 1} can reasonably be thought of as a dictionary with duplicate keys.

NOTE As I recall SGML (this shows my age) allows multiple entity declarations, as in <!ENTITY key "original"> <!ENTITY key "updated">

And as I recall, in SGML the first value "original" is the one that is in effect. This is what happens with the LaTeX command \providecommand.

FURTHER LINKS [6] https://docs.python.org/3/reference/expressions.html#dictionary-displays [7] https://cwe.mitre.org/data/definitions/561.html # CWE-561: Dead Code

-- Jonathan _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

Ka-Ping Yee wrote:

len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense.

You might as well say that using the + operator on vectors is nonsense, because len(v1 + v2) is not in general equal to len(v1) + len(v2). Yet mathematicians are quite happy to talk about "addition" of vectors. -- Greg

On Thu, Mar 7, 2019 at 10:52 AM Josh Rosenberg wrote:

The closest I can come to a thorough definition of what + does in Python (and most languages) right now is that:

1. Returns a new thing of the same type (or a shared coerced type for number weirdness) 2. That combines the information of the input operands 3. Is associative ((a + b) + c produces the same thing as a + (b + c)) (modulo floating point weirdness) 4. Is "reversible": Knowing the end result and *one* of the inputs is sufficient to determine the value of the other input; that is, for c = a + b, knowing any two of a, b and c allows you to determine a single unambiguous value for the remaining value (numeric coercion and floating point weirdness make this not 100%, but you can at least know a value equal to other value; e.g. for c = a + b, knowing c is 5.0 and a is 1.0 is sufficient to say that b is equal to 4, even if it's not necessarily an int or float). For numbers, reversal is done with -; for sequences, it's done by slicing c using the length of a or b to "subtract" the elements that came from a/b. 5. (Actual addition only) Is commutative (modulo floating point weirdness); a + b == b + a 6. (Concatenation only) Is order preserving (really a natural consequence of #4, but a property that people expect)

Allowing dicts to get involved in + means:

1. Fewer consistent rules apply to +; 2. The particular idiosyncrasies of Python dict ordering and "which value wins" rules are now tied to +. for concatenation, there is only one set of possible rules AFAICT so every language naturally agrees on behavior, but dict merging obviously has many possible rules that would be unlikely to match the exact rules of any other language except by coincidence). a winning on order and b winning on value is a historical artifact of how Python's dict developed; I doubt any other language would intentionally choose to split responsibility like that if they weren't handcuffed by history.

Again, there's nothing wrong with making dict merges easier. But it shouldn't be done by (further) abusing +.

Lots of words that basically say: Stuff wouldn't be perfectly pure. But adding dictionaries is fundamentally *useful*. It is expressive. It will, in pretty much all situations, do exactly what someone would expect, based on knowledge of how Python works in other areas. The semantics for edge cases have to be clearly defined, but they'll only come into play on rare occasions; most of the time, for instance, we don't have to worry about identity vs equality in dictionary keys. If you tell people "adding two dictionaries combines them, with the right operand winning collisions", it won't matter that this isn't how lists or floats work; it'll be incredibly useful as it is. Practicality. Let's have some. ChrisA

Now, this belongs as a separate PEP, and I probably will write one, but I propose: d1 << d2 makes a copy of d1 and merges d2 into it, and when the keys conflict, d2 takes priority. (Works like copy/update.) d1 + d2 makes a new dictionary, taking keys from d1 and d2. If d1 and d2 have a different value for same key, a KeyError is thrown.

Ka-Ping Yee writes:

On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico wrote:

...

But adding dictionaries is fundamentally *useful*. It is expressive.

It is useful. It's just that + is the wrong name.

First, let me say that I prefer ?!'s position here, so my bias is made apparent. I'm also aware that I have biases so I'm sympathetic to those who take a different position. Rather than say it's "wrong", let me instead point out that I think it's pragmatically troublesome to use "+". I can think of at least four interpretations of "d1 + d2" 1. update 2. multiset (~= Collections.Counter addition) 3. addition of functions into the same vector space (actually, a semigroup will do ;-), and this is the implementation of Collections.Counter 4. "fiberwise" set addition (ie, of functions into relations) and I'm very jet-lagged so I may be missing some. There's also the fact that the operations denoted by "|" and "||" are often implemented as "short-circuiting", and therefore not commutative, while "+" usually is (and that's reinforced for mathematicians who are trained to think of "+" as the operator for Abelian groups, while "*" is a (possibly) non-commutative operator. I know commutativity of "+" has been mentioned before, but the non-commutativity of "|" -- and so unsuitability for many kinds of dict combination -- hasn't been emphasized before IIRC. Since "|" (especially "|=") *is* suitable for "update", I think we should reserve "+" for some future commutative extension. In the spirit of full disclosure: Of these, 2 is already implemented and widely used, so we don't need to use dict.__add__ for that. I've never seen 4 in the mathematical literature (union of relations is not the same thing). 3, however, is very common both for mappings with small domain and sparse representation of mappings with a default value (possibly computed then cached), and "|" is not suitable for expressing that sort of addition (I'm willing to say it's "wrong" :-). Steve

On Thu, Mar 7, 2019 at 9:12 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:

Ka-Ping Yee writes:

...

On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico wrote:

...

...

But adding dictionaries is fundamentally *useful*. It is expressive.

It is useful. It's just that + is the wrong name.

First, let me say that I prefer ?!'s position here, so my bias is made apparent. I'm also aware that I have biases so I'm sympathetic to those who take a different position.

TBH, I am warming up to "|" as well.

Rather than say it's "wrong", let me instead point out that I think it's pragmatically troublesome to use "+". I can think of at least four interpretations of "d1 + d2"

1. update 2. multiset (~= Collections.Counter addition)

I guess this explains the behavior of removing results <= 0; it makes sense as multiset subtraction, since in a multiset a negative count makes little sense. (Though the name Counter certainly doesn't seem to imply multiset.)

3. addition of functions into the same vector space (actually, a semigroup will do ;-), and this is the implementation of Collections.Counter 4. "fiberwise" set addition (ie, of functions into relations)

and I'm very jet-lagged so I may be missing some.

There's also the fact that the operations denoted by "|" and "||" are often implemented as "short-circuiting", and therefore not commutative, while "+" usually is (and that's reinforced for mathematicians who are trained to think of "+" as the operator for Abelian groups, while "*" is a (possibly) non-commutative operator. I know commutativity of "+" has been mentioned before, but the non-commutativity of "|" -- and so unsuitability for many kinds of dict combination -- hasn't been emphasized before IIRC.

I've never heard of single "|" being short-circuiting. ("||" of course is infamous for being that in C and most languages derived from it.) And "+" is of course used for many non-commutative operations in Python (e.g. adding two lists/strings/tuples together). It is only *associative*, a weaker requirement that just says (A + B) + C == A + (B + C). (This is why we write A + B + C, since the grouping doesn't matter for the result.) Anyway, while we're discussing mathematical properties, and since SETL was briefly mentioned, I found an interesting thing in math. For sets, union and intersection are distributive over each other. I can't type the operators we learned in high school, so I'll use Python's set operations. We find that A | (B & C) == (A | B) & (A | C). We also find that A & (B | C) == (A & B) | (A & C). Note that this is *not* the case for + and * when used with (mathematical) numbers: * distributes over +: a * (b + c) == (a * b) + (a * c), but + does not distribute over *: a + (b * c) != (a + b) * (a + c). So in a sense, SETL (which uses + and * for union and intersection) got the operators wrong. Note that in Python, + and * for sequences are not distributive this way, since (A + B) * n is not the same as (A * n) + (B * n). OTOH A * (n + m) == A * n + A * m. (Assuming A and B are sequences of the same type, and n and m are positive integers.) If we were to use "|" and "&" for dict "union" and "intersection", the mutual distributive properties will hold.

Since "|" (especially "|=") *is* suitable for "update", I think we should reserve "+" for some future commutative extension.

One argument is that sets have an update() method aliased to "|=", so this makes it more reasonable to do the same for dicts, which also have a. update() method, with similar behavior (not surprising, since sets were modeled after dicts).

In the spirit of full disclosure: Of these, 2 is already implemented and widely used, so we don't need to use dict.__add__ for that. I've never seen 4 in the mathematical literature (union of relations is not the same thing). 3, however, is very common both for mappings with small domain and sparse representation of mappings with a default value (possibly computed then cached), and "|" is not suitable for expressing that sort of addition (I'm willing to say it's "wrong" :-).

-- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:

I guess this explains the behavior of removing results <= 0; it makes sense as multiset subtraction, since in a multiset a negative count makes little sense. (Though the name Counter certainly doesn't seem to imply multiset.)

It doesn't even behave consistently as a multiset, since c[k] -= n is happy to let the value go negative.

For sets, union and intersection are distributive over each other.

Note that this is *not* the case for + and * when used with (mathematical) numbers... So in a sense, SETL (which uses + and * for union and intersection got the operators wrong.

But in another sense, it didn't. In Boolean algebra, "and" and "or" (which also distribute over each other) are often written using the same notations as multiplication and addition. There's no rule in mathematics saying that these notations must be distributive in one direction but not the other. -- Greg

Guido van Rossum wrote:

I guess everybody's high school math(s) class was different. I don't ever recall seeing + and * for boolean OR/AND; we used ∧ and ∨.

Boolean algebra was only touched on briefly in my high school years. I can't remember exactly what notation was used, but it definitely wasn't ∧ and ∨ -- I didn't encounter those until much later. However, I've definitely seen texts on boolean alegbra in relation to logic circuits that write 'A and B' as 'AB', and 'A or B' as 'A + B'. (And also use an overbar for negation instead of the mathematical ¬). Maybe it depends on whether you're a mathematician or an engineer? The multiplication-addition notation seems a lot more readable when you have a complicated boolean expression, so I can imagine it being favoured by pragmatic engineering type people. -- Greg

Thanks to everyone who has contributed to the discussion, I have been reading all the comments even if I haven't responded. I'm currently working on an update to the PEP which will, I hope, improve some of the failings of the current draft. -- Steven

On Sat, Mar 09, 2019 at 11:39:39AM -0800, Stephan Hoyer wrote:

Would __iadd__ and __isub__ be added to collections.abc.MutableMapping?

No, that will not be part of the PEP. The proposal is only to change dict itself. If people want to add this to MutableMapping, that could be considered seperately. -- Steven

On Wed, 6 Mar 2019 00:46:57 +0000 Josh Rosenberg wrote:

Overloading + lacks the clear descriptive aspect of update that describes the goal of the operation, and contradicts conventions (in Python and elsewhere) about how + works (addition or concatenation, and a lot of people don't even like it doing the latter, though I'm not that pedantic).

A couple "rules" from C++ on overloading are "*Whenever the meaning of an operator is not obviously clear and undisputed, it should not be overloaded.* *Instead, provide a function with a well-chosen name.*" and "*Always stick to the operator’s well-known semantics".* (Source: https://stackoverflow.com/a/4421708/364696 , though the principle is restated in many other places).

Agreed with this. What is so useful exactly in this new dict operator that it hasn't been implemented, say, 20 years ago? I rarely find myself merging dicts and, when I do, calling dict.update() is entirely acceptable (I think the "{**d}" notation was already a mistake, making a perfectly readable operation more cryptic simply for the sake of saving a few keystrokes). Built-in operations should be added with regard to actual user needs (such as: a first-class notation for matrix multiplication, making formulas easier to read and understand), not a mere "hmm this might sometimes be useful". Besides, if I have two dicts with e.g. lists as values, I *really* dislike the fact that the + operator will clobber the values rather than concatenate them. It's a recipe for confusion. Regards Antoine.

On Mon, 4 Mar 2019 15:57:38 -0800 Guido van Rossum wrote:

...

Those two points make me uncomfortable with "+=" strictly behaving like ".update()".

And yet that's how it works for lists. (Note that dict.update() still has capabilities beyond +=, since you can also invoke it with keyword args.)

Yeah, well.... I do think "+=" for lists was a mistake. I *still* have trouble remembering the exact difference between "list +=" and "list.extend" (yes, there is one: one accepts more types than the other... which one it is, and why, I never remember; and, of course, there might be the obscure performance difference because of CPython's execution details). I should not have to remember whether I want to use "list +=" or "list.extend" every time I need to extend a list. There is a virtue to """There should be one-- and preferably only one --obvious way to do it""" and we shouldn't break it more than we already did. Regards Antoine.

On Fri, Mar 15, 2019 at 12:20:21PM +0100, Antoine Pitrou wrote:

Agreed with this. What is so useful exactly in this new dict operator that it hasn't been implemented, say, 20 years ago?

One could say the same thing about every new feature. Since Python 1.5 was so perfect, why add Unicode, decorators, matrix multiplication, async, descriptors, Decimal, iterators, ... Matrix multiplication is a perfect example: adding the @ operator could have been done in Python 0.1 if anyone had thought of it, but it took 15 years of numerical folk "whinging" about the lack until it happened: https://mail.python.org/pipermail/python-ideas/2014-March/027053.html In some ways, it is often easier to get community buy-in for *big* changes, provided they are backwards compatible. With a big change, people often either want it, or don't care one way or another. (Sometimes because the big change is too big or complicated or difficult for them to understand -- I feel that way about async. Some day I'll deal with it, but right now it's so far down my list of priorities that I have no opinion on anything to do with async.) But *little* changes are easy enough for everyone to understand, and so they trigger the impulse to bike-shed. Everyone has an opinion on whether or not dicts should support an update operator, and whether to spell it + or | or <- or << or something else. Or the infamous := operator, which ultimately is a useful but minor syntactic and semantic change but generated a huge amount of debate, argument and negativity. A far smaller change to the language than adding type hinting, but it generated far more argument. I still remember being told in no uncertain terms by the core devs that adding a clear() method to lists was a waste of time because there was already a perfectly good way to spell it with slicing. And then ABCs came along and now lists have a clear method. So opinions change too. Things happen when they happen, because if they had happened earlier we wouldn't still be arguing about them.

I rarely find myself merging dicts and, when I do, calling dict.update() is entirely acceptable

The code we write is shaped by the operators and methods that exist. You use dict.update() because *it exists* so when you want a new dict merged with another, you write the code that is possible today: new = spam.copy() new.update(eggs) process(new) and you are content because you "rarely find myself merging dicts". But perhaps those who *frequently* merge dicts have a different option, and would prefer to write one line rather than three and avoid naming something that doesn't need a name: process(spam + eggs) # or spam|eggs if you prefer

(I think the "{**d}" notation was already a mistake, making a perfectly readable operation more cryptic simply for the sake of saving a few keystrokes).

I don't know if it was a mistake, but disatisfaction with its lack of readability and discoverability is one of the motivations of this PEP. [...]

Besides, if I have two dicts with e.g. lists as values, I *really* dislike the fact that the + operator will clobber the values rather than concatenate them. It's a recipe for confusion.

Are you confused that the update method clobbers list values rather than concatenate them? I doubt that you are. So why would it be confusing to say that + does a copy-and-update? (In any case, popular opinion may be shifting towards preferring the | operator over + so perhaps confusion over concatenation may not be an issue in the future.) -- Steven

On Fri, Mar 15, 2019 at 12:25:22PM +0100, Antoine Pitrou wrote:

Yeah, well.... I do think "+=" for lists was a mistake. I *still* have trouble remembering the exact difference between "list +=" and "list.extend" (yes, there is one: one accepts more types than the other... which one it is, and why, I never remember;

Both accept arbitrary iterables, and the documentation suggests that they are the same: https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types Perhaps you are thinking of the difference between list + list versus list += iterable? [...]

There is a virtue to

"""There should be one-- and preferably only one --obvious way to do it"""

"It" here refers to two different things: "I want to update a dict in place": The Obvious Way is to use the update method; the fact that += works as well is just a side-effect of the way augmented assignments are defined. "I want a new dict that merges two existing dicts": The Obvious Way is to use the merge operator (possibly spelled + but that's not written in stone yet). -- Steven

On Fri, Mar 15, 2019 at 12:34:45PM +0100, Antoine Pitrou wrote:

On Thu, 7 Mar 2019 10:58:02 +1100 Chris Angelico wrote:

...

Lots of words that basically say: Stuff wouldn't be perfectly pure.

Chris, please learn to think twice before contributing what is essentially a trivialization of someone else's arguments. You're not doing anything useful here, and are just sounding like an asshole who wants to shut people up.

I don't think you are being fair here, and I'd rather avoid getting into unhelpful arguments about tone and whether Chris is "trivializing" (a perjorative term) or "simplifying" (a more neutral term) Josh's position. But if you feel that Chris (and I) have missed parts of Josh's argument, then by all means point out what we missed. Josh, the same applies to you: I do want to give your objections a fair hearing in the updated PEP, so if you think I've missed something, please point it out. In context, I think Chris' response was valid: he was responding to a post by Josh whose entire argument was that using + for dict merging is an abuse of the + symbol because it isn't like numeric addition. If there is more to Josh's argument, can you point out to me what I have missed please? That's a genuine request, not a rhetorical question. Here's Josh's argument: https://mail.python.org/pipermail/python-ideas/2019-March/055733.html and for context, here is Chris' dismissal of Josh's argument: https://mail.python.org/pipermail/python-ideas/2019-March/055734.html and his explanation of why he is dismissing it. Chris is well within his right to dismiss an argument that doesn't impress him, which he did by summarizing it as "Stuff wouldn't be perfectly pure". (Pure in the sense of being like numeric addition.) I think that's pretty much an accurate summary: Josh apparently doesn't like using + for anything that isn't purely like + for real numbers. He calls using + for concatentation a "minor abuse" of the operator and argues that it would be bad for dict meging to use + because merging has different properties to numeric addition. (He has also criticised the use of + for concatenation in at least one other post.) He even gives qualified support for a dict merge operator: "there's nothing wrong with making dict merges easier" but just doesn't like the choice of + as the operator. He's entitled to his opinion, and Chris is entitled to dismiss it. (Aside: your email appears to have broken threading. I'm not sure why, your other emails seem to be threaded okay.) -- Steven

On Fri, Mar 15, 2019 at 4:36 AM Antoine Pitrou wrote:

On Thu, 7 Mar 2019 10:58:02 +1100 Chris Angelico wrote:

...

Lots of words that basically say: Stuff wouldn't be perfectly pure.

Chris, please learn to think twice before contributing what is essentially a trivialization of someone else's arguments. You're not doing anything useful here, and are just sounding like an asshole who wants to shut people up.

Watch the tone please.

On Fri, Mar 15, 2019 at 11:15 AM Steven D'Aprano wrote:

On Fri, Mar 15, 2019 at 10:34:45AM -0700, Brett Cannon wrote:

...

Watch the tone please.

Brett, you might have missed my comment about wanting to avoid unhelpful arguments about tone, but if you are going to complain about people's tone, the considerate thing to do is to say what it is that you're objecting to.

The phrasing of "just sounding like an asshole who wants to shut people up" is unnecessary.

Otherwise we're left guessing as to what it is and whether or not you are making an implied threat to apply the CoC.

No implied "threat". If it was an official warning then I would have said so.

I responded to Antoine's post earlier, but thought that it was a respectful disagreement. Do you think that's not the case?

I think it skirts the edge of being disrespectful, hence the request to please be aware of how one comes across.

On Sat, Mar 16, 2019 at 09:04:22PM +1300, Greg Ewing wrote:

Another random thought about this: Mathematicians use addition as a metaphor for quite a range of different things, but they tend to only use the symbols ∪ and ∩ for actual sets, or things that are very set-like. So maybe that's an argument for using '+' rather than '|' for dict merging.

If one views an ordered dict as an assoc list, '+' would mean prepending the new values to the existing ones. If one views an unordered dict as a set of ordered pairs, '|' would make sense. Stefan Krah

On Sat, 16 Mar 2019 01:59:07 +1100 Steven D'Aprano wrote:

On Fri, Mar 15, 2019 at 12:25:22PM +0100, Antoine Pitrou wrote:

...

Yeah, well.... I do think "+=" for lists was a mistake. I *still* have trouble remembering the exact difference between "list +=" and "list.extend" (yes, there is one: one accepts more types than the other... which one it is, and why, I never remember;

Both accept arbitrary iterables, and the documentation suggests that they are the same:

https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types

Perhaps you are thinking of the difference between list + list versus list += iterable?

Hmm, it looks like I misremembered indeed. Thanks for correcting this. Regards Antoine.

On Sat, Mar 16, 2019 at 12:39 AM Antoine Pitrou wrote:

On Sat, 16 Mar 2019 01:41:59 +1100 Steven D'Aprano wrote:

...

Matrix multiplication is a perfect example: adding the @ operator could have been done in Python 0.1 if anyone had thought of it, but it took 15 years of numerical folk "whinging" about the lack until it happened:

Not so perfect, as the growing use of Python for scientific computing has made it much more useful to promote a dedicated matrix multiplication operator than, say, 15 or 20 years ago.

Theres more to it than that, really, but not really relevant here...

This is precisely why I worded my question this way: what has changed in the last 20 years that make a "+" dict operator more compelling today than it was? Do we merge dicts much more frequently than we did?

The analogy doesn't hold because @ was a new operator -- a MUCH bigger change than dimply defining the use of + (or | ) for dicts. I wouldn't mind the new operator if its meaning was clear-cut. But

here we have potential for confusion, both for writers and readers of code.

but it's NOT a new operator, it is making use of an existing one, and sure you could guess at a couple meanings, but the merge one is probably one of the most obvious to guess, and one quick test and you know -- I really can't see it being a ongoing source of confusion. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Mar 21, 2019 at 10:35 PM Antoine Pitrou wrote:

...

but it's NOT a new operator, it is making use of an existing one, and sure you could guess at a couple meanings, but the merge one is probably one of the most obvious to guess, and one quick test and you know -- I really can't see it being a ongoing source of confusion.

Did you actually read what I said? The problem is not to understand what dict.__add__ does. It's to understand what code using the + operator does, without knowing upfront whether the inputs are dicts.

The + operator adds two things together. I don't understand the issue here. You can add integers: 1 + 2 == 3 You can add floats: 0.5 + 1.25 == 1.75 You can add lists: [1,2] + [3,4] == [1,2,3,4] You can add strings: "a" + "b" == "ab" And soon you'll be able to add dictionaries. The exact semantics need to be defined, but it's not fundamentally changing how you interpret the + operator. I don't understand the panic here - or rather, I don't understand why it's happening NOW, not back when lists got the ability to be added (if that wasn't in the very first release). Conversely, if it's the | operator, it's a matter of merging, and the same is true. You can merge integers, treating them as bit sets. You can merge sets. And now you'll be able to merge dictionaries. Same same. ChrisA

On Thu, Mar 21, 2019 at 11:45 PM Antoine Pitrou wrote:

On Thu, 21 Mar 2019 23:35:36 +1100 Chris Angelico wrote:

...

On Thu, Mar 21, 2019 at 10:35 PM Antoine Pitrou wrote:

...

...

but it's NOT a new operator, it is making use of an existing one, and sure you could guess at a couple meanings, but the merge one is probably one of the most obvious to guess, and one quick test and you know -- I really can't see it being a ongoing source of confusion.

Did you actually read what I said? The problem is not to understand what dict.__add__ does. It's to understand what code using the + operator does, without knowing upfront whether the inputs are dicts.

The + operator adds two things together. I don't understand the issue here.

I'm not expecting you to understand, either.

... then, in the interests of productive discussion, could you please explain? What is it about dict addition that makes it harder to understand than other addition? ChrisA

On 21/03/2019 11:34, Antoine Pitrou wrote:

On Wed, 20 Mar 2019 15:46:24 -1000 Christopher Barker wrote:

...

...

This is precisely why I worded my question this way: what has changed in the last 20 years that make a "+" dict operator more compelling today than it was? Do we merge dicts much more frequently than we did?

The analogy doesn't hold because @ was a new operator -- a MUCH bigger change than dimply defining the use of + (or | ) for dicts.

But it's less disruptive when reading code, because "x @ y" is unambiguous: it's a matrix multiplication. "x + y" can be many different things, and now it can be one more thing.

"x @ y" is unambiguous once you know what it means. Until then, it's just mysterious.

...

I wouldn't mind the new operator if its meaning was clear-cut. But

...

here we have potential for confusion, both for writers and readers of code.

but it's NOT a new operator, it is making use of an existing one, and sure you could guess at a couple meanings, but the merge one is probably one of the most obvious to guess, and one quick test and you know -- I really can't see it being a ongoing source of confusion.

Did you actually read what I said? The problem is not to understand what dict.__add__ does. It's to understand what code using the + operator does, without knowing upfront whether the inputs are dicts.

Welcome to polymorphism. -- Rhodri James *-* Kynesim Ltd

On Fri, Mar 22, 2019 at 12:17 AM Serhiy Storchaka wrote:

21.03.19 14:51, Chris Angelico пише:

...

... then, in the interests of productive discussion, could you please explain? What is it about dict addition that makes it harder to understand than other addition?

Currently the + operator has 2 meanings for builtin types (both are widely used), after adding it for dicts it will have 3 meanings.

3 > 2, is not?

I suppose you could call it two (numeric addition and sequence concatenation), but there are subtleties to the way that lists concatenate that don't apply to strings (esp since lists are mutable), so I'd call it at least three already. And what about non-builtin types? You can add two numpy arrays and it does pairwise addition, quite different from how lists add together. But in every case, the + operator means "add these things together". It will be the same with dicts: you add the dicts together. Antoine has stated that the problem is NOT understanding what dict.__add__ does, so I am at a loss as to what the problem IS. We *already* have many different definitions of "add", according to the data types involved. That is exactly what polymorphism is for. Why is it such a bad thing for a dict? Now, my own opinion is that | would be a better operator than +, but it's only a weak preference, and I'd be happy with either. Also, to my understanding, the concerns about "what does addition mean" apply identically to "what does Or mean", but as we've already seen, my understanding doesn't currently extend as far as comprehending this issue. Hence asking. ChrisA

21.03.19 15:24, Chris Angelico пише:

On Fri, Mar 22, 2019 at 12:17 AM Serhiy Storchaka wrote:

...

21.03.19 14:51, Chris Angelico пише:

...

... then, in the interests of productive discussion, could you please explain? What is it about dict addition that makes it harder to understand than other addition?

Currently the + operator has 2 meanings for builtin types (both are widely used), after adding it for dicts it will have 3 meanings.

3 > 2, is not?

I suppose you could call it two (numeric addition and sequence concatenation), but there are subtleties to the way that lists concatenate that don't apply to strings (esp since lists are mutable), so I'd call it at least three already.

I do not understand what are these subtleties that you treat list concatenation different from string concatenation. Could you please explain? In any case, it does not matter how you count meanings, n + 1 > n.

I'd like to make a plea to people: I get it, there is now significant opposition to using the + symbol for this proposed operator. At the time I wrote the first draft of the PEP, there was virtually no opposition to it, and the | operator had very little support. This has clearly changed. At this point I don't think it is productive to keep making subjective claims that + will be more confusing or surprising. You've made your point that you don't like it, and the next draft^1 of the PEP will make that clear. But if you have *concrete examples* of code that currently is easy to understand, but will be harder to understand if we add dict.__add__, then please do show me! For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled. And to those who support this PEP, code examples where a dict merge operator will help are most welcome! ^1 Coming Real Soon Now™. -- Steven

Le 21 mars 2019 à 17:43:31, Steven D'Aprano (steve@pearwood.info(mailto:steve@pearwood.info)) a écrit:

I'd like to make a plea to people:

I get it, there is now significant opposition to using the + symbol for this proposed operator. At the time I wrote the first draft of the PEP, there was virtually no opposition to it, and the | operator had very little support. This has clearly changed.

At this point I don't think it is productive to keep making subjective claims that + will be more confusing or surprising. You've made your point that you don't like it, and the next draft^1 of the PEP will make that clear.

But if you have *concrete examples* of code that currently is easy to understand, but will be harder to understand if we add dict.__add__, then please do show me!

For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled.

Thanks for the work you are doing on this PEP and for debunking my misconceptions regarding types, I’m currently learning a lot about them. I don’t know if it matters but I’m in favor of the method

And to those who support this PEP, code examples where a dict merge operator will help are most welcome!

Not matter the notation you end up choosing, I think this code: https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81 which is part of a widely used library to validate JWTs would greatly benefit from a new merge to merge dicts. (This package is 78 on https://hugovk.github.io/top-pypi-packages/) Rémi

^1 Coming Real Soon Now™.

-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, 21 Mar 2019 at 17:27, Rémi Lapeyre wrote:

Le 21 mars 2019 à 17:43:31, Steven D'Aprano (steve@pearwood.info(mailto:steve@pearwood.info)) a écrit:

...

I'd like to make a plea to people:

I get it, there is now significant opposition to using the + symbol for this proposed operator. At the time I wrote the first draft of the PEP, there was virtually no opposition to it, and the | operator had very little support. This has clearly changed.

At this point I don't think it is productive to keep making subjective claims that + will be more confusing or surprising. You've made your point that you don't like it, and the next draft^1 of the PEP will make that clear.

But if you have *concrete examples* of code that currently is easy to understand, but will be harder to understand if we add dict.__add__, then please do show me!

For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled.

Thanks for the work you are doing on this PEP and for debunking my misconceptions regarding types, I’m currently learning a lot about them.

I don’t know if it matters but I’m in favor of the method

...

And to those who support this PEP, code examples where a dict merge operator will help are most welcome!

Not matter the notation you end up choosing, I think this code: https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81

which is part of a widely used library to validate JWTs would greatly benefit from a new merge to merge dicts. (This package is 78 on https://hugovk.github.io/top-pypi-packages/)

It's already got a function that does the job. How much benefit is there *really* from being able to replace it with d1 + d2 once you drop support for Python < 3.8? But point taken that new code would have been able to avoid the function in the first place. ... or would it? def merge_dict(original, updates): if not updates: return original With the + operator. d1 + None will fail with an error. With your code, updates=None means "return the original unchanged". Does that matter with your current code? The point is that in many real world cases, you'd write a function *anyway*, to handle corner cases, and a new operator doesn't make much difference at that point. Having said all of that, I'm mostly indifferent to the idea of having a built in "dictionary merge" capability - I doubt I'd use it *much*, but if it were there I'm sure I'd find useful it on the odd occasion. I'm somewhat against an operator, I really don't see why this couldn't be a method (although the asymmetry in d1.merge(d2) makes me have a mild preference for a class method or standalone function). I can't form an opinion between + and |, I find | significantly uglier (I tend to avoid using it for sets, in favour of the union method) but I am mildly uncomfortable with more overloading of +. Serious suggestion - why not follow the lead of sets, and have *both* an operator and a method? And if you think that's a bad idea, it would be worth considering *why* it's a bad idea for dictionaries, when it's OK for sets (and "well, I didn't like it when sets did it" isn't sufficient ;-)) And having said that, I'll go back to lurking and not really caring one way or the other. Paul

On 21/03/2019 17:06, Antoine Pitrou wrote:

On Fri, 22 Mar 2019 03:42:00 +1100 Steven D'Aprano wrote:

...

For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled.

I'd rather see a method. Dict merging just doesn't occur often enough that an operator is desirable for it.

Analogous to the relationship between list.sort() and sorted(), I can't help but think that a dict.merge() method would be a terrible idea. A merged() function is more defensible.

...

And to those who support this PEP, code examples where a dict merge operator will help are most welcome!

I don't use Python often enough to have much to offer, I'm afraid. The sort of occasion I would use dict merging is passing modified environments to subcommands. Something like: def process(): if time_to_do_thing1(): thing1(base_env + thing1_env_stuff + env_tweaks) if time_to_do_thing2(): thing2(base_env + thing2_env_stuff + env_tweaks) ...and so on. The current syntax for doing this is a tad verbose: def process(): if time_to_do_thing1(): env = base_env.copy() env.update(thing1_env_stuff) env.update(env_tweaks) thing1(env) del env if time_to_do_thing2(): env = base_env.copy() env.update(thing2_env_stuff) env.update(env_tweaks) thing2(env) del env -- Rhodri James *-* Kynesim Ltd

Brandt Bucher wrote:

Before:

tmp = keep.copy() tmp.update(separate) result = function(param=tmp) del tmp

After:

result = f(param=keep+separate)

I'd rewrite this example as: Before: fn(param={**keep, **separate}) After: fn(param=keep + separate) -- Jonathan

On 3/21/19 1:01 PM, Jonathan Fine wrote:

Rémi Lapeyre wrote:

...

Not matter the notation you end up choosing, I think this code: https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81 [...] would greatly benefit from a new merge to merge dicts.

I've looked at the merge_dict defined in this code. It's similar to

def gapfill(self, other):

# See also: https://cobrapy.readthedocs.io/en/latest/gapfilling.html # Cobra's gapfill adds items to a model, to meet a requirement.

for key in other.keys(): if key not in self: self[key] = other[key]

(This is code I've written, that's not yet on PyPi.) The usage is different. Instead of writing one of aaa = merge_dict(aaa, bbb) ccc = merge_dict(aaa, bbb) you write one of gapfill(aaa, bbb) aaa.gapfill(bbb) # If gapfill added to dict methods.

With merge_dict, you never really know if ccc is the same object as aaa, or a different one. Sometimes this is important.

With gapfill, you get the same behaviour as the already familiar and loved dict.update. But of course with a different merge rule.

With gapfill, I can never remeber whether it's gapfill(aaa, bbb) or gapfill(bbb, aaa). This is always important. :-) At least with aaa.gapfill(bbb), I have some sense of the "direction" of the asymmetry, or I would if I had some frame of reference into which to put the "gapfill" operation. (With the proposed + or | operator syntax, that gets lost.)

I'd also like to add what I consider to be another point in favor of an operator: Throughout all of these related threads, I have seen many typos and misspellings of current dict merging idioms, from messing up the number of asterisks in "{**a, **b}", to even Guido(!) accidentally writing the common copy/update idiom as d = d1.copy() d = d1.update(d2) in a thoughtful email... and it was then copied-and-pasted (unquoted and verbatim) by others! I still have yet to see somebody (even those who claim to be confused by it) mess up the PEP's current definition of "+" or "+=" in this context. Brandt

Antoine Pitrou wrote:

Note that, if you're able to live with a third-party dependency, the `toolz` package has what you need (and lots of other things too): https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.merge

I suggest that the supporters of dict + dict make (and put up on PyPi) a pure-Python subclass of dict that has the desired properties. This would 1. Clarify and document the syntax and semantics. 2. Help with exploration and testing. 3. Provide a 'back-port' mechanism to current Python. 4. Give the proposal the benefit of practical experience. I find this last very important, when we can do it. And we can, in this case. Language changes are 'cast in stone' and hard to reverse. And afterwards, on this list, we're sometime told that we've 'missed the boat' for a particular change. Let's take the benefit of a reference pure Python implementation, when we can. Steven D'A. Please would you include or respond to this suggestion, in the next revision of the PEP. -- Jonathan

I dislike the symbol '+' to mean "dictionary merging with value updates." I have no objection to, and mildly support, adding '|' with this meaning. It's not really possible to give "that one example" where + for meeting makes code less clear... In my eyes it would be EVERY such use. Every example presented in this thread or in the PEP feels wrong to me. I know about operator overloading and dunder methods and custom classes. My intuition about '+' from math, other programming languages, and Python, simply does not lead me to expect the proposed meaning. On Thu, Mar 21, 2019, 12:43 PM Steven D'Aprano wrote:

I'd like to make a plea to people:

I get it, there is now significant opposition to using the + symbol for this proposed operator. At the time I wrote the first draft of the PEP, there was virtually no opposition to it, and the | operator had very little support. This has clearly changed.

At this point I don't think it is productive to keep making subjective claims that + will be more confusing or surprising. You've made your point that you don't like it, and the next draft^1 of the PEP will make that clear.

But if you have *concrete examples* of code that currently is easy to understand, but will be harder to understand if we add dict.__add__, then please do show me!

For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled.

And to those who support this PEP, code examples where a dict merge operator will help are most welcome!

^1 Coming Real Soon Now™.

-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Mar 22, 2019 at 3:42 AM Steven D'Aprano wrote:

And to those who support this PEP, code examples where a dict merge operator will help are most welcome!

Since Python examples don't really exist yet, I'm reaching for another language that DOES have this feature. Pike's mappings (broadly equivalent to Python's dicts) can be added (actually, both + and | are supported), with semantics equivalent to PEP 584's. Translated into Python syntax, here's a section from the implementation of Process.run(): def run(cmd, modifiers={}): ... ... p = Process(cmd, modifiers + { "stdout": mystdout->pipe(), "stderr": mystderr->pipe(), "stdin": mystdin->pipe(), }) In Val.TimeTZ, a subclass that adds a timezone attribute overrides a mapping-returning method to incorporate the timezone in the result mapping. Again, translated into Python syntax: def tm(self): return super().tm() + {"timezone": self.timezone} To spawn a subprocess with a changed environment variable: //from the Process.create_process example Process.create_process(({ "/usr/bin/env" }), (["env" : getenv() + (["TERM":"vt100"]) ])); # equivalent Python code subprocess.Popen("/usr/bin/env", env=os.environ + {"TERM": "vt100"}) All of these examples could be done with the double-star syntax, as they all use simple literals. But addition looks a lot cleaner IMO, and even more so if you're combining multiple variables rather than using literals. ChrisA

On Thu, Mar 21, 2019 at 03:10:48PM -0700, Brandt Bucher wrote:

For anyone interested in "trying it out": if you're not against cloning and compiling CPython yourself, here is a PEP 584 C implementation I have PR'd against master right now. I'm keeping it in sync with the draft PEP as it changes, so subtraction performance is not overly optimized yet, but it will show you the *exact* behavior outlined in the PEP on the dict builtin and its subclasses. The relevant branch is called "addiction". You can clone it from:

That's great, thank you! For the sake of comparisons, could you support | as an alias? That will allow people to get a feel for whether a+b or a|b looks nicer. (For the record, the PEP isn't set in stone in regards to the choice of operator.

https://github.com/brandtbucher/cpython.git

-- Steven

On 3/21/19 6:46 PM, Steven D'Aprano wrote:

Antoine and Serhiy seem to worry that there are existing uses of + which are currently easy to understand but will become less so if dict.__add__ is added. I respect that worry, even if I doubt that they are correct.

If someone can demonstrate that their fear is well-founded, that would be an excellent counter-argument to the PEP's proposal to use +.

https://docs.python.org/3.8/library/collections.html has some examples using collections.Counter, which is clearly described as being a subclass of dict. Amongst the examples: c + d # add two counters together: c[x] + d[x] That's the + operator operating on two dicts (don't make me quote the Liskov Substitution Principle), but doing something really different than the base operator. So if I know that c and d (or worse, that one of them) is a dict, then interpreting c + d becomes much more interesting, but arguably no worse than c.update(d). Yes, it's "just" polymorphism, but IMO it violates the Principle of Least Surprise. My apologies if this is covered elsewhere in this thread, or it doesn't meet the bar Steven set.