Accepting PEP 584: Add Union Operators To dict
Now that the last bits of discussion about PEP 584 have been settled (we will *not* be calling the `copy` or `update` methods) I am accepting this PEP. Congratulations Steven and Brandt! Thanks to everyone else who gave their opinion. Formally, I will just send my recommendation to accept the PEP to the Steering Council -- they will then let us know whether they agree, and once that's done we can update the PEP with the "Accepted" status and land the implementation (https://github.com/python/cpython/pull/12088). (Hm, the PEP should probably also link to that PR rather than to Brandt's private branch.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
17.02.20 23:02, Guido van Rossum пише:
Now that the last bits of discussion about PEP 584 have been settled (we will *not* be calling the `copy` or `update` methods) I am accepting this PEP. Congratulations Steven and Brandt!
Thanks to everyone else who gave their opinion.
Formally, I will just send my recommendation to accept the PEP to the Steering Council -- they will then let us know whether they agree, and once that's done we can update the PEP with the "Accepted" status and land the implementation (https://github.com/python/cpython/pull/12088). (Hm, the PEP should probably also link to that PR rather than to Brandt's private branch.)
Comments to the PEP. In the disadvantages of existing syntax `{**d1, **d2}` (https://www.python.org/dev/peps/pep-0584/#d1-d2) it claims:
{**d1, **d2} ignores the types of the mappings and always returns a dict. type(d1)({**d1, **d2}) fails for dict subclasses such as defaultdict that have an incompatible __init__ method.
But it was decided that `d1 | d2` also should ignore the types of the operands and always return a dict. And it accepts only dicts, not general mappings, in difference to `{**d1, **d2}`. So the only disadvantage of `{**d1, **d2}` is that it is not well known and "looks ugly". But it supports general mappings. There is a similar note about ChainMap (https://www.python.org/dev/peps/pep-0584/#collections-chainmap)
Like dict unpacking, it is tricky to get it to honor the desired subclass. For the same reason, type(d1)(ChainMap(d2, d1)) fails for some subclasses of dict.
which is not relevant to the final implementation of PEP 584. The pure-Python implementation of the non-inplace operator can be simpler if use dict unpacking: def __or__(self, other): if not isinstance(other, dict): return NotImplemented return {**self, **other} def __ror__(self, other): if not isinstance(other, dict): return NotImplemented return {**other, **self} I suggest to include to objections that non-inplace merging of two dicts is much less common operation than concatenating of two strings or other sequences, and that the existing syntax `{**d1, **d2}` completely satisfies the need. The in-place operator is completely equivalent to dict.update(). So the possible benefit of adding support of two operators for dict is much less than common bar for changes in the builtin types. This is why I still -0 for this feature. In the initial form it contradicted the behavior of other builtin types and operators, but after fixing contradictions it became pretty useless.
I am accepting this PEP. Congratulations Steven and Brandt!
Thank you for your guidance, especially the suggestions late last year. And thanks Steven for taking me on as a co-author and shaping the bulk of the proposal.
Hm, the PEP should probably also link to that PR rather than to Brandt's private branch.
Agreed. This can probably be changed when the status is updated. Now, addressing Serhiy's points :)...
...it was decided that `d1 | d2` also should ignore the types of the operands and always return a dict. And it accepts only dicts, not general mappings, in difference to `{**d1, **d2}`. So the only disadvantage of `{**d1, **d2}` is that it is not well known and "looks ugly".
Not quite. While this point *has* been weakened a bit with the recent semantic change, you don't mention that `dict` subclasses can (and likely would) override the `__or__` trio with wrapped `super()` calls. So while `{**d1, **d2}` will *never* be anything but a `dict`, I can trust that well-written `dict` subclasses my code encounters will still be able to preserve themselves with `d1 | d2`, if desired.
The pure-Python implementation of the non-inplace operator can be simpler if use dict unpacking.
The purpose of the pure-Python implementation in the PEP is to be a clear, understandable example that serves as a codification of the semantics, not to be the shortest one-liner. The suggested changes make the code less readable, making it harder to see exactly what will happen if I do `self | other`. I'm against changing it, for that reason.
I suggest to include to objections that non-inplace merging of two dicts is much less common operation than concatenating of two strings or other sequences, and that the existing syntax {**d1, **d2} completely satisfies the need.
This is really two objections. Regarding the commonality of this operation, we've provided eighteen detailed examples of third-party library code that are candidates for these new operators. To the best of my knowledge, a large survey like this is unprecedented in a PEP. Whether or not it is more common than a different operation on other types isn't relevant here. We have gone above and beyond in demonstrating the use cases in detail. A rewording of your second objection has already been addressed: https://www.python.org/dev/peps/pep-0584/#more-than-one-way-to-do-it I feel that I've adequately responded to the other points you've raised here in this email (please let me know if I missed anything, and perhaps Steven may want to chime in as well). I don't know what the policy for changing the PEP is once it's gotten to this stage, but it's already grown to a huge size in its attempts to address every possible concern that has arisen over the past year (even some that, in my opinion, weren't serious proposals that needed to be acknowledged). I'd prefer not to bloat it even more this late in the game, if at all possible.
On Tue, Feb 18, 2020 at 9:40 AM Brandt Bucher <brandtbucher@gmail.com> wrote:
Now, addressing Serhiy's points :)...
...it was decided that `d1 | d2` also should ignore the types of the operands and always return a dict. And it accepts only dicts, not general mappings, in difference to `{**d1, **d2}`. So the only disadvantage of `{**d1, **d2}` is that it is not well known and "looks ugly".
Not quite. While this point *has* been weakened a bit with the recent semantic change, you don't mention that `dict` subclasses can (and likely would) override the `__or__` trio with wrapped `super()` calls. So while `{**d1, **d2}` will *never* be anything but a `dict`, I can trust that well-written `dict` subclasses my code encounters will still be able to preserve themselves with `d1 | d2`, if desired.
In particular, you may want to update the implementations of defaultdict and ordereddict. (are there others?) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
In particular, you may want to update the implementations of defaultdict and ordereddict. (are there others?)
I have a checklist for PEP 584 follow-up PRs I'm planning to make after (and if) it is accepted. typeshed stubs, docs, and `defaultdict` were on it. I'll add `OrderedDict` as well (for some reason I didn't think it was a `dict` subclass). `Counter` already defines this operator for other purposes, so no need to do anything there. I'm not aware of other public subclasses, but I may just try importing the whole stdlib and recursing through `dict.__subclasses__()` just to be sure.
Just to follow up on this, here are the subclasses I've found. Should be updated: - collections.OrderedDict - collections.defaultdict - http.cookies.BaseCookie - http.cookies.Morsel - http.cookies.SimpleCookie Don’t think so: - typing.TypedDict Already defines the operator: - collections.Counter Not Public (or at least not documented): - _strptime.TimeRE - builtins.StgDict (this one's only available internally in _ctypes). - email._encoded_words._QByteMap - enum._EnumDict - logging.config.ConvertingDict - multiprocessing.pool._PoolCache - urllib.parse.Quoter
On Sun., 23 Feb. 2020, 5:32 am Brandt Bucher, <brandtbucher@gmail.com> wrote:
Just to follow up on this, here are the subclasses I've found.
Should be updated: - collections.OrderedDict - collections.defaultdict - http.cookies.BaseCookie - http.cookies.Morsel - http.cookies.SimpleCookie
They're not dict subclasses, but we'll also want to look at collections.UserDict, collections.ChainMap, and types.MappingProxyType. Finally, while the PEP quite reasonably doesn't propose making __or__ a *required* part of the mutable mapping API, collections.Mapping and collections.MutableMapping *could* provide concrete method implementations that make subclasses behave in a way that's similar to built-in dicts by default. Cheers, Nick.
we'll also want to look at collections.UserDict, collections.ChainMap, and types.MappingProxyType.
UserDict has already been updated in the PR (it has tests that fail if it doesn't have all of dict's methods). I'll look into the others... thanks for reminding me!
collections.Mapping and collections.MutableMapping could provide concrete method implementations that make subclasses behave in a way that's similar to built-in dicts
Hm, haven't thought too much about this (I don't have much experience with the ABCs). Would they return dicts, or call the self.copy and self.update methods? Those are just hypothetical questions for now; I don't necessarily want to dig too far into that discussion again. But I agree that it's definitely worth considering. ;)
On Sat, Feb 22, 2020 at 4:45 PM Brandt Bucher <brandtbucher@gmail.com> wrote:
[quoting Nick]
collections.Mapping and collections.MutableMapping could provide concrete method implementations that make subclasses behave in a way that's similar to built-in dicts
Hm, haven't thought too much about this (I don't have much experience with the ABCs). Would they return dicts, or call the self.copy and self.update methods?
Those are just hypothetical questions for now; I don't necessarily want to dig too far into that discussion again. But I agree that it's definitely worth considering. ;)
Mapping and MutableMapping don't have copy() methods -- these are intentionally narrower interfaces than dict. I think we should leave them alone. The PEP does *not* propose to add `|` to Mapping, and that's intentional. Because of this I think we should also not add `|=` to MutableMapping, even though it does have an update() method. Adding it (even with a default implementation) could cause discrepancies to virtual subclasses registered with MutableMapping.register(C). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Mon, 24 Feb 2020 at 12:10, Guido van Rossum <guido@python.org> wrote:
On Sat, Feb 22, 2020 at 4:45 PM Brandt Bucher <brandtbucher@gmail.com> wrote:
[quoting Nick]
collections.Mapping and collections.MutableMapping could provide concrete method implementations that make subclasses behave in a way that's similar to built-in dicts
Hm, haven't thought too much about this (I don't have much experience with the ABCs). Would they return dicts, or call the self.copy and self.update methods?
Those are just hypothetical questions for now; I don't necessarily want to dig too far into that discussion again. But I agree that it's definitely worth considering. ;)
Mapping and MutableMapping don't have copy() methods -- these are intentionally narrower interfaces than dict. I think we should leave them alone. The PEP does *not* propose to add `|` to Mapping, and that's intentional. Because of this I think we should also not add `|=` to MutableMapping, even though it does have an update() method. Adding it (even with a default implementation) could cause discrepancies to virtual subclasses registered with MutableMapping.register(C).
That's an excellent point regarding virtual subclasses - I was only thinking about concrete subclasses. So if folks want to mimic the full dict API, they can inherit from UserDict, while Mapping and MutableMapping continue with their existing narrower APIs. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Feb 22, 2020 at 11:28 AM Brandt Bucher <brandtbucher@gmail.com> wrote:
Just to follow up on this, here are the subclasses I've found.
Should be updated: - collections.OrderedDict - collections.defaultdict
SGTM. - http.cookies.BaseCookie
- http.cookies.Morsel - http.cookies.SimpleCookie
Are these three cookie classes sufficiently popular that we need to support `|` on them?
Don’t think so: - typing.TypedDict
Agreed, since at runtime this doesn't exist -- calling TypedDict(...) returns a plain dict instance.
Already defines the operator: - collections.Counter
Not Public (or at least not documented): - _strptime.TimeRE - builtins.StgDict (this one's only available internally in _ctypes). - email._encoded_words._QByteMap - enum._EnumDict - logging.config.ConvertingDict - multiprocessing.pool._PoolCache - urllib.parse.Quoter
Yeah, let's stay away from these. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Are these three cookie classes sufficiently popular that we need to support `|` on them?
I don't have much experience with `http`, so I figured I'd open a BPO issue and let the relevant folks make the call. The main reason I'm considering these is that `Morsel` was updated back in 3.5 with `copy` and `update` overrides: https://bugs.python.org/issue2211
On Mon, Feb 24, 2020 at 10:42 AM Brandt Bucher <brandtbucher@gmail.com> wrote:
Are these three cookie classes sufficiently popular that we need to support `|` on them?
I don't have much experience with `http`, so I figured I'd open a BPO issue and let the relevant folks make the call. The main reason I'm considering these is that `Morsel` was updated back in 3.5 with `copy` and `update` overrides:
OK, that removes my objection. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Tue, Feb 18, 2020 at 05:35:38PM -0000, Brandt Bucher wrote:
I am accepting this PEP. Congratulations Steven and Brandt!
Thank you for your guidance, especially the suggestions late last year. And thanks Steven for taking me on as a co-author and shaping the bulk of the proposal.
Thank you Guido, and thank you Brandt for volunteering to do the heavy lifting with the proof of concept code etc. I never could have done this without you. [...]
I feel that I've adequately responded to the other points you've raised here in this email (please let me know if I missed anything, and perhaps Steven may want to chime in as well).
No, I'm good. Thank you to everyone for your constructive criticism and suggestions. -- Steven
On Sat, Feb 22, 2020 at 10:09 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Feb 18, 2020 at 05:35:38PM -0000, Brandt Bucher wrote:
I am accepting this PEP. Congratulations Steven and Brandt!
Thank you for your guidance, especially the suggestions late last year. And thanks Steven for taking me on as a co-author and shaping the bulk of the proposal.
Thank you Guido, and thank you Brandt for volunteering to do the heavy lifting with the proof of concept code etc. I never could have done this without you.
You're welcome. Thanks for providing the initial impetus to make this happen! And thanks a lot to Brandt for his implementation work. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
18.02.20 19:35, Brandt Bucher пише:
...it was decided that `d1 | d2` also should ignore the types of the operands and always return a dict. And it accepts only dicts, not general mappings, in difference to `{**d1, **d2}`. So the only disadvantage of `{**d1, **d2}` is that it is not well known and "looks ugly".
Not quite. While this point *has* been weakened a bit with the recent semantic change, you don't mention that `dict` subclasses can (and likely would) override the `__or__` trio with wrapped `super()` calls. So while `{**d1, **d2}` will *never* be anything but a `dict`, I can trust that well-written `dict` subclasses my code encounters will still be able to preserve themselves with `d1 | d2`, if desired.
Yes, but this is a different thing. You *must* to implement new `__or__` in a subclass to make the behavior of `d1 | d2` be different from `{**d1, **d2}`. Actually, there was already an example -- Counter. It is not an argument for implementing `__or__` in dict. It is an argument for implementing `__or__` in other classes. So to add a value to it you have to look at from from the opposite side: you want an operation for merging OrderedDict, defaultdict, etc and add `dict.__or__` just for symmetry, even if it is not actually necessary for dicts.
The pure-Python implementation of the non-inplace operator can be simpler if use dict unpacking.
The purpose of the pure-Python implementation in the PEP is to be a clear, understandable example that serves as a codification of the semantics, not to be the shortest one-liner. The suggested changes make the code less readable, making it harder to see exactly what will happen if I do `self | other`. I'm against changing it, for that reason.
Agree, the current code makes easier to implement such operations in other mapping classes.
Regarding the commonality of this operation, we've provided eighteen detailed examples of third-party library code that are candidates for these new operators. To the best of my knowledge, a large survey like this is unprecedented in a PEP. Whether or not it is more common than a different operation on other types isn't relevant here. We have gone above and beyond in demonstrating the use cases in detail.
Let's take a closer look at these examples. Unfortunately the PEP does not provide links to the original code, so the context is not always known. IPython/zmq/ipkernel.py IPython/zmq/kernelapp.py matplotlib/delaunay/triangulate.py numpy/ma/core.py praw/internal.py pygments/lexer.py sphinx/highlighting.py sphinx/quickstart.py sympy/abc.py sympy/parsing/maxima.py sympy/printing/ccode.py and sympy/printing/fcode.py In almost all examples it is guaranteed that the result is a dict. You create a dict by calling dict() or using dict display and then update it. So the form `{**d1, **d2}` can be used in all these cases. In some examples, like in praw/internal.py, it may be especially good: data = {'name': six.text_type(user), 'type': relationship, **kwargs} In some cases, like in sympy/abc.py the proposed transformation is just not correct if arguments have overridden `__or__`. Currently it is guaranteed that the result is a dict, but with proposed code it is no longer true. matplotlib/legend.py sphinx/domains/__init__.py sphinx/ext/doctest.py sphinx/ext/inheritance_diagram.py In these cases you merge a user provided dict with some default value which is a dict by default, but may be overridden in subclasses. Although it is unlikely. matplotlib/backends/backend_svg.py The current code uses `{**attrib, **extra}`. https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/backends... requests/sessions.py The proposed transformation makes the code less efficient -- it makes redundant copying. sympy/utilities/runtests.py Sorry, but the current code globs = globs.copy() if extraglobs is not None: globs.update(extraglobs) looks much clearer to me than the proposed globs = globs | (extraglobs if extraglobs is not None else {})
On Thu, Feb 27, 2020 at 7:41 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
sympy/utilities/runtests.py
Sorry, but the current code
globs = globs.copy() if extraglobs is not None: globs.update(extraglobs)
looks much clearer to me than the proposed
globs = globs | (extraglobs if extraglobs is not None else {})
Is there a reason for not writing it as: globs = globs | (extraglobs or {}) ? That reads fairly well to me. ChrisA
27.02.20 10:46, Chris Angelico пише:
On Thu, Feb 27, 2020 at 7:41 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
sympy/utilities/runtests.py
Sorry, but the current code
globs = globs.copy() if extraglobs is not None: globs.update(extraglobs)
looks much clearer to me than the proposed
globs = globs | (extraglobs if extraglobs is not None else {})
Is there a reason for not writing it as:
globs = globs | (extraglobs or {})
? That reads fairly well to me.
Sure. Although it would work different if extraglobs is a false value different from None, empty mapping and iterable. But if we ignore such subtle details, it could be written also as globs = {**globs, **(extraglobs or {})}
On Thu, Feb 27, 2020 at 8:15 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
27.02.20 10:46, Chris Angelico пише:
On Thu, Feb 27, 2020 at 7:41 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
sympy/utilities/runtests.py
Sorry, but the current code
globs = globs.copy() if extraglobs is not None: globs.update(extraglobs)
looks much clearer to me than the proposed
globs = globs | (extraglobs if extraglobs is not None else {})
Is there a reason for not writing it as:
globs = globs | (extraglobs or {})
? That reads fairly well to me.
Sure. Although it would work different if extraglobs is a false value different from None, empty mapping and iterable.
Yes, technically it's different. But other than this silently ignoring errors like passing 0 rather than None, it's unlikely to have any material difference.
But if we ignore such subtle details, it could be written also as
globs = {**globs, **(extraglobs or {})}
Yeah, but now I think the pipe syntax has a definite advantage. ChrisA
Thanks for your response Serhiy.
Yes, but this is a different thing. You must to implement new __or__ in a subclass to make the behavior of d1 | d2 be different from {**d1, **d2}.
Really? I'm sorry, but this doesn't feel like a real argument to me. There was no clear compatible path before. Now there is. It works by default, returning a dict for dict subclasses. If you want to change the return type (or do something else), you do have to do a little work. Same for every other overridden thing in any other subclass. Besides, our argument was just the opposite: "you can't implement anything in a subclass to make the type of {**d1, **d2} be different". Now that we've provided a way, it's a problem because you have to define custom behavior (if you want it)?
you want an operation for merging OrderedDict, defaultdict, etc and add dict.__or__ just for symmetry, even if it is not actually necessary for dicts.
Except that now every dict subclass gets this for free. And now everything quacks alike. Are you really suggesting that it would make more sense to implement this for all subclasses, but not dict itself? I don't think you are, but it sure sounds like it.
Let's take a closer look at these examples.
I appreciate that you've taken the time to go through each one, but please note that the PEP finishes with:
The above examples show that sometimes the | operator leads to a clear increase in readability... However other examples using the | operator lead to long, complex single expressions... As with any other language feature, the programmer should use their own judgement about whether | improves their code.
We're not suggesting that all of the examples would be better, or even equivalent with the new code. During the discussions last year, people kept asking for before-and-after examples. We decided to extract real examples rather than fabricate toys, so for the purposes of this PEP you can assume that they stand on their own, void of context. For example, I considered removing the `globs` example because, like you, I thought it was uglier. However, I figured that it would be wrong to omit it, since the goal of the section was to show as many candidates for the operator that we could trivially find, not "all of the ones that Brandt likes better with |". :) I'm not sure that continuing this thread of discussion is very productive. As I have mentioned before, the huge PEP is roughly: - 50% responses to arguments. - 35% examples. - 10% specification. - 5% links to prior discussions. We've addressed anything that we felt needed addressing, and they have been enough to persuade the necessary decision makers. Many of the points you make have been responded to in some way or another in this monster of a document. :)
First off, apologies for entering the discussion after it has formally ended. I'm not sure this has any relevance now that the PEP is accepted and the reference implementation merged. If not, sorry and feel free to ignore. "Leftmost Value (First-Seen) Wins" was rejected because
it is not clear that this behavior has many use-cases
This is probably based on a thorough analysis of usage patterns, but the result still surprises me. I could not find the related discussion in the mail archives. Could you elaborate, or point to more detailed information? In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset". For a container-type, I would expect this operation, informally spoken, to be applied element-wise. In other words, the resulting dict would have the union of keys, with values supplied from `other` only if keys are missing in `value`. This would allow users to provide defaults in a dictionary of settings, like this: settings |= defaults As this usage generalizes a (perceived) common idiom, I would expect people to get bitten by this. Claudio On Mon, 17 Feb 2020 at 22:05, Guido van Rossum <guido@python.org> wrote:
Now that the last bits of discussion about PEP 584 have been settled (we will *not* be calling the `copy` or `update` methods) I am accepting this PEP. Congratulations Steven and Brandt!
Thanks to everyone else who gave their opinion.
Formally, I will just send my recommendation to accept the PEP to the Steering Council -- they will then let us know whether they agree, and once that's done we can update the PEP with the "Accepted" status and land the implementation (https://github.com/python/cpython/pull/12088). (Hm, the PEP should probably also link to that PR rather than to Brandt's private branch.)
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6KT2KIOT... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset". For a container-type, I would expect this operation, informally spoken, to be applied element-wise. In other words, the resulting dict would have the union of keys, with values supplied from `other` only if keys are missing in `value`.
This would allow users to provide defaults in a dictionary of settings, like this:
settings |= defaults
As this usage generalizes a (perceived) common idiom, I would expect people to get bitten by this.
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.) The pattern I'm familiar with for applying defaults always starts with the defaults and then adds the local overrides, so the PEP 584 behavior fits perfectly. (Probably because in Python before PEP 584 the most common operation is dict.update(), which has the same semantics.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I think the "if unset" behavior is well handled by collections.ChainMap. But I do think that fact should be prominent in any documentation of the new dict Union operator. On Wed, Feb 26, 2020, 11:06 AM Guido van Rossum <guido@python.org> wrote:
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset". For a container-type, I would expect this operation, informally spoken, to be applied element-wise. In other words, the resulting dict would have the union of keys, with values supplied from `other` only if keys are missing in `value`.
This would allow users to provide defaults in a dictionary of settings, like this:
settings |= defaults
As this usage generalizes a (perceived) common idiom, I would expect people to get bitten by this.
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.)
The pattern I'm familiar with for applying defaults always starts with the defaults and then adds the local overrides, so the PEP 584 behavior fits perfectly. (Probably because in Python before PEP 584 the most common operation is dict.update(), which has the same semantics.)
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JHXN342S... Code of Conduct: http://python.org/psf/codeofconduct/
Here's our current proposal for docs. Is there anything you'd like to add? https://github.com/python/cpython/pull/18659/files On Wed, Feb 26, 2020 at 8:12 AM David Mertz <mertz@gnosis.cx> wrote:
I think the "if unset" behavior is well handled by collections.ChainMap. But I do think that fact should be prominent in any documentation of the new dict Union operator.
On Wed, Feb 26, 2020, 11:06 AM Guido van Rossum <guido@python.org> wrote:
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset". For a container-type, I would expect this operation, informally spoken, to be applied element-wise. In other words, the resulting dict would have the union of keys, with values supplied from `other` only if keys are missing in `value`.
This would allow users to provide defaults in a dictionary of settings, like this:
settings |= defaults
As this usage generalizes a (perceived) common idiom, I would expect people to get bitten by this.
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.)
The pattern I'm familiar with for applying defaults always starts with the defaults and then adds the local overrides, so the PEP 584 behavior fits perfectly. (Probably because in Python before PEP 584 the most common operation is dict.update(), which has the same semantics.)
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JHXN342S... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Thu., 27 Feb. 2020, 2:03 am Guido van Rossum, <guido@python.org> wrote:
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset".
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.)
I was thinking that bash scripting might be an example, but I double checked, and that's spelled 'VAR="${$VAR:-default value}" ' make has 'VAR ?= "default value"' C# uses "??=" for null coalescence on assignment. So I've also never come across "|=" being used for this purpose. Cheers, Nick.
Maybe it's more the `a ||= b` that acts like a `a = a or b` in Python? But then I don't think there is a confusion on Python side because languages with a || operator usually already has a simple | with a different meaning. Le jeu. 27 févr. 2020 à 00:28, Nick Coghlan <ncoghlan@gmail.com> a écrit :
On Thu., 27 Feb. 2020, 2:03 am Guido van Rossum, <guido@python.org> wrote:
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset".
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.)
I was thinking that bash scripting might be an example, but I double checked, and that's spelled 'VAR="${$VAR:-default value}" '
make has 'VAR ?= "default value"'
C# uses "??=" for null coalescence on assignment.
So I've also never come across "|=" being used for this purpose.
Cheers, Nick.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HMKYUZP5... Code of Conduct: http://python.org/psf/codeofconduct/
-- Antoine Rozo
So I've also never come across "|=" being used for this purpose.
IIRC, the JavaScript implementation of "|=" can potentially be used in the way Claudio described it, instead it's based on the truthiness of the left-hand operand rather than it being "unset". But it works in that context because "null" and "undefined" are considered falsey [1]. For example:
var value = null; var other = 2; value |= other; console.log(value); 2
So it effectively works like "value | other", but also sets "value" to "other" iff "value" is falsey. When the left-hand operand is truthy, it effectively does nothing.
var value = 3; var other = 2; value |= other; console.log(value); 3
Also worth noting, since "value |= other" translates to "value = value | other", it works as a bitwise OR operator; not as a catch-all for assigning a default value:
var value = null; var other = "test"; value |= other; console.log(value); 0
Instead, you'd have to use the standard OR operator, like this "value = value || other" (since "||=" is invalid syntax):
var value = null; var other = "test"; value = value || other; console.log(value); test
FWIW, I have very rarely seen "|=" used as an operator in JS, but I've seen "value = value || other" used a decent amount. --- [1] - https://developer.mozilla.org/en-US/docs/Glossary/Falsy On Wed, Feb 26, 2020 at 6:26 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu., 27 Feb. 2020, 2:03 am Guido van Rossum, <guido@python.org> wrote:
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset".
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.)
I was thinking that bash scripting might be an example, but I double checked, and that's spelled 'VAR="${$VAR:-default value}" '
make has 'VAR ?= "default value"'
C# uses "??=" for null coalescence on assignment.
So I've also never come across "|=" being used for this purpose.
Cheers, Nick.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HMKYUZP5... Code of Conduct: http://python.org/psf/codeofconduct/
Let me split my case into two points: 1) The intuition that the right-hand side of `a | b` is a fallback value 2) The claim that `a |= b` is a common idiom to assign defaults About 2) It appears that the idiom in 2) is less common than I assumed. My expectations were shaped by working with a C++ codebase where the `|=` operator was sometimes overloaded in such a way (since there is no `||=` operator in this language). Unfortunately, I have not found similar uses of `|=` elsewhere (e.g. in Boost). To illustrate this with a specific case nonetheless, that C++ codebase had a generic data type modelled after Haskell's Maybe. In Haskell, `Maybe a` is a type which can be empty (`Nothing`) or hold a value (`Just a`). In the C++ implementation, the operation `a |= b` would assign `just(b)` iff `a` was not empty. About 1) I still believe that there is an intuition that the right operand of `|` is a fallback value, and this intuition may lead people to wrong use of dict unions. To make this intuition more explicit: Consider other operations that behave like OR in a Boolean algebra, such as logical OR, and set union. Conceptually, these operations perform one or multiple checks on the left operand, and only consider the right operand if the check "failed". - With logical OR, short-circuit semantics are widespread (cf. C, UNIX shell, or Python itself). The right operand is only evaluated if the left operand evaluates to false in a Boolean context. - With set union, implementations typically start by copying the left operand, and add entries from the right operand if they are not already present in the copy. This is also what CPython does in `set_union` (via `set_add_entry`). As another case in point, overriding items from the right operand is already provided by `dict.update`. A `|=` operator which provides for the complementary operation of filling in missing items would have made for a more orthogonal set of operations. When formulated in terms of `|`, the two operations only differ in the order of operands.
Here's our current proposal for docs. Is there anything you'd like to add? https://github.com/python/cpython/pull/18659/files
I find this quite clear. It already points out the behaviour in question. To conclude, different semantics for dict union would have been preferable in my view, but I guess that ship has sailed. Other than changing dict union semantics I don't think there is anything actionable left here. Maybe my input still has some value in clarifying expectations some users may have for this new feature. Thank you for your openness about this late input to the discussion. On Thu, 27 Feb 2020 at 10:35, Kyle Stanley <aeros167@gmail.com> wrote:
So I've also never come across "|=" being used for this purpose.
IIRC, the JavaScript implementation of "|=" can potentially be used in the way Claudio described it, instead it's based on the truthiness of the left-hand operand rather than it being "unset". But it works in that context because "null" and "undefined" are considered falsey [1]. For example:
var value = null; var other = 2; value |= other; console.log(value); 2
So it effectively works like "value | other", but also sets "value" to "other" iff "value" is falsey. When the left-hand operand is truthy, it effectively does nothing.
var value = 3; var other = 2; value |= other; console.log(value); 3
Also worth noting, since "value |= other" translates to "value = value | other", it works as a bitwise OR operator; not as a catch-all for assigning a default value:
var value = null; var other = "test"; value |= other; console.log(value); 0
Instead, you'd have to use the standard OR operator, like this "value = value || other" (since "||=" is invalid syntax):
var value = null; var other = "test"; value = value || other; console.log(value); test
FWIW, I have very rarely seen "|=" used as an operator in JS, but I've seen "value = value || other" used a decent amount.
---
[1] - https://developer.mozilla.org/en-US/docs/Glossary/Falsy
On Wed, Feb 26, 2020 at 6:26 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu., 27 Feb. 2020, 2:03 am Guido van Rossum, <guido@python.org> wrote:
On Wed, Feb 26, 2020 at 7:43 AM Claudio Jolowicz <cjolowicz@gmail.com> wrote:
In my experience, the expression `value |= other` is a common idiom across programming languages to provide a default for `value` if it is "unset".
Interesting. Can you point to specific examples of this? In what other languages have you seen this? (Not that it would make us change PEP 584, but if this appears common we could probably warn about it prominently in docs and tutorials.)
I was thinking that bash scripting might be an example, but I double checked, and that's spelled 'VAR="${$VAR:-default value}" '
make has 'VAR ?= "default value"'
C# uses "??=" for null coalescence on assignment.
So I've also never come across "|=" being used for this purpose.
Cheers, Nick.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HMKYUZP5... Code of Conduct: http://python.org/psf/codeofconduct/
participants (10)
-
Antoine Rozo -
Brandt Bucher -
Chris Angelico -
Claudio Jolowicz -
David Mertz -
Guido van Rossum -
Kyle Stanley -
Nick Coghlan -
Serhiy Storchaka -
Steven D'Aprano