This is a great PEP. Just recently I needed this and was surprised that nothing of the sort had been implemented yet (I looked for quite some time). I have one suggestion: Wouldn't it be useful for these operators to also accept sets (functionally acting like a dict with None for all values)? This would make it very elegant to 'normalize' dicts by pruning (dict & set) or padding (set | dict) dictionaries. I would find this useful for efficient data sanitation purposes, as when processing input from web forms. Am 05.02.2020 02:28 schrieb Guido van Rossum:

Thanks Brandt (and Steven of course)! If there are no objections by next week I'll recommend this to the Steering Council for acceptance.

In the meantime, I am wondering about the reference implementation -- is it suitable to submit as a PR? Or is it a toy written in pure Python? (I found it a little tricky to follow your branch.)

On Tue, Feb 4, 2020 at 4:57 PM Brandt Bucher <brandtbucher@gmail.com> wrote:

...

Steven D'Aprano and I have pushed a third draft of PEP 584:

https://www.python.org/dev/peps/pep-0584/ [1]

The accompanying reference implementation is on GitHub:

https://github.com/brandtbucher/cpython/tree/addiction [2]

For those who have been following the discussions over the past year on python-ideas, this new draft does not contain much additional content; the most notable difference is that the choice of operator has been changed from + to |. The rest of the revisions are mostly reformatting and reorganizing the information to bring the document in line with PEP standards.

Please let us know what you think – we'd love to hear any *new* feedback that hasn't yet been addressed in the PEP or the related discussions it links to!

Thanks!

Brandt _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ [3] Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/TTIKCDIP...

...

[4] Code of Conduct: http://python.org/psf/codeofconduct/ [5]

--

--Guido van Rossum (python.org/~guido [6]) _Pronouns: he/him __(why is my pronoun here?)_ [7]

Links: ------ [1] https://www.python.org/dev/peps/pep-0584/ [2] https://github.com/brandtbucher/cpython/tree/addiction [3] https://mail.python.org/mailman3/lists/python-dev.python.org/ [4] https://mail.python.org/archives/list/python-dev@python.org/message/TTIKCDIP... [5] http://python.org/psf/codeofconduct/ [6] http://python.org/~guido [7] http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NFVVGNXB... Code of Conduct: http://python.org/psf/codeofconduct/

I have one suggestion: Wouldn't it be useful for these operators to also accept sets (functionally acting like a dict with None for all values)?

Why None? Why not 0, or False, or 42? This sort of thing belongs more in a function or method, IMHO.

Well, in their defense, None is the null object in Python, so it would be a natural choice. With that said, I am strongly against this for several reasons: - The proposal in its current form is very easy to wrap your head around: "|" takes dicts, "|=" takes anything dict.update does. - Python doesn't allow sets to be dictionaries anywhere else. It's much more natural to use a dict like a set than to use a set like a dict. Hence, this PEP! - A consistent argument that we've gotten since the very beginning is something to the effect of "I don't know what types are being used in the expression 'a | b'." While we argue that this isn't the issue in practice that people make it out to be, accepting sets here would definitely muddy the waters.

This would make it very elegant to 'normalize' dicts by pruning (dict & set) or padding (set | dict) dictionaries.

Well, if the PEP lands, the padding part is easy: padded = dict.fromkeys(expected) | given Or even better, just do the dict.fromkeys construction at module level once, and do: padded = EXPECTED | given Or, implement the behavior in a subclass if you truly need it to be a set! We're intentionally very subclass-friendly (unlike list, for example).

I agree that if we want to go down the road of creating a copy to allow for subclasses then we should define a dunder method for such a use, even if it's redundant in the face of dict.copy().

Hi Brandt, very nice PEP. I have two questions here. First:

- The proposal in its current form is very easy to wrap your head around: "|" takes dicts, "|=" takes anything dict.update does.

I see this asymmetry between the | and |= mentioned a few times in the PEP, but I don't see any rationale other than "the authors have decided". I am not saying that this is the wrong decision, but the reasoning behind this choice is not obvious, and I think it might be a good idea to include the rationale in the PEP. I'd say the asymmetry between list's `__add__` and `__iadd__` semantics is actually fairly confusing for anyone who hasn't encountered it before: >>> a = [] >>> a = a + "one"                                          --------------------------------------------------------------------------- TypeError                                 Traceback (most recent call last) <ipython-input-5-f2c4cda7ee5b> in <module> ----> 1 a = a + "one" TypeError: can only concatenate list (not "str") to list >>> a += "one" >>> a ['o', 'n', 'e'] I think most people would be surprised at the difference in semantics here and this also an example of a situation where it's not obvious that the "call `list.extend`" behavior is the right thing to do. It would be nice to see why you rejected: 1. Giving |= `.update` semantics rather than the semantics you chose for |. 2. Giving `|` the same semantics as `|=`. Second question: The specification mentions "Dict union will return a new dict containing the left operand merged with the right operand, which must be a dict (or an instance of a dict subclass)." Can you clarify if it is part of the spec that it will always return a `dict` even if one or both of the operands is a dict subclass? You mentioned in another post that this is deliberately intended to be subclass-friendly, but if it always returns `dict`, then I would expect this:     >>> class MyDict(dict):     ...    pass     ...     >>> MyDict({1: 2}) | MyDict({3: 4})     {1: 2, 3: 4} I realize that there's a lot of precedent for this with other builtin types (int subclasses reverting to int with +, etc), though generally the justifications for this are two-fold: 1. For symmetrical operations like addition it's not obvious which operand's type should prevail, particularly if the two are both different dict subclasses. I think in this case | is already asymmetrical and it would be relatively natural to think of the right thing to do as "make a copy of the LHS then update it with the RHS" (thus retaining the subtype of the LHS). 2. Subclasses may override the constructor in unpredictable ways, so we don't know how to construct arbitrary subtypes. I /think/ this objection could be satisfied by using logic equivalent to `copy.copy` for the LHS when it is a dict subclass, and then using the mapping protocol on the RHS. Unless there is compelling reason to do otherwise, I am in favor of trying to retain subclass identity after operations, but either way it would be good to be explicit about it in the specification and maybe include a bit of the rationale one way or the other. Best, Paul

05.02.20 14:27, Musbur пише:

I have one suggestion: Wouldn't it be useful for these operators to also accept sets (functionally acting like a dict with None for all values)? This would make it very elegant to 'normalize' dicts by pruning (dict & set) or padding (set | dict) dictionaries. I would find this useful for efficient data sanitation purposes, as when processing input from web forms.

d = {} d |= {(1, 2)} What is d now? {1: 2} or {(1, 2): None}

On Thu, Feb 6, 2020 at 7:42 PM Musbur <musbur@posteo.org> wrote:

Depends on what d is.

...

...

...

type({}) <class 'dict'>

So the result is {(1, 2): None}, but the ambiguity comes from the definition of {}, not from my proposal.

Actually, what Serhiy hinted at was a consequence (and, I would say, a rather weird corner case) of the current definition of |= as being "equivalent to update()". Since update() will accept a number of things, including an iterable of pairs, it's actually possible to use a set of two-element tuples as a representation of keys and values. It seems pretty unlikely that anyone would use a *set* for this (as opposed to, say, a list or generator), but it does mean that your proposal would conflict with that. Which in turn means you're actually asking for |= to special-case sets; not impossible, by any means, but it's a definite change and not simply a logical extension of current behaviour. I don't think supporting sets in |= is worth this confusion, especially since there's no easy way to override the choice of value. It'd be just as easy to do thing|=dict.fromkeys(s) and avoid the whole issue. However, IMO the &= operator would be better able to accept a set, and this is something that wouldn't conflict (and thus can be safely added later). ChrisA

06.02.20 21:38, Brandt Bucher пише:

One issue that's come up during PR review with Guido and Serhiy is, when evaluating `a | b`, whether or not the default implementation should call an overridden `copy()` method on `a` in order to create an instance of the correct subclass (rather than a plain-ol' `dict`).

It would create an exception of two rules: 1. Operators on subclasses of builtin classes do not depend on overridden methods of arguments (except the corresponding dunder method). `list.__add__` and `set.__or__` do not call copy() and extend()/update(). You should override the corresponding dunder method to change the behavior of the operator. 2. Operators do not depend on non-dunder methods. This looks to me as a direct violation of the principle "Special cases aren't special enough to break the rules."

...

I think the standard handling of subclasses in Python builtins is wrong, and I don't wish to emulate that wrong behaviour without a really good reason. Or at least a better reason than "other methods break subclassing unless explicitly overloaded, so this should do so too". Or at least not without a fight :-)

We can discuss and change the standard handling of subclasses in Python builtins. But if it be changed, it should be changed for all builtin classes, without exceptions and special cases. This could resolve a problem with rule 1. But there is still a problem with rule 2.

On 2/6/20 4:23 PM, Serhiy Storchaka wrote:

It would create an exception of two rules:

1. Operators on subclasses of builtin classes do not depend on overridden methods of arguments (except the corresponding dunder method). `list.__add__` and `set.__or__` do not call copy() and extend()/update(). You should override the corresponding dunder method to change the behavior of the operator.

2. Operators do not depend on non-dunder methods.

This looks to me as a direct violation of the principle "Special cases aren't special enough to break the rules."

I may not fully understand the implications of #1, but I would think you could implement the semantics Brandt wants using only dunder methods and copy.copy (which itself dispatches to one of a number of dunder methods - __copy__, __reduce__, __setstate__, depending on which ones are defined - we could presumably avoid the `copy` import by partially porting that logic into `__or__`): def __or__(self, other):     new_value = copy.copy(self)     for key in other.__keys__():         new_value.__setitem__(key, other.__getitem__(key))     return new_value Obviously the actual implementation would be in C and handle more edge cases and whatnot, but I think the desired semantics can all be achieved using only magic methods on the objects themselves (though we'd probably want to bypass all that stuff in favor of a "fast path" in the case of `dict` | `dict`). Best, Paul

Sorry Paul, I sent my reply too soon. I see what you're saying, and I'm pretty firmly -1 on reinventing (or importing) copy.copy. We already have an API for copying a dict (dict.copy). I still fail to see problem with using a method that doesn't start and end with underscores, other than that we "haven't done it". Brandt

...

We already have an API for copying a dict (dict.copy). I still fail to see problem with using a method that doesn't start and end with underscores, other than that we "haven't done it".

Before Python 3.0, iterators had a next() method, and that was explicitly and consciously changed to __next__(). The arguments there seem relevant here too.

Thanks for bringing this up. PEP 3114 does a great job of explaining how and why Python uses dunders for *language-level* constructs. I would probably be opposed to language features or built-in functions doing magical things with the `copy` method of dicts; however, in this case, it's being used by the dict itself! As the PEP 3114 says:

In Python, double underscores before and after a name are used to distinguish names that belong to the language itself... Not all things that are called "protocols" are made of methods with double-underscore names... even though the read method is part of the file protocol, it does not have double underscores because there is no language construct that implicitly invokes x.read().

The language itself doesn't call `copy`, but `dict` should feel free to. Especially if it makes life easier for subclasses (which PEP 584 actually encourages users to make in the "Rejected Semantics" section: https://www.python.org/dev/peps/pep-0584/#rejected-semantics)!

07.02.20 01:00, Paul G пише:

I will note that it doesn't seem to be true that operators never call standard methods. Looks like date.__add__ calls date.toordinal and date.fromordinal (at least in the pure Python implementation), and datetime calls those plus tzinfo.utcoffset. Not sure if the rule Serhiy is citing is only intended to apply to builtins, though.

It is an implementation detail. The C implementation does not call date.toordinal and date.fromordinal.

After a few days of thinking and experimenting, I’ve been convinced that `copy` (and also `__copy__`) is not the right protocol for what we want to do here. I believe that 584 can likely continue without subclass-preserving behavior, but that better behavior could perhaps could be added to *all* built-in types later, since it’s outside the scope of this PEP.

My opinion is that Python built-in types make subclassing unnecessarily(?) awkward to use, and I would like to see that change.

Yes! But, on further reflection, I don’t think this is the correct way of approaching it.

For example, consider subclassing float. If you want your subclass to be actually usable, you have to write a whole bunch of boilerplate, otherwise the first time you perform arithmetic on your subclass, it will be converted to a regular old float… This is painful and adds a great amount of friction to subclassing.

`float` is a *perfect* example of the problems with the way things are currently, so let’s focus on this. Currently, subclassing `float` requires ~30 overridden methods of repetitive (but non-trivial) boilerplate to get everything working right. However, calling the `float` equivalent of `dict.copy()` on the LHS before proceeding with the default implementation wouldn’t help us, because floats (like many built-in types) are immutable. So a new, plain, built-in `float` would still be returned by the default machinery. It doesn’t know how to construct a new, different instance of our subclass, and it can’t change one it’s already built. This leads me to believe that we’re approaching the problem wrong. Rather than making a copy and working on it, I think the problem would be better served by a protocol that runs the default implementation, *then* calls some under hook on the subclass to build a new instance. Let’s call this method `__build__`. I’m not sure what its arguments would look like, but it would probably need at least `self`, and an instance of the built-in base class (in this case a `float`), and return a new instance of the subclass based on the two. It would likely also need to work with `cls` instead of `self` for `classmethod` constructors like `dict.fromkeys`, or have a second hook for that case. By subclassing `float` and defining `__build__` to something like this: ``` class MyFloat(float): … def __build__(self, result): Return MyFloat(result, some_state=self.some_state) … ``` I could now trust the built-in `float` machinery to try calling `lhs.__build__(result)` on the result that *would* have been returned *before* returning it. This is a simple example, but a protocol like this would work for mutables as well.

A more pertinent example, from dict itself:

If `dict` *were* to grow more operators, they would likely be `^`, `&`, and `-`. You can consider the case of subclassing `set` or `frozenset`, since they currently has those. Calling `lhs.copy()` first before updating is fine for additive operations like `|`, but for subtractive operations like the others, this can be very bad for performance, especially if we’re now *required* to call them. Again, doing things the default way, and *then* constructing the appropriate subclass in an agreed-upon way seems like the path to take here.

Changing all builtins is a big, backwards-incompatible change.

If implemented right, a system like the one described above (`__build__`) wouldn’t be backward-incompatible, as long as nobody was already using the name. Just food for thought. I think this is a much bigger issue than PEP 584, but I'm convinced that the consistent status quo should prevail until a suitable solution for all types can be worked out (if ever).

So can we just finish PEP 584 without the .copy() call? Surely the more general solution will not be ready for Python 3.9, while PEP 584 is nearly done. (I'm also skeptical about a general solution, but I'd rather stay out of that discussion for a while longer, and maybe you all come up with something good.) On Sun, Feb 16, 2020 at 8:46 AM Brandt Bucher <brandtbucher@gmail.com> wrote:

After a few days of thinking and experimenting, I’ve been convinced that `copy` (and also `__copy__`) is not the right protocol for what we want to do here. I believe that 584 can likely continue without subclass-preserving behavior, but that better behavior could perhaps could be added to *all* built-in types later, since it’s outside the scope of this PEP.

...

My opinion is that Python built-in types make subclassing unnecessarily(?) awkward to use, and I would like to see that change.

Yes! But, on further reflection, I don’t think this is the correct way of approaching it.

...

For example, consider subclassing float. If you want your subclass to be actually usable, you have to write a whole bunch of boilerplate, otherwise the first time you perform arithmetic on your subclass, it will be converted to a regular old float… This is painful and adds a great amount of friction to subclassing.

`float` is a *perfect* example of the problems with the way things are currently, so let’s focus on this.

Currently, subclassing `float` requires ~30 overridden methods of repetitive (but non-trivial) boilerplate to get everything working right. However, calling the `float` equivalent of `dict.copy()` on the LHS before proceeding with the default implementation wouldn’t help us, because floats (like many built-in types) are immutable. So a new, plain, built-in `float` would still be returned by the default machinery. It doesn’t know how to construct a new, different instance of our subclass, and it can’t change one it’s already built.

This leads me to believe that we’re approaching the problem wrong. Rather than making a copy and working on it, I think the problem would be better served by a protocol that runs the default implementation, *then* calls some under hook on the subclass to build a new instance.

Let’s call this method `__build__`. I’m not sure what its arguments would look like, but it would probably need at least `self`, and an instance of the built-in base class (in this case a `float`), and return a new instance of the subclass based on the two. It would likely also need to work with `cls` instead of `self` for `classmethod` constructors like `dict.fromkeys`, or have a second hook for that case.

By subclassing `float` and defining `__build__` to something like this:

``` class MyFloat(float): … def __build__(self, result): Return MyFloat(result, some_state=self.some_state) … ```

I could now trust the built-in `float` machinery to try calling `lhs.__build__(result)` on the result that *would* have been returned *before* returning it. This is a simple example, but a protocol like this would work for mutables as well.

...

A more pertinent example, from dict itself:

If `dict` *were* to grow more operators, they would likely be `^`, `&`, and `-`. You can consider the case of subclassing `set` or `frozenset`, since they currently has those. Calling `lhs.copy()` first before updating is fine for additive operations like `|`, but for subtractive operations like the others, this can be very bad for performance, especially if we’re now *required* to call them. Again, doing things the default way, and *then* constructing the appropriate subclass in an agreed-upon way seems like the path to take here.

...

Changing all builtins is a big, backwards-incompatible change.

If implemented right, a system like the one described above (`__build__`) wouldn’t be backward-incompatible, as long as nobody was already using the name.

Just food for thought. I think this is a much bigger issue than PEP 584, but I'm convinced that the consistent status quo should prevail until a suitable solution for all types can be worked out (if ever). _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HNJ6RVV... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

So can we just finish PEP 584 without the .copy() call?

Sounds good. I’ll update the implementation and open a PR to the PEPs repo later tonight. Thanks everybody!