[Python-ideas] Binary arithmetic does not always call subclasses first

[Steven D'Aprano]

On Sun, Apr 23, 2017 at 06:23:12PM -0700, Stephan Hoyer wrote:
> I recently filed this as a bug, and was asked to repost to python-dev or
> python-ideas for greater visibility:
> http://bugs.python.org/issue30140
> 
> Without further ado, here is my original report:
[...]

> The reference documentation on binary arithmetic [3] states:
> 
> > Note: If the right operand's type is a subclass of the left operand’s
> type and that subclass provides the reflected method for the operation,
> this method will be called before the left operand’s non-reflected method.
> This behavior allows subclasses to override their ancestors’ operations.
> 
> However, this isn't actually done if the right operand merely inherits from
> the left operand's type. In practice, CPython requires that the right
> operand defines a different method before it defers to it. Note that the
> behavior is different for comparisons, which defer to subclasses regardless
> of whether they implement a new method [4].

I think that's a bug. At the very least, comparisons and operators 
should behave the same.

I think that the comparison behaviour is correct: the subclass 
reflected method should be called, even if the method isn't explicitly 
over-ridden.


> I think this behavior is a mistake and should be corrected. It is just as
> useful to write generic binary arithmetic methods that are well defined on
> subclasses as generic comparison operations.

Agreed.


[...]
> Here is a simple example, of a well-behaved type that implements addition
> by wrapping its value and that returns NotImplemented when the other
> operand has the wrong type:

I often write classes like your example, and it is humbling to realise 
that they are buggy! I must admit I misread the docs and didn't notice 
the "and overrides" language, and clearly I didn't test my classes 
sufficiently or else I would have discovered this myself :-(


> A does not defer to subclass B:
> 
> >>> A(1) + B(1)
> A(2)
> 
> But it does defer to subclass C, which defines new methods (literally
> copied/pasted) for __add__/__radd__:
> 
> >>> A(1) + C(1)
> C(2)

To me, that's the deciding point. We have two ways to go, and one of 
them encourages the copy-and-paste anti-pattern, the other doesn't.

I think that's a Bad Thing and we should treat the comparison behaviour 
as correct, and the other operators are buggy. An explicit over-ride 
shouldn't be necessary.


> Mark Dickinson's response:
>
> The "Coercion rules" section[1] of the Python 2.7 docs is a bit more
> explicit about the intent:
> 
> """
> Exception to the previous item: if the left operand is an instance of a
> built-in type or a new-style class, and the right operand is an instance of
> a proper subclass of that type or class and overrides the base’s __rop__()
> method, the right operand’s __rop__() method is tried before the left
> operand’s __op__() method.
> """
> 
> so the check for an override was clearly intentional, rather than an
> implementation convenience or accident. (It's also clearly intentional in
> the source and comments.)

The next paragraph tells us:

    This is done so that a subclass can completely override binary 
    operators. Otherwise, the left operand’s __op__() method would 
    always accept the right operand: when an instance of a given class 
    is expected, an instance of a subclass of that class is always 
    acceptable.


but as far as I can tell, the comparison behaviour equally accomplishes 
that, without an explicit over-ride.


> The 3.x docs don't have the "and overrides"
> language; I haven't figured out why and when that language changed.
> 
> [1]
> https://docs.python.org/release/2.7.6/reference/datamodel.html#coercion-rules