[Python-ideas] Binary arithmetic does not always call subclasses first

[Steven D'Aprano]

On Mon, Apr 24, 2017 at 05:57:17PM +1200, Greg Ewing wrote:
> Stephan Hoyer wrote:
> >In practice, CPython requires that the 
> >right operand defines a different method before it defers to it.
> 
> I'm not sure exactly what the rationale for this behaviour is,
> but it's probably something along the lines that the left
> method should already know how to deal with that combination
> of types, and right methods are only supposed to be called
> as a fallback if the left method can't handle the operands,
> so calling it in that situation would be wrong.

I've never seen that rationale before, and I don't think I would agree 
with it. And it goes against the rationale in the docs:

    [...] the right operand’s __rop__() method is tried before the 
    left operand’s __op__() method.

    This is done so that a subclass can completely override binary 
    operators. Otherwise, the left operand’s __op__() method would 
    always accept the right operand: when an instance of a given 
    class is expected, an instance of a subclass of that class is 
    always acceptable.


I think your rationale goes against the intention as documented. There's 
no expectation that __rop__ methods are only to be called when the 
__op__ method can't handle the operands. 

In general, which operand "wins" should depend on the classes, not on 
whether they happen to be on the left or right of the operator. (Except 
in the case where we cannot decide between the operands, in which case 
we break ties by preferring the __op__.) The reason is that subclasses 
are usually intended to be more specialised than their parent, and so 
they ought to be given priority in mixed operations.

Given classes X, Y(X), with instances x and y, we should expect that the 
more specialised class (namely Y) gets called first whether we write:

    x ⊕ y

or 

    y ⊕ x

for any operator ⊕.

As documented in the 3 docs, we get that for free: the interpreter 
correctly calls the __op__ or __rop__ method, as needed, and the class 
author doesn't have to think about it. That's how it's documented, but 
not how it's implemented.

The alternative is that every class has to include boilerplate testing 
for subclasses, as you say:


> Following that logic, the wrapper's __add__ method in your
> example needs to allow for the subclassing case, e.g.
> 
>    def __add__(self, other):
>        t1 = type(self)
>        t2 = type(other)
>        t = t2 if issubclass(t2, t1) else t1
>        return t(self.value + other.value)

but that's bad. That makes each and every class (that might ever be 
subclassed) responsible for checking for subclasses, instead of putting 
the check in one place (whichever part of the interpreter handles 
calling __op__/__rop__ methods).

Remember that a specialised subclass might not overload the __op__ and 
__rop__ methods themselves. It might overload a data attribute, or 
another method that __op__ / __rop__ call.

class A:
    def __add__(self, other):
        self.log()
        ...
    __radd__ = __add__

class B(A):
    def log(self):
        ...

A() + B()

As the more specialised instance (a subclass of A), the right hand 
operand should get the priority.


> >the behavior is different for comparisons, which defer to 
> >subclasses regardless of whether they implement a new method
> 
> Comparisons are a bit different, because they don't have
> separate left and right methods, although it's hard to see
> exactly how that affects the logic.

It doesn't affect the logic, and comparisons implement exactly the 
documented (in 3) behaviour. The only difference is that the 
reversed methods aren't spelled __rop__:

    __eq__ and __ne__ are their own reflection;
    __lt__ and __gt__
    __le__ and __ge__


For example:

py> class A(object):
...     def __lt__(self, other):
...         print("lt", self)
...         return True
...     def __gt__(self, other):
...         print("gt", self)
...         return False
...
py> class B(A):
...     pass
...
py> A() < B()
gt <__main__.B object at 0xb7a9e8ec>
False


The more specialised class (B) has its method called, even though it 
isn't over-ridden.



-- 
Steve