[New-bugs-announce] [issue30140] Binary arithmetic does not always call subclasses first

[Stephan Hoyer]

New submission from Stephan Hoyer:

We are writing a system for overloading NumPy operations (see PR [1] and design doc [2]) that is designed to copy and extend Python's system for overloading binary operators.

The reference documentation on binary arithmetic [3] states:

> Note: If the right operand's type is a subclass of the left operand’s type and that subclass provides the reflected method for the operation, this method will be called before the left operand’s non-reflected method. This behavior allows subclasses to override their ancestors’ operations.

However, this isn't actually done if the right operand merely inherits from the left operand's type. In practice, CPython requires that the right operand defines a different method before it defers to it. Note that the behavior is different for comparisons, which defer to subclasses regardless of whether they implement a new method [4].

I think this behavior is a mistake and should be corrected. It is just as useful to write generic binary arithmetic methods that are well defined on subclasses as generic comparison operations. In fact, this is exactly the design pattern we propose for objects implementing special operators like NumPy arrays (see NDArrayOperatorsMixin in [1] and [2]).

Here is a simple example, of a well-behaved that implements addition by wrapping its value and returns NotImplemented when the other operand has the wrong type:

class A:
   def __init__(self, value):
       self.value = value
   def __add__(self, other):
       if not isinstance(other, A):
           return NotImplemented
       return type(self)(self.value + other.value)
   __radd__ = __add__
   def __repr__(self):
       return f'{type(self).__name__}({self.value!r})'

class B(A):
    pass

class C(A):
   def __add__(self, other):
       if not isinstance(other, A):
           return NotImplemented
       return type(self)(self.value + other.value)
   __radd__ = __add__

A does not defer to B:

>>> A(1) + B(1)
A(2)

But it does defer to C, which defines new methods (literally copied/pasted) for __add__/__radd__:

>>> A(1) + C(1)
C(2)

With the current behavior, special operator implementations need to explicitly account for the possibility that they are being called from a subclass by returning NotImplemented. My guess is that this is rarely done, which means that most of these methods are broken when used with subclasses, or subclasses needlessly reimplement these methods.

Can we fix this logic for Python 3.7?

[1] https://github.com/numpy/numpy/pull/8247
[2] https://github.com/charris/numpy/blob/406bbc652424fff332f49b0d2f2e5aedd8191d33/doc/neps/ufunc-overrides.rst
[3] https://docs.python.org/3/reference/datamodel.html#object.__ror__
[4] http://bugs.python.org/issue22052

----------
messages: 292149
nosy: Stephan Hoyer
priority: normal
severity: normal
status: open
title: Binary arithmetic does not always call subclasses first
type: behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30140>
_______________________________________