[Numpy-discussion] subclassing ndarray in python3

[Pauli Virtanen]

Hi Darren,

to, 2010-03-11 kello 11:11 -0500, Darren Dale kirjoitti:
> Now that the trunk has some support for python3, I am working on
> making Quantities work with python3 as well. I'm running into some
> problems related to subclassing ndarray that can be illustrated with a
> simple script, reproduced below. It looks like there is a problem with
> the reflected operations, I see problems with __rmul__ and __radd__,
> but not with __mul__ and __add__:

Thanks for testing. I wish the test suite was more complete (hint!
hint! :)

Yes, Python 3 introduced some semantic changes in how subclasses of
builtin classes (= written in C) inherit the __r*__ operations.

Below I'll try to explain what is going on. We probably need to change
some things to make things work better on Py3, within the bounds we are
able to.

Suggestions are welcome. The most obvious one could be to explicitly
implement __rmul__ etc. on Python 3.

[clip]
> class A(np.ndarray):
>     def __new__(cls, *args, **kwargs):
>         return np.ndarray.__new__(cls, *args, **kwargs)
> 
> class B(A):
>     def __mul__(self, other):
>         return self.view(A).__mul__(other)
>     def __rmul__(self, other):
>         return self.view(A).__rmul__(other)
>     def __add__(self, other):
>         return self.view(A).__add__(other)
>     def __radd__(self, other):
>         return self.view(A).__radd__(other)
[clip]
> print('A __rmul__:')
> print(a.__rmul__(2))
> # yields NotImplemented
> print(a.view(np.ndarray).__rmul__(2))
> # yields NotImplemented

Correct. ndarray does not implement __rmul__, but relies on an automatic
wrapper generated by Python.

The automatic wrapper (wrap_binaryfunc_r) does the following:

1. Is `type(other)` a subclass of `type(self)`?
   If yes, call __mul__ with swapped arguments.
2. If not, bail out with NotImplemented.

So it bails out.

Previously, the ndarray type had a flag that made Python to skip the
subclass check. That does not exist any more on Python 3, and is the
root of this issue.

> print(2*a)
> # ok !!??

Here, Python checks

1. Does nb_multiply from the left op succeed? Nope, since floats don't
   know how to multiply ndarrays.

2. Does nb_multiply from the right op succeed? Here the execution
   passes *directly* to array_multiply, completely skipping the __rmul__
   wrapper.

   Note also that in the C-level number protocol there is only a single
   multiplication function for both left and right multiplication.

[clip]
> print('B __rmul__:')
> print(b.__rmul__(2))
> # yields NotImplemented
> print(b.view(A).__rmul__(2))
> # yields NotImplemented
> print(b.view(np.ndarray).__rmul__(2))
> # yields NotImplemented
> print(2*b)
> # yields: TypeError: unsupported operand type(s) for *: 'int' and 'B'

But here, the subclass calls the wrapper ndarray.__rmul__, which wants
to be careful with types, and hence fails.

Yes, probably explicitly defining __rmul__ for ndarray could be the
right solution. Please file a bug report on this.

Cheers,
Pauli