subclassing ndarray in python3
Now that the trunk has some support for python3, I am working on making Quantities work with python3 as well. I'm running into some problems related to subclassing ndarray that can be illustrated with a simple script, reproduced below. It looks like there is a problem with the reflected operations, I see problems with __rmul__ and __radd__, but not with __mul__ and __add__: import numpy as np class A(np.ndarray): def __new__(cls, *args, **kwargs): return np.ndarray.__new__(cls, *args, **kwargs) class B(A): def __mul__(self, other): return self.view(A).__mul__(other) def __rmul__(self, other): return self.view(A).__rmul__(other) def __add__(self, other): return self.view(A).__add__(other) def __radd__(self, other): return self.view(A).__radd__(other) a = A((10,)) b = B((10,)) print('A __mul__:') print(a.__mul__(2)) # ok print(a.view(np.ndarray).__mul__(2)) # ok print(a*2) # ok print('A __rmul__:') print(a.__rmul__(2)) # yields NotImplemented print(a.view(np.ndarray).__rmul__(2)) # yields NotImplemented print(2*a) # ok !!?? print('B __mul__:') print(b.__mul__(2)) # ok print(b.view(A).__mul__(2)) # ok print(b.view(np.ndarray).__mul__(2)) # ok print(b*2) # ok print('B __add__:') print(b.__add__(2)) # ok print(b.view(A).__add__(2)) # ok print(b.view(np.ndarray).__add__(2)) # ok print(b+2) # ok print('B __rmul__:') print(b.__rmul__(2)) # yields NotImplemented print(b.view(A).__rmul__(2)) # yields NotImplemented print(b.view(np.ndarray).__rmul__(2)) # yields NotImplemented print(2*b) # yields: TypeError: unsupported operand type(s) for *: 'int' and 'B' print('B __radd__:') print(b.__radd__(2)) # yields NotImplemented print(b.view(A).__radd__(2)) # yields NotImplemented print(b.view(np.ndarray).__radd__(2)) # yields NotImplemented print(2+b) # yields: TypeError: unsupported operand type(s) for +: 'int' and 'B'
Hi Darren, to, 2010-03-11 kello 11:11 -0500, Darren Dale kirjoitti:
Now that the trunk has some support for python3, I am working on making Quantities work with python3 as well. I'm running into some problems related to subclassing ndarray that can be illustrated with a simple script, reproduced below. It looks like there is a problem with the reflected operations, I see problems with __rmul__ and __radd__, but not with __mul__ and __add__:
Thanks for testing. I wish the test suite was more complete (hint! hint! :) Yes, Python 3 introduced some semantic changes in how subclasses of builtin classes (= written in C) inherit the __r*__ operations. Below I'll try to explain what is going on. We probably need to change some things to make things work better on Py3, within the bounds we are able to. Suggestions are welcome. The most obvious one could be to explicitly implement __rmul__ etc. on Python 3. [clip]
class A(np.ndarray): def __new__(cls, *args, **kwargs): return np.ndarray.__new__(cls, *args, **kwargs)
class B(A): def __mul__(self, other): return self.view(A).__mul__(other) def __rmul__(self, other): return self.view(A).__rmul__(other) def __add__(self, other): return self.view(A).__add__(other) def __radd__(self, other): return self.view(A).__radd__(other) [clip] print('A __rmul__:') print(a.__rmul__(2)) # yields NotImplemented print(a.view(np.ndarray).__rmul__(2)) # yields NotImplemented
Correct. ndarray does not implement __rmul__, but relies on an automatic wrapper generated by Python. The automatic wrapper (wrap_binaryfunc_r) does the following: 1. Is `type(other)` a subclass of `type(self)`? If yes, call __mul__ with swapped arguments. 2. If not, bail out with NotImplemented. So it bails out. Previously, the ndarray type had a flag that made Python to skip the subclass check. That does not exist any more on Python 3, and is the root of this issue.
print(2*a) # ok !!??
Here, Python checks 1. Does nb_multiply from the left op succeed? Nope, since floats don't know how to multiply ndarrays. 2. Does nb_multiply from the right op succeed? Here the execution passes *directly* to array_multiply, completely skipping the __rmul__ wrapper. Note also that in the C-level number protocol there is only a single multiplication function for both left and right multiplication. [clip]
print('B __rmul__:') print(b.__rmul__(2)) # yields NotImplemented print(b.view(A).__rmul__(2)) # yields NotImplemented print(b.view(np.ndarray).__rmul__(2)) # yields NotImplemented print(2*b) # yields: TypeError: unsupported operand type(s) for *: 'int' and 'B'
But here, the subclass calls the wrapper ndarray.__rmul__, which wants to be careful with types, and hence fails. Yes, probably explicitly defining __rmul__ for ndarray could be the right solution. Please file a bug report on this. Cheers, Pauli
Hi Pauli, On Thu, Mar 11, 2010 at 3:38 PM, Pauli Virtanen <pav@iki.fi> wrote:
Thanks for testing. I wish the test suite was more complete (hint! hint! :)
I'll be happy to contribute, but lately I get a few 15-30 minute blocks a week for this kind of work (hence the short attempt to work on Quantities this morning), and its not likely to let up for about 3 weeks.
Yes, probably explicitly defining __rmul__ for ndarray could be the right solution. Please file a bug report on this.
Done: http://projects.scipy.org/numpy/ticket/1426 Cheers, and *thank you* for all you have already done to support python-3, Darren
participants (2)
-
Darren Dale -
Pauli Virtanen