While attempting to clean up some of the more squamous aspects of numpy's operator dispatch code , I've encountered a situation where the semantics we want and are using are possible according to CPython-the-interpreter, but AFAICT ought not to be possible according to Python-the-language, i.e., it's not clear to me whether it's possible even in principle to implement an object that works the way numpy.ndarray does in any other interpreter. Which makes me a bit nervous, so I wanted to check if there was any ruling on this.
Specifically, the quirk we are relying on is this: in CPython, if you do
[1, 2] * my_object
then my_object's __rmul__ gets called *before* list.__mul__, *regardless* of the inheritance relationship between list and type(my_object). This occurs as a side-effect of the weirdness involved in having both tp_as_number->nb_multiply and tp_as_sequence->sq_repeat in the C API -- when evaluating "a * b", CPython tries a's nb_multiply, then b's nb_multiply, then a's sq_repeat, then b's sq_repeat. Since list has an sq_repeat but not an nb_multiply, this means that my_object's nb_multiply gets called before any list method.
Here's an example demonstrating how weird this is. list.__mul__ wants an integer, and by "integer" it means "any object with an __index__ method". So here's a class that list is happy to be multiplied by -- according to the ordinary rules for operator dispatch, in the example below Indexable.__mul__ and __rmul__ shouldn't even get a look-in:
In : class Indexable(object): ...: def __index__(self): ...: return 2 ...:
In : [1, 2] * Indexable() Out: [1, 2, 1, 2]
But, if I add an __rmul__ method, then this actually wins:
In : class IndexableWithMul(object): ...: def __index__(self): ...: return 2 ...: def __mul__(self, other): ...: return "indexable forward mul" ...: def __rmul__(self, other): ...: return "indexable reverse mul"
In : [1, 2] * IndexableWithMul() Out: 'indexable reverse mul'
In : IndexableWithMul() * [1, 2] Out: 'indexable forward mul'
NumPy arrays, of course, correctly define both __index__ method (which raises an array on general arrays but coerces to int for arrays that contain exactly 1 integer), and also defines an nb_multiply slot which accepts lists and performs elementwise multiplication:
In : [1, 2] * np.array(2) Out: array([2, 4])
And that's all great! Just what we want. But the only reason this is possible, AFAICT, is that CPython 'list' is a weird type with undocumented behaviour that you can't actually define using pure Python code.
Should I be worried?