Re: [Python-Dev] Python-versus-CPython question for __mul__ dispatch

I expect you can make something that behaves like list by defining __mul__ and __rmul__ and returning NotImplemented. On Thursday, May 14, 2015, Stefan Richthofer wrote:

...

...

Should I be worried?

You mean should *I* be worried ;)

Stuff like this is highly relevant for JyNI, so thanks very much for clarifying this subtle behavior. It went onto my todo-list right now to ensure that JyNI will emulate this behavior as soon as I am done with gc-support. (Hopefully it will be feasible, but I can only tell in half a year or so since there are currently other priorities.) Still, this "essay" potentially will save me a lot of time.

So, everybody please feel encouraged to post things like this as they come up. Maybe there could be kind of a pitfalls-page somewhere in the docs collecting these things.

Best

Stefan

...

Gesendet: Freitag, 15. Mai 2015 um 02:45 Uhr Von: "Nathaniel Smith" javascript:;> An: "Python Dev" javascript:;> Betreff: [Python-Dev] Python-versus-CPython question for __mul__ dispatch

Hi all,

While attempting to clean up some of the more squamous aspects of numpy's operator dispatch code [1][2], I've encountered a situation where the semantics we want and are using are possible according to CPython-the-interpreter, but AFAICT ought not to be possible according to Python-the-language, i.e., it's not clear to me whether it's possible even in principle to implement an object that works the way numpy.ndarray does in any other interpreter. Which makes me a bit nervous, so I wanted to check if there was any ruling on this.

Specifically, the quirk we are relying on is this: in CPython, if you do

[1, 2] * my_object

then my_object's __rmul__ gets called *before* list.__mul__, *regardless* of the inheritance relationship between list and type(my_object). This occurs as a side-effect of the weirdness involved in having both tp_as_number->nb_multiply and tp_as_sequence->sq_repeat in the C API -- when evaluating "a * b", CPython tries a's nb_multiply, then b's nb_multiply, then a's sq_repeat, then b's sq_repeat. Since list has an sq_repeat but not an nb_multiply, this means that my_object's nb_multiply gets called before any list method.

Here's an example demonstrating how weird this is. list.__mul__ wants an integer, and by "integer" it means "any object with an __index__ method". So here's a class that list is happy to be multiplied by -- according to the ordinary rules for operator dispatch, in the example below Indexable.__mul__ and __rmul__ shouldn't even get a look-in:

In [3]: class Indexable(object): ...: def __index__(self): ...: return 2 ...:

In [4]: [1, 2] * Indexable() Out[4]: [1, 2, 1, 2]

But, if I add an __rmul__ method, then this actually wins:

In [6]: class IndexableWithMul(object): ...: def __index__(self): ...: return 2 ...: def __mul__(self, other): ...: return "indexable forward mul" ...: def __rmul__(self, other): ...: return "indexable reverse mul"

In [7]: [1, 2] * IndexableWithMul() Out[7]: 'indexable reverse mul'

In [8]: IndexableWithMul() * [1, 2] Out[8]: 'indexable forward mul'

NumPy arrays, of course, correctly define both __index__ method (which raises an array on general arrays but coerces to int for arrays that contain exactly 1 integer), and also defines an nb_multiply slot which accepts lists and performs elementwise multiplication:

In [9]: [1, 2] * np.array(2) Out[9]: array([2, 4])

And that's all great! Just what we want. But the only reason this is possible, AFAICT, is that CPython 'list' is a weird type with undocumented behaviour that you can't actually define using pure Python code.

Should I be worried?

-n

[1] https://github.com/numpy/numpy/pull/5864 [2] https://github.com/numpy/numpy/issues/5844

-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org javascript:; https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.d...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org javascript:; https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (on iPad)