[Numpy-discussion] Implementations of ndarray.__array_function__ (and ndarray.__array_ufunc__)

[Marten van Kerkwijk]

On Sun, Nov 4, 2018 at 8:57 PM Stephan Hoyer <shoyer at gmail.com> wrote:

> On Sun, Nov 4, 2018 at 8:45 AM Marten van Kerkwijk <
> m.h.vankerkwijk at gmail.com> wrote:
>
>> Does the above make sense? I realize that the same would be true for
>> `__array_ufunc__`, though there the situation is slightly trickier since it
>> is not as easy to bypass any further override checks. Nevertheless, it does
>> seem like it would be correct to do the same there. (And if we agree this
>> is the case, I'd quite happily implement it -- with the merger of
>> multiarray and umath it has become much easier to do.)
>>
>
> Marten actually implemented a draft version of this already in
> https://github.com/numpy/numpy/pull/12328 :). I found reading over the PR
> helpful for understand this proposal.
>
> I guess the practical import of this change is that it makes it (much?)
> easier to write __array_function__ for ndarray subclasses: if there's a
> function where NumPy's default function works fine, you don't need to
> bother with returning anything other than NotImplemented from
> __array_function__. It's sort of like NotImplementedButCoercible, but only
> for ndarray subclasses.
>

Yes, return NotImplemented if there is another array, or, even simpler,
just call super.

Note that it is not quite like `NotImplementedButCoercible`, since no
actual coercion to ndarray would necessarily be needed - with adherence to
the Liskov substitution principle, the subclass might stay intact (if only
partially initialized).


> One minor downside is that this might make it harder to eventually
> deprecate and/or contemplate removing checks for 'mean' methods in
> functions like np.mean(), because __array_function__ implementers might
> still be relying on this.
>

I think this is somewhat minor indeed, since we can (and should) insist
that subclasses here properly behave as subclasses, so if an
ndarray-specific implementation breaks a subclass, that might well indicate
that the subclass is not quite good enough (and we can now point out there
is a way to override the function). It might also indicate that the code
itself could be better - that would be a win.

But so far, I think this makes sense.
>
> The PR includes additional changes to np.core.overrides, but I'm not sure
> if those are actually required here (or rather only possible due to this
> change). I guess they are needed if you want to be able to count on
> ndarray.__array_function__ being called after subclass __array_function__
> methods.
>

It is mostly a transfer of functionality from `get_override_types_and_args`
to the place where the implementation is decided upon. Perhaps more logical
even if we do not pursue this.

>
> I'm not sure I like this part: it means that ndarray.__array_function__
> actually gets called when other arguments implement __array_function__. For
> interactions with objects that aren't ndarray subclasses this is entirely
> pointless and would unnecessarily slow things down, since
> ndarray._array_function__ will always return NotImplemented.
>

Agreed here. I did in fact think about it, but wasn't sure (and didn't have
time to think how to check) that the gain in time for cases where an
ndarray comes before the relevant array mimic (and there thus a needless
call to ndarray.__array_function__ can be prevented) was worth it compared
to the cost of attempting to do the removal for cases where the array mimic
came first or where there was no regular ndarray in the first place. But I
think this is an implementation detail; for now, let me add a note to the
PR about it.

All the best,

Marten
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20181105/88ccd32c/attachment.html>