[Numpy-discussion] __skip_array_function__ discussion summary

[Sebastian Berg]

Hi all,

This is an attempt from me to wrap up the discussion a bit so that
others can chime in if they want to.

NumPy 1.17 will ship with `__array_function__` a way for array like
projects (dask, cupy) to override almost all numpy functions [0]. This
addition is uncontroversial.
NumPy 1.17 will _not_ ship with the `__skip_array_funciton__` following
a longer dicussion. For those interested, I tried to give an very short
overview over the topic below.


The discussion here is around the addition of `__skip_array_function__`
which would allow code to use:

np.ones_like.__skip_array_function__(*args)

to reuse the current implementation in numpy (i.e. directly call the
current code). This can simplify things drastically for some array-
likes, since they do not have to provide an alternative implementation.
However, PR-13585 [1] sparked a more detailed discussion, since it was
going to add the use of `__skip_array_function__` internally in numpy
[2].

The issue is exposure of implementation details. If we do not use it
internally, a user may implement their own `np.empty_like` and rely on
`np.ones_like` to use `np.empty_like` [3] internally. Thus,
`np.ones_like(my_array_like)` can work without `my_array_like` having
any special code for `np.ones_like`.

The PR exposes the issue that if `np.ones_like` is changed to call
`np.empty_like.__skip_array_function__` internally, this will break the
users `my_array_like` (it will not call their own `np.empty_like`
implementation.

We could expect users to fix up such breaking changes, but it exposes
how fragile the interaction of user types using
`__skip_array_function__` and changes in the specific implementation
used by numpy can be in some cases.

The second option would be to make sure we use
`__skip_array_function__` internally, so that users cannot expect
`np.ones_like` to work because they made `np.empty_like` work in the
above example (does not increase the "API surface" of NumPy).

Plus it increases the issue that the numpy code itself is less readable
if we use `__skip_array_function__` internally in many/all places.

Those two options further have very different goals in mind for the
final usage of the protocol. So that right now the solution is to step
back, not include the addition and rather gain experience with the
NumPy 1.17 release that includes `__array_function__` but not
`__skip_array_function`.


I hope this may help those interested who did not follow the full
discussion, can't say I feel I am very good at summarizing. For details
I encourage you to have a look at the PR discussion and the recent
mails to the list.

Best,

Sebastian


[0] http://www.numpy.org/neps/nep-0018-array-function-protocol.html#implementations-in-terms-of-a-limited-core-api
[1] https://github.com/numpy/numpy/pull/13585
[2] Mostly for slight optimization.
[3] It also uses `np.copyto` which can be overridden as well.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190523/2d348e30/attachment-0001.sig>