[Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer shoyer at gmail.com
Wed Aug 29 17:07:58 EDT 2018

On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk <
m.h.vankerkwijk at gmail.com> wrote:

> Absolutely fine to have to deal with future chances - my main point is
> that by accepting the NEP, I think numpy is committing to provide some way
> to override whatever functions __array_function__ is introduced for, i.e.,
> we cannot reasonably go back to not providing any way to override such a
> function (but by marking the feature experimental, we *can* change the way
> the override of any given function is implemented).

I'm not entirely sure that we're ready to make this commitment yet, but I
think this ties into our decision about what sort of user opt-ins to

I'd like to structure this decision in terms of the types of breaking
changes we contemplate possibly making in this future with regards to this

1. Abandoning this approach of allowing overrides for NumPy's high level
API entirely, in favor of requiring the use of a dedicated namespace like
numpy.api. This might arise if we decide that the overhead of checking for
__array_function__ attributes is too much, or if it's simply too confusing
to allow NumPy functions to be overloaded in arbitrary ways.
2. Changing the general __array_function__ protocol in a breaking way,
e.g., to eliminate the "types" argument.
3. Replacing __array_function__ with another override mechanism, either (a)
in general (e.g., __array_function_v2__ without the "types" argument) or
(b) for specific functions (e.g., switching to a more specific protocol
like __array_ufunc__).
4. Removing the possibility of overriding specific functions altogether,
because we want to require using a more generic interface (e.g., to require
overriding np.stack implicitly via np.expand_dims/np.concatenate instead).

Possible changes (1) or (2) would be good reasons for requiring end-users
to make an intentional choice to enable __array_function__. Either of these
changes would break everyone who uses the protocol. Concern about change
(1) would be a particularly good reason to require an opt-in, because it's
concerned about impact on users who don't use any new functionality.

Possible changes (3) or (4) would be annoying to users, but in my mind
would not justify requiring an explicit opt-in. Both could be achieved
without immediately breaking users by making use of a standard deprecation
cycle, i.e., by still calling __array_function__ but issuing a
FutureWarning. The main reason for requiring the explicit opt-in would be
if we don't want to need to bother with a deprecation cycle.

To be clear, with the exception of change (3)(b), I don't think any of
these are particularly likely or desirable outcomes. So in my personal
opinion, I don't think we need the explicit opt-in. But I'm willing to make
to go ahead with requiring the opt-in if that's what it takes to get
everyone on board -- as long as we don't make the opt-in too onerous in a
way that imposes hard limits on this protocol can be used. (I would be OK
with an explicit Python API for enabling this feature that raises a
FutureWarning when it's called, without any API for turning it off.)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180829/a5b8d305/attachment-0001.html>

More information about the NumPy-Discussion mailing list