[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

[Stephan Hoyer]

On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

> I think we've chosen to try the former - dispatch on functions so we can
> reuse the NumPy API. It could work out well, it could give some long-term
> maintenance issues, time will tell. The question is now if and how to plug
> the gap that __array_function__ left. It's main limitation is "doesn't work
> for functions that don't have an array-like input" - that left out ~10-20%
> of functions. So now we have a proposal for a structural solution to that
> last 10-20%. It seems logical to want that gap plugged, rather than go back
> and say "we shouldn't have gone for the first 80%, so let's go no further".
>

I'm excited about solving the remaining 10-20% of use cases for flexible
array dispatching, but the unumpy interface suggested here
(numpy.overridable) feels like a redundant redo of __array_function__ and
__array_ufunc__.

I would much rather continue to develop specialized protocols for the
remaining usecases. Summarizing those I've seen in this thread, these
include:
1. Overrides for customizing array creation and coercion.
2. Overrides to implement operations for new dtypes.
3. Overriding implementations of NumPy functions, e.g., FFT and ufuncs with
MKL.

(1) could mostly be solved by adding np.duckarray() and another function
for duck array coercion. There is still the matter of overriding np.zeros
and the like, which perhaps justifies another new protocol, but in my
experience the use-cases for truly an array from scratch are quite rare.

(2) should be tackled as part of overhauling NumPy's dtype system to better
support user defined dtypes. But it should definitely be in the form of
specialized protocols, e.g., which pass in preallocated arrays to into
ufuncs for a new dtype. By design, new dtypes should not be able to
customize the semantics of array *structure*.

(3) could potentially motivate a new solution, but it should exist *inside*
of select existing NumPy implementations, after checking for overrides with
__array_function__. If the only option NumPy provides for overriding np.fft
is to implement np.overrideable.fft, I doubt that would suffice to convince
MKL developers from monkey patching it -- they already decided that a
separate namespace is not good enough for them.

I also share Nathaniel's concern that the overrides in unumpy are too
powerful, by allowing for control from arbitrary function arguments and
even *non-local* control (i.e., global variables) from context managers.
This level of flexibility can make code very hard to debug, especially in
larger codebases.

Best,
Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190909/4e3878c0/attachment.html>