[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

[Sebastian Berg]

On Mon, 2019-09-09 at 20:32 -0700, Stephan Hoyer wrote:
> On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> > I think we've chosen to try the former - dispatch on functions so
> > we can reuse the NumPy API. It could work out well, it could give
> > some long-term maintenance issues, time will tell. The question is
> > now if and how to plug the gap that __array_function__ left. It's
> > main limitation is "doesn't work for functions that don't have an
> > array-like input" - that left out ~10-20% of functions. So now we
> > have a proposal for a structural solution to that last 10-20%. It
> > seems logical to want that gap plugged, rather than go back and say
> > "we shouldn't have gone for the first 80%, so let's go no further".
> > 
> 
> I'm excited about solving the remaining 10-20% of use cases for
> flexible array dispatching, but the unumpy interface suggested here
> (numpy.overridable) feels like a redundant redo of __array_function__
> and __array_ufunc__.
> 
> I would much rather continue to develop specialized protocols for the
> remaining usecases. Summarizing those I've seen in this thread, these
> include:
> 1. Overrides for customizing array creation and coercion.
> 2. Overrides to implement operations for new dtypes.
> 3. Overriding implementations of NumPy functions, e.g., FFT and
> ufuncs with MKL.
> 
> (1) could mostly be solved by adding np.duckarray() and another
> function for duck array coercion. There is still the matter of
> overriding np.zeros and the like, which perhaps justifies another new
> protocol, but in my experience the use-cases for truly an array from
> scratch are quite rare.
> 

There is an issue open about adding more functions for that. Made me
wonder if giving a method of choosing the duck-array whose
`__array_function__` is used, could not solve it reasonably.
Similar to explicitly choosing a specific template version to call in
templated code. In other words `np.arange<type(duck_array)>(100)` (but
with a completely different syntax, probably hidden away only for
libraries to use).


Maybe it is indeed time to write up a list of options to plug that
hole, and then see where it brings us.

Best,

Sebastian


> (2) should be tackled as part of overhauling NumPy's dtype system to
> better support user defined dtypes. But it should definitely be in
> the form of specialized protocols, e.g., which pass in preallocated
> arrays to into ufuncs for a new dtype. By design, new dtypes should
> not be able to customize the semantics of array *structure*.
>
> (3) could potentially motivate a new solution, but it should exist
> *inside* of select existing NumPy implementations, after checking for
> overrides with __array_function__. If the only option NumPy provides
> for overriding np.fft is to implement np.overrideable.fft, I doubt
> that would suffice to convince MKL developers from monkey patching it
> -- they already decided that a separate namespace is not good enough
> for them.
> 
> I also share Nathaniel's concern that the overrides in unumpy are too
> powerful, by allowing for control from arbitrary function arguments
> and even *non-local* control (i.e., global variables) from context
> managers. This level of flexibility can make code very hard to debug,
> especially in larger codebases.
> 
> Best,
> Stephan
> 
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190909/08a19808/attachment.sig>