[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

Eric Wieser wieser.eric+numpy at gmail.com
Tue Sep 10 01:26:14 EDT 2019


> In other words `np.arange<type(duck_array)>(100)` (but
with a completely different syntax, probably hidden away only for
libraries to use).

It sounds an bit like you're describing factory classmethods there. Is the
solution to this problem to move (leaving behind aliases) `np.arange` to
`ndarray.arange`, `np.zeros` to `ndarray.zeros`, etc - callers then would
use `type(duckarray).zeros` if they're trying to generalize.

Eric

On Mon, Sep 9, 2019, 21:18 Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Mon, 2019-09-09 at 20:32 -0700, Stephan Hoyer wrote:
> > On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <ralf.gommers at gmail.com>
> > wrote:
> > > I think we've chosen to try the former - dispatch on functions so
> > > we can reuse the NumPy API. It could work out well, it could give
> > > some long-term maintenance issues, time will tell. The question is
> > > now if and how to plug the gap that __array_function__ left. It's
> > > main limitation is "doesn't work for functions that don't have an
> > > array-like input" - that left out ~10-20% of functions. So now we
> > > have a proposal for a structural solution to that last 10-20%. It
> > > seems logical to want that gap plugged, rather than go back and say
> > > "we shouldn't have gone for the first 80%, so let's go no further".
> > >
> >
> > I'm excited about solving the remaining 10-20% of use cases for
> > flexible array dispatching, but the unumpy interface suggested here
> > (numpy.overridable) feels like a redundant redo of __array_function__
> > and __array_ufunc__.
> >
> > I would much rather continue to develop specialized protocols for the
> > remaining usecases. Summarizing those I've seen in this thread, these
> > include:
> > 1. Overrides for customizing array creation and coercion.
> > 2. Overrides to implement operations for new dtypes.
> > 3. Overriding implementations of NumPy functions, e.g., FFT and
> > ufuncs with MKL.
> >
> > (1) could mostly be solved by adding np.duckarray() and another
> > function for duck array coercion. There is still the matter of
> > overriding np.zeros and the like, which perhaps justifies another new
> > protocol, but in my experience the use-cases for truly an array from
> > scratch are quite rare.
> >
>
> There is an issue open about adding more functions for that. Made me
> wonder if giving a method of choosing the duck-array whose
> `__array_function__` is used, could not solve it reasonably.
> Similar to explicitly choosing a specific template version to call in
> templated code. In other words `np.arange<type(duck_array)>(100)` (but
> with a completely different syntax, probably hidden away only for
> libraries to use).
>
>
> Maybe it is indeed time to write up a list of options to plug that
> hole, and then see where it brings us.
>
> Best,
>
> Sebastian
>
>
> > (2) should be tackled as part of overhauling NumPy's dtype system to
> > better support user defined dtypes. But it should definitely be in
> > the form of specialized protocols, e.g., which pass in preallocated
> > arrays to into ufuncs for a new dtype. By design, new dtypes should
> > not be able to customize the semantics of array *structure*.
> >
> > (3) could potentially motivate a new solution, but it should exist
> > *inside* of select existing NumPy implementations, after checking for
> > overrides with __array_function__. If the only option NumPy provides
> > for overriding np.fft is to implement np.overrideable.fft, I doubt
> > that would suffice to convince MKL developers from monkey patching it
> > -- they already decided that a separate namespace is not good enough
> > for them.
> >
> > I also share Nathaniel's concern that the overrides in unumpy are too
> > powerful, by allowing for control from arbitrary function arguments
> > and even *non-local* control (i.e., global variables) from context
> > managers. This level of flexibility can make code very hard to debug,
> > especially in larger codebases.
> >
> > Best,
> > Stephan
> >
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190909/cc22c072/attachment-0001.html>


More information about the NumPy-Discussion mailing list