[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

[Sebastian Berg]

On Mon, 2019-09-09 at 22:26 -0700, Eric Wieser wrote:
> > In other words `np.arange<type(duck_array)>(100)` (but
> with a completely different syntax, probably hidden away only for
> libraries to use).
> 
> It sounds an bit like you're describing factory classmethods there.
> Is the solution to this problem to move (leaving behind aliases)
> `np.arange` to `ndarray.arange`, `np.zeros` to `ndarray.zeros`, etc -
> callers then would use `type(duckarray).zeros` if they're trying to
> generalize.
> 

Yeah, factory classmethod is probably the better way to describe it.
The question is where you hide them away conveniently (and how to
access them). And of course if/what completely different alternatives
exist.

In a sense, `__array_function__` is a bit like a collection of operator
dunder methods, I guess. So, we need another collection for
classmethods. And that was the quick, possibly silly, idea to also use
`__array_function__`.

So yeah, there is not much of a point in not simply creating another
place for them, or even using individual dunder classmethods. But we
still an "operator"/function to access them nicely, unless we want to
force `type(duckarray).…` on library authors.

I guess the important thing is mostly what would be convenient to
downstreams implementers.

- Sebastian


> Eric
> 
> On Mon, Sep 9, 2019, 21:18 Sebastian Berg <sebastian at sipsolutions.net
> > wrote:
> > On Mon, 2019-09-09 at 20:32 -0700, Stephan Hoyer wrote:
> > > On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <
> > ralf.gommers at gmail.com>
> > > wrote:
> > > > I think we've chosen to try the former - dispatch on functions
> > so
> > > > we can reuse the NumPy API. It could work out well, it could
> > give
> > > > some long-term maintenance issues, time will tell. The question
> > is
> > > > now if and how to plug the gap that __array_function__ left.
> > It's
> > > > main limitation is "doesn't work for functions that don't have
> > an
> > > > array-like input" - that left out ~10-20% of functions. So now
> > we
> > > > have a proposal for a structural solution to that last 10-20%.
> > It
> > > > seems logical to want that gap plugged, rather than go back and
> > say
> > > > "we shouldn't have gone for the first 80%, so let's go no
> > further".
> > > > 
> > > 
> > > I'm excited about solving the remaining 10-20% of use cases for
> > > flexible array dispatching, but the unumpy interface suggested
> > here
> > > (numpy.overridable) feels like a redundant redo of
> > __array_function__
> > > and __array_ufunc__.
> > > 
> > > I would much rather continue to develop specialized protocols for
> > the
> > > remaining usecases. Summarizing those I've seen in this thread,
> > these
> > > include:
> > > 1. Overrides for customizing array creation and coercion.
> > > 2. Overrides to implement operations for new dtypes.
> > > 3. Overriding implementations of NumPy functions, e.g., FFT and
> > > ufuncs with MKL.
> > > 
> > > (1) could mostly be solved by adding np.duckarray() and another
> > > function for duck array coercion. There is still the matter of
> > > overriding np.zeros and the like, which perhaps justifies another
> > new
> > > protocol, but in my experience the use-cases for truly an array
> > from
> > > scratch are quite rare.
> > > 
> > 
> > There is an issue open about adding more functions for that. Made
> > me
> > wonder if giving a method of choosing the duck-array whose
> > `__array_function__` is used, could not solve it reasonably.
> > Similar to explicitly choosing a specific template version to call
> > in
> > templated code. In other words `np.arange<type(duck_array)>(100)`
> > (but
> > with a completely different syntax, probably hidden away only for
> > libraries to use).
> > 
> > 
> > Maybe it is indeed time to write up a list of options to plug that
> > hole, and then see where it brings us.
> > 
> > Best,
> > 
> > Sebastian
> > 
> > 
> > > (2) should be tackled as part of overhauling NumPy's dtype system
> > to
> > > better support user defined dtypes. But it should definitely be
> > in
> > > the form of specialized protocols, e.g., which pass in
> > preallocated
> > > arrays to into ufuncs for a new dtype. By design, new dtypes
> > should
> > > not be able to customize the semantics of array *structure*.
> > >
> > > (3) could potentially motivate a new solution, but it should
> > exist
> > > *inside* of select existing NumPy implementations, after checking
> > for
> > > overrides with __array_function__. If the only option NumPy
> > provides
> > > for overriding np.fft is to implement np.overrideable.fft, I
> > doubt
> > > that would suffice to convince MKL developers from monkey
> > patching it
> > > -- they already decided that a separate namespace is not good
> > enough
> > > for them.
> > > 
> > > I also share Nathaniel's concern that the overrides in unumpy are
> > too
> > > powerful, by allowing for control from arbitrary function
> > arguments
> > > and even *non-local* control (i.e., global variables) from
> > context
> > > managers. This level of flexibility can make code very hard to
> > debug,
> > > especially in larger codebases.
> > > 
> > > Best,
> > > Stephan
> > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at python.org
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190909/79d20ecb/attachment.sig>