[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API
sebastian at sipsolutions.net
Tue Sep 10 02:11:58 EDT 2019
On Mon, 2019-09-09 at 22:26 -0700, Eric Wieser wrote:
> > In other words `np.arange<type(duck_array)>(100)` (but
> with a completely different syntax, probably hidden away only for
> libraries to use).
> It sounds an bit like you're describing factory classmethods there.
> Is the solution to this problem to move (leaving behind aliases)
> `np.arange` to `ndarray.arange`, `np.zeros` to `ndarray.zeros`, etc -
> callers then would use `type(duckarray).zeros` if they're trying to
Yeah, factory classmethod is probably the better way to describe it.
The question is where you hide them away conveniently (and how to
access them). And of course if/what completely different alternatives
In a sense, `__array_function__` is a bit like a collection of operator
dunder methods, I guess. So, we need another collection for
classmethods. And that was the quick, possibly silly, idea to also use
So yeah, there is not much of a point in not simply creating another
place for them, or even using individual dunder classmethods. But we
still an "operator"/function to access them nicely, unless we want to
force `type(duckarray).…` on library authors.
I guess the important thing is mostly what would be convenient to
> On Mon, Sep 9, 2019, 21:18 Sebastian Berg <sebastian at sipsolutions.net
> > wrote:
> > On Mon, 2019-09-09 at 20:32 -0700, Stephan Hoyer wrote:
> > > On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <
> > ralf.gommers at gmail.com>
> > > wrote:
> > > > I think we've chosen to try the former - dispatch on functions
> > so
> > > > we can reuse the NumPy API. It could work out well, it could
> > give
> > > > some long-term maintenance issues, time will tell. The question
> > is
> > > > now if and how to plug the gap that __array_function__ left.
> > It's
> > > > main limitation is "doesn't work for functions that don't have
> > an
> > > > array-like input" - that left out ~10-20% of functions. So now
> > we
> > > > have a proposal for a structural solution to that last 10-20%.
> > It
> > > > seems logical to want that gap plugged, rather than go back and
> > say
> > > > "we shouldn't have gone for the first 80%, so let's go no
> > further".
> > > >
> > >
> > > I'm excited about solving the remaining 10-20% of use cases for
> > > flexible array dispatching, but the unumpy interface suggested
> > here
> > > (numpy.overridable) feels like a redundant redo of
> > __array_function__
> > > and __array_ufunc__.
> > >
> > > I would much rather continue to develop specialized protocols for
> > the
> > > remaining usecases. Summarizing those I've seen in this thread,
> > these
> > > include:
> > > 1. Overrides for customizing array creation and coercion.
> > > 2. Overrides to implement operations for new dtypes.
> > > 3. Overriding implementations of NumPy functions, e.g., FFT and
> > > ufuncs with MKL.
> > >
> > > (1) could mostly be solved by adding np.duckarray() and another
> > > function for duck array coercion. There is still the matter of
> > > overriding np.zeros and the like, which perhaps justifies another
> > new
> > > protocol, but in my experience the use-cases for truly an array
> > from
> > > scratch are quite rare.
> > >
> > There is an issue open about adding more functions for that. Made
> > me
> > wonder if giving a method of choosing the duck-array whose
> > `__array_function__` is used, could not solve it reasonably.
> > Similar to explicitly choosing a specific template version to call
> > in
> > templated code. In other words `np.arange<type(duck_array)>(100)`
> > (but
> > with a completely different syntax, probably hidden away only for
> > libraries to use).
> > Maybe it is indeed time to write up a list of options to plug that
> > hole, and then see where it brings us.
> > Best,
> > Sebastian
> > > (2) should be tackled as part of overhauling NumPy's dtype system
> > to
> > > better support user defined dtypes. But it should definitely be
> > in
> > > the form of specialized protocols, e.g., which pass in
> > preallocated
> > > arrays to into ufuncs for a new dtype. By design, new dtypes
> > should
> > > not be able to customize the semantics of array *structure*.
> > >
> > > (3) could potentially motivate a new solution, but it should
> > exist
> > > *inside* of select existing NumPy implementations, after checking
> > for
> > > overrides with __array_function__. If the only option NumPy
> > provides
> > > for overriding np.fft is to implement np.overrideable.fft, I
> > doubt
> > > that would suffice to convince MKL developers from monkey
> > patching it
> > > -- they already decided that a separate namespace is not good
> > enough
> > > for them.
> > >
> > > I also share Nathaniel's concern that the overrides in unumpy are
> > too
> > > powerful, by allowing for control from arbitrary function
> > arguments
> > > and even *non-local* control (i.e., global variables) from
> > context
> > > managers. This level of flexibility can make code very hard to
> > debug,
> > > especially in larger codebases.
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at python.org
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 833 bytes
Desc: This is a digitally signed message part
More information about the NumPy-Discussion