[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API
einstein.edison at gmail.com
Tue Sep 10 09:05:55 EDT 2019
On 10.09.19 05:32, Stephan Hoyer wrote:
> On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <ralf.gommers at gmail.com
> <mailto:ralf.gommers at gmail.com>> wrote:
> I think we've chosen to try the former - dispatch on functions so
> we can reuse the NumPy API. It could work out well, it could give
> some long-term maintenance issues, time will tell. The question is
> now if and how to plug the gap that __array_function__ left. It's
> main limitation is "doesn't work for functions that don't have an
> array-like input" - that left out ~10-20% of functions. So now we
> have a proposal for a structural solution to that last 10-20%. It
> seems logical to want that gap plugged, rather than go back and
> say "we shouldn't have gone for the first 80%, so let's go no
> I'm excited about solving the remaining 10-20% of use cases for
> flexible array dispatching, but the unumpy interface suggested here
> (numpy.overridable) feels like a redundant redo of __array_function__
> and __array_ufunc__.
> I would much rather continue to develop specialized protocols for the
> remaining usecases. Summarizing those I've seen in this thread, these
> 1. Overrides for customizing array creation and coercion.
> 2. Overrides to implement operations for new dtypes.
> 3. Overriding implementations of NumPy functions, e.g., FFT and ufuncs
> with MKL.
> (1) could mostly be solved by adding np.duckarray() and another
> function for duck array coercion. There is still the matter of
> overriding np.zeros and the like, which perhaps justifies another new
> protocol, but in my experience the use-cases for truly an array from
> scratch are quite rare.
While they're rare for libraries like XArray; CuPy, Dask and
PyData/Sparse need these.
> (2) should be tackled as part of overhauling NumPy's dtype system to
> better support user defined dtypes. But it should definitely be in the
> form of specialized protocols, e.g., which pass in preallocated arrays
> to into ufuncs for a new dtype. By design, new dtypes should not be
> able to customize the semantics of array *structure*.
We already have a split in the type system with e.g. Cython's buffers,
Numba's parallel type system. This is a different issue altogether, e.g.
allowing a unyt dtype to spawn a unyt array, rather than forcing a
re-write of unyt to cooperate with NumPy's new dtype system.
> (3) could potentially motivate a new solution, but it should exist
> *inside* of select existing NumPy implementations, after checking for
> overrides with __array_function__. If the only option NumPy provides
> for overriding np.fft is to implement np.overrideable.fft, I doubt
> that would suffice to convince MKL developers from monkey patching it
> -- they already decided that a separate namespace is not good enough
> for them.
That has already been addressed by Ralf in another email. We're
proposing to merge that into NumPy proper.
Also, you're missing a few:
4. Having default implementations that allow overrides of a large part
of the API while defining only a small part. This holds for e.g.
5. Generation of Random numbers (overriding RandomState). CuPy has its
own implementation which would be nice to override.
> I also share Nathaniel's concern that the overrides in unumpy are too
> powerful, by allowing for control from arbitrary function arguments
> and even *non-local* control (i.e., global variables) from context
> managers. This level of flexibility can make code very hard to debug,
> especially in larger codebases.
Backend switching needs global context, in any case. There isn't a good
way around that other than the class dundermethods outlined in another
thread, which would require rewrites of large amounts of code.
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the NumPy-Discussion