[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

Sebastian Berg sebastian at sipsolutions.net
Tue Sep 10 13:51:04 EDT 2019


On Tue, 2019-09-10 at 17:28 +0200, Hameer Abbasi wrote:
> On 07.09.19 22:06, Sebastian Berg wrote:
> > On Fri, 2019-09-06 at 14:45 -0700, Ralf Gommers wrote:
> > 
> > <snip>
> > 
> > Let me try to move the discussion from the github issue here (this
> > may
> > not be the best place). (
> > https://github.com/numpy/numpy/issues/14441
> > which asked for easier creation functions together with
> > `__array_function__`).
> > 
> > I think an important note mentioned here is how users interact with
> > unumpy, vs. __array_function__. The former is an explicit opt-in,
> > while
> > the latter is implicit choice based on an `array-like` abstract
> > base
> > class and functional type based dispatching.
> > 
> > To quote NEP 18 on this: "The downsides are that this would require
> > an
> > explicit opt-in from all existing code, e.g., import numpy.api as
> > np,
> > and in the long term would result in the maintenance of two
> > separate
> > NumPy APIs. Also, many functions from numpy itself are already
> > overloaded (but inadequately), so confusion about high vs. low
> > level
> > APIs in NumPy would still persist."
> > (I do think this is a point we should not just ignore, `uarray` is
> > a
> > thin layer, but it has a big surface area)
> > 
> > Now there are things where explicit opt-in is obvious. And the FFT
> > example is one of those, there is no way to implicitly choose
> > another
> > backend (except by just replacing it, i.e. monkeypatching) [1]. And
> > right now I think these are _very_ different.
> > 
> > 
> > Now for the end-users choosing one array-like over another, seems
> > nicer
> > as an implicit mechanism (why should I not mix sparse, dask and
> > numpy
> > arrays!?). This is the promise `__array_function__` tries to make.
> > Unless convinced otherwise, my guess is that most library authors
> > would
> > strive for implicit support (i.e. sklearn, skimage, scipy).
> You can, once you register the backend it becomes implicit, so all 
> backends are tried until one succeeds. Unless you explicitly say "I
> do 
> not want another backend" (only/coerce=True).

The thing here being "once you register the backend". Thus requiring at
least in some form an explicit opt-in by the end user. Also, unless you
use the with statement (with all the scoping rules attached), you
cannot plug the coercion/creation hole left by `__array_function__`.

> > Circling back to creation and coercion. In a purely Object type
> > system,
> > these would be classmethods, I guess, but in NumPy and the
<snip>

> > def my_library_func(array_like):
> >     exp = np.exp(array_like)
> >     idx = np.arange(len(exp))
> >     return idx, exp
> > 
> > Would have all the information for implicit opt-in/Array-like
> > support,
> > but cannot do it right now. This is what I have been wondering, if
> > uarray/unumpy, can in some way help me make this work (even
> > _without_
> > the end user opting in). The reason is that simply, right now I am
> > very
> > clear on the need for this use case, but not sure about the need
> > for
> > end user opt in, since end users can just use dask.arange().
> 
> Sure, the end user can, but library authors cannot. And end users
> may 
> want to easily port code to GPU or between back-ends, just as
> library 
> authors might.

Yes, but library authors want to solve the particular thing above right
now, and I am still not sure how uarray helps there. If it does, then
only with a added complexity _and_ (at least currently) explicit end-
user opt-in.

Now, I am not a particularly good judge for these things, but I have
been trying to figure out how things can improve with it and still I am
tempted to say that uarray is a giant step in no particular direction
at all.
Of course it _can_ solve everything, but right now it seems like it
might require a py2 -> py3 like transition. And even then it is so
powerful, that it probably comes with its own bunch of issues (such as
far away side effects due to scoping of with statements).

Best,

Sebastian



> > Cheers,
> > 
> > Sebastian
> > 
> > 
> > [1] To be honest, I do think a lot of the "issues" around
> > monkeypatching exists just as much with backend choosing, the main
> > difference seems to me that a lot of that:
> >     1. monkeypatching was not done explicit
> >        (import mkl_fft; mkl_fft.monkeypatch_numpy())?
> >     2. A backend system allows libaries to prefer one locally?
> >        (which I think is a big advantage)
> > 
> > [2] There are the options of adding `linspace_like` functions
> > somewhere
> > in a numpy submodule, or adding `linspace(...,
> > array_type=np.ndarray)`,
> > or simply inventing a new "protocl" (which is not really a
> > protocol?),
> > and make it `ndarray.__numpy_like_creation_functions__.arange()`.
> 
> Handling things like RandomState can get complicated here.
> 
> <snip>
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190910/1d792fa5/attachment.sig>


More information about the NumPy-Discussion mailing list