[Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules

[Sebastian Berg]

On Thu, 2020-04-09 at 22:11 -0500, Sebastian Berg wrote:
> On Thu, 2020-04-09 at 13:52 +0200, Ralf Gommers wrote:
> > On Thu, Apr 9, 2020 at 12:02 AM Sebastian Berg <
> > sebastian at sipsolutions.net>
> > wrote:
> > 
> <snip>
> > 
> > I think it would be nice to have a separate NEP 37 implementation
> > outside
> > of NumPy to play with. Unlike __array_function__, I don't think it
> > has to
> > go into NumPy immediately. This avoids the whole "experimental API"
> > issue,
> 
> Fair enough, I have created a hopefully working start here:
> 
> https://github.com/seberg/numpy_dispatch
> 
> (this is not tested much at all yet, so it could be very buggy).
> 
> There are a couple of additional features that I added.
> 
> 1. A global opt-in (it is impossible to opt-out once opted in!)
> 2. A local opt-in (to guarantee opt-in if global flag is not set)
> 3. I added features to allow transitioning::
> 
>       get_array_module(*arrays, modules="numpy",
>             future_modules=("dask.array", "cupy"), fallback="warn")


There is no immediate need to put modules and future_modules and
fallback in there. The main convenience it gives is that we can more
easily provide the user to opt-in context manager to opt-in to the new
behaviour.
Without that, libraries will have to do these checks, that is not
difficult. But if we wish to provide a context manager to opt all of
that in, the library will need additional API to query our context
manager state. Or every library needs their own solution, which does
not seem desirable (although it means you cannot opt-in internal
functions accidentally to newer behaviour).

- Sebastian

> 
>    Will give FutureWarning/DeprecationWarning where necessary, in the
>    above "numpy" is supported, dask and cupy are supported but not
>    enabled by default. `None` works to say "all modules".
>    Once the transition is done, just move dask and cupy into
> `modules`
>    and remove `fallback=None`.
> 4. If there are FutureWarnings/DeprecationWarnigs the user needs to
> be
>    able to opt-in to future behaviour. Opting out can be done by
>    casting inputs. Opting-in is done using::
> 
>       with future_dispatch_behavior():
>           call_library_function()
> 
> Obviously, we may not want these features, but I was curious how we
> could provide the tools to allow clean transitions.
> 
> Both context managers should be thread-safe, but I did not test that.
> 
> The best try would probably be cupy and sklearn again, so I will give
> a
> ping on the sklearn PR. To make that easier, I tried to hack a bit of
> a
> "util" to allow testing (please scroll down on the readme on github).
> 
> Best,
> 
> Sebastian
> 
> 
> 
> > it would be quite useful to test this with, e.g., CuPy + scikit-
> > learn
> > without being stuck with any decisions in a released NumPy version.
> > Also
> > makes switching on/off very easy for users, just (don't) `pip
> > install
> > numpy-array-module`.
> > 
> > Cheers,
> > Ralf
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200409/345a09e3/attachment-0001.sig>