[Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules

[Sebastian Berg]

On Thu, 2020-04-09 at 13:52 +0200, Ralf Gommers wrote:
> On Thu, Apr 9, 2020 at 12:02 AM Sebastian Berg <
> sebastian at sipsolutions.net>
> wrote:
> 
<snip>
> > 
> 
> I think it would be nice to have a separate NEP 37 implementation
> outside
> of NumPy to play with. Unlike __array_function__, I don't think it
> has to
> go into NumPy immediately. This avoids the whole "experimental API"
> issue,

Fair enough, I have created a hopefully working start here:

https://github.com/seberg/numpy_dispatch

(this is not tested much at all yet, so it could be very buggy).

There are a couple of additional features that I added.

1. A global opt-in (it is impossible to opt-out once opted in!)
2. A local opt-in (to guarantee opt-in if global flag is not set)
3. I added features to allow transitioning::

      get_array_module(*arrays, modules="numpy",
            future_modules=("dask.array", "cupy"), fallback="warn")

   Will give FutureWarning/DeprecationWarning where necessary, in the
   above "numpy" is supported, dask and cupy are supported but not
   enabled by default. `None` works to say "all modules".
   Once the transition is done, just move dask and cupy into `modules`
   and remove `fallback=None`.
4. If there are FutureWarnings/DeprecationWarnigs the user needs to be
   able to opt-in to future behaviour. Opting out can be done by
   casting inputs. Opting-in is done using::

      with future_dispatch_behavior():
          call_library_function()

Obviously, we may not want these features, but I was curious how we
could provide the tools to allow clean transitions.

Both context managers should be thread-safe, but I did not test that.

The best try would probably be cupy and sklearn again, so I will give a
ping on the sklearn PR. To make that easier, I tried to hack a bit of a
"util" to allow testing (please scroll down on the readme on github).

Best,

Sebastian



> it would be quite useful to test this with, e.g., CuPy + scikit-learn
> without being stuck with any decisions in a released NumPy version.
> Also
> makes switching on/off very easy for users, just (don't) `pip install
> numpy-array-module`.
> 
> Cheers,
> Ralf

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200409/fc37cb73/attachment.sig>