[Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules

Sun Apr 12 07:00:05 EDT 2020

On Fri, Apr 10, 2020 at 7:18 PM Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> On Fri, 2020-04-10 at 18:19 +0200, Ralf Gommers wrote:
> > On Fri, Apr 10, 2020 at 3:03 PM Sebastian Berg <
> > sebastian at sipsolutions.net>
> > wrote:
> >
> > > On Fri, 2020-04-10 at 12:27 +0200, Ralf Gommers wrote:
> > > > > 3. I added features to allow transitioning::
> > > > >
> > > > >       get_array_module(*arrays, modules="numpy",
> > > > >             future_modules=("dask.array", "cupy"),
> > > > > fallback="warn")
> > > > >
> > > > >    Will give FutureWarning/DeprecationWarning where necessary,
> > > > > in
> > > > the
> > > > >    above "numpy" is supported, dask and cupy are supported but
> > > > > not
> > > > >    enabled by default. `None` works to say "all modules".
> > > > >    Once the transition is done, just move dask and cupy into
> > > > `modules`
> > > > >    and remove `fallback=None`.
> > > > >
> > > >
> > > > So future_modules explicitly excludes compatible libraries that
> > > > are
> > > > not
> > > > listed. Why would you want anyone to do that? I don't understand
> > > > "supported
> > > > but not enabled", and it looks undesirable to me to special-case
> > > > any
> > > > library in this mechanism.
> > >
> > > We hav two (or three) types of modules (either could be "all").
> > >
> >
> > I think we only have modules that implement __array_module__, and
> > ones that
> > don't.
> >
> >
> > > 1. Supported modules that we dispatch to.
> > > 2. Modules that are supported but will be dispatched to by default
> > > only
> > >    in the future. So if the user got a future_module, they will get
> > > a
> > >    FutureWarning. They have to chose to cast the inputs or opt-in
> > > to
> > >    the future behaviour.
> > > 3. Unsupported modules: If this is resolved it is an error. I
> > > currently
> > >    assume that this does not need to be a negative list.
> > >
> > > You need to distinguish those somehow, since you need a way to
> > > transition. Even if you expect that modules would always be *all*
> > > modules, `numpy` is still the only accepted module originally.
> > >
> > > So, as I said, `future_modules` is only about transitioning and
> > > enabling `FutureWarning`s. Does not have to live there, but we need
> > > a
> > > way to transition.
> > >
> >
> > Sorry, I still don't get it - transition what? You seem to be
> > operating on
> > the assumption that the users of get_array_module want or need to
> > control
> > which numpy-like libraries they allow and which they don't. That
> > seems
> > fundamentally wrong. How would you treat, for example, an array
> > library
> > that is developed privately inside some company?
> >
>
> Well, you still need to transition from NumPy -> allow everything, so
> for now please just ignore that part if you like and use/assume:
>
>    get_array_module(...,
>        modules="numpy", future_modules=None, fallback="warn")
>
> during the transition, and:
>
>    get_array_module(...)
>
> after it. After all this is a draft-project right now, so it is just as
> much about trying out what can be done.
> It is not unlikely that this transition burden will be put more on the
> library in any case, but it shows that it can be done.
>
>
> As to my "fundamentally wrong" assumption. Should libraries goal be to
> support everything? Definitely!
>
> But... I do not want to make that decision for libraries, so I if
> library authors tell me that they have no interest in it, all the
> better. Until then I am more than happy to keep that option on the
> table. If just as a thought for library authors to consider their
> options.
>
> Possible, brainstorming, reasons could be:
>
> 1. Say I currently heavily use cython code, so I am limited to NumPy
> (or at least arrays that can expose a buffer/`__array_interface__`).
> Now if someone adds a CUDA implementation, I would support cupy arrays,
> but not distributed arrays.
> I admit maybe checking that at function entry like this is the wrong
> approach there.
>

If you need a particular feature, then checking for that feature (e.g.
`hasattr(__array_interface__)`, and same for __cuda_array_interface__)
seems like the right thing to do.

Ralf

> 2. To limit to certain types is to say "We know (and test) that our
> library works with xarray, Dask, NumPy, and CuPy". Now you can say that
> is also a misconception, because if you stick to just NumPy API you
> should know that it will "just work" with everything. But in practice
> it seems like it might happen?
> In that case you may want to actually allow any odd array and just put
> a warning, a bit like the transition warnings I put in for testing.
>
>
> ---
>
> There are two other things I am wondering about.
>
> 1. Subclasses may want to return their superclasses module (even by
> default?), in which case their behaviour depends on the superclass
> module behaviour. Further a library would need to use `np.asanyarray()`
> to prevent the subclass from being cast to the superclass.
>
> 2. There is one transition that does not quite exists. What if an
> array-like starts implementing or expands `array-module`?
> That seems fine, but in that case the array-like will have to provide
> the `opt-in` context manager with a FutureWarning.
> The transition from no `__array_module__` to implementing it may need
> some thought, but I expect it is fine: The array-like simply always
> gives a FutureWarning, although it cannot know what will actually
> happen in the future (no change, error, or array-like takes control).
>
> - Sebastian
>
>
> > Cheers,
> > Ralf
> >
> >
> >
> > > These options do not have to be handled by us, it only helps here
> > > with
> > > having context managers to opt-in to new behaviour, and maybe to
> > > get an
> > > idea for how transitions can look like.
> > > Alternatively, we could all to create project specific context
> > > managers
> > > to do the same and avoid possible scoping issues even more.
> > >
> > > - Sebastian
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at python.org
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200412/96c46331/attachment.html>