[Numpy-discussion] NEP 37: A dispatch protocol for NumPy-like modules

[Ralf Gommers]

On Sun, Feb 23, 2020 at 3:31 PM Stephan Hoyer <shoyer at gmail.com> wrote:

> On Thu, Feb 6, 2020 at 12:20 PM Sebastian Berg <sebastian at sipsolutions.net>
> wrote:
>
>>
>> Another thing about backward compatibility: What is our vision there
>> actually?
>> This NEP will *not* give the *end user* the option to opt-in! Here,
>> opt-in is really reserved to the *library user* (e.g. sklearn). (I did
>> not realize this clearly before)
>>
>> Thinking about that for a bit now, that seems like the right choice.
>> But it also means that the library requires an easy way of giving a
>> FutureWarning, to notify the end-user of the upcoming change. The end-
>> user will easily be able to convert to a NumPy array to keep the old
>> behaviour.
>> Once this warning is given (maybe during `get_array_module()`, the
>> array module object/context would preferably be passed around,
>> hopefully even between libraries. That provides a reasonable way to
>> opt-in to the new behaviour without a warning (mainly for library
>> users, end-users can silence the warning if they wish so).
>>
>
> I don't think NumPy needs to do anything about warnings. It is
> straightforward for libraries that want to use use get_array_module() to
> issue their own warnings before calling get_array_module(), if desired.
>

> Or alternatively, if a library is about to add a new __array_module__
> method, it is straightforward to issue a warning inside the new
> __array_module__ method before returning the NumPy functions.
>

I don't think this is quite enough. Sebastian points out a fairly important
issue. One of the main rationales for the whole NEP, and the argument in
multiple places (
https://numpy.org/neps/nep-0037-array-module.html#opt-in-vs-opt-out-for-users)
is that it's now opt-in while __array_function__ was opt-out. This isn't
really true - the problem is simply *moved*, from the duck array libraries
to the array-consuming libraries. The end user will still see the backwards
incompatible change, with no way to turn it off. It will be easier with
__array_module__ to warn users, but this should be expanded on in the NEP.

Also, I'm still not sure I agree with the tone of the discussion on this
topic. It's very heavily inspired by what the JAX devs are telling you (the
NEP still says PyTorch and scipy.sparse as well, but that's not true in
both cases). If you ask Dask and CuPy for example, they're quite happy with
__array_function__ and there haven't been many complaints about backwards
compat breakage.

Cheers,
Ralf



_______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20200223/693a3fa1/attachment.html>