[Numpy-discussion] __array_function related regression for 1.17.0rc1

[Ralf Gommers]

On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias <jni at fastmail.com> wrote:

> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote:
>
> This is addressed in the NEP, see bullet 1 under "Partial implementation
> of NumPy's API":
>
> http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api
>
> My concern is that fallback coercion behavior makes it difficult to
> reliably implement "strict" overrides of NumPy's API. Fallback coercion is
> certainly useful for interactive use, but it isn't really appropriate for
> libraries.
>
>
Do you mean "fallback coercion in NumPy itself", or "at all"? Right now
there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users
will keep wanting to do that. Forcing everyone to write
`np.median(np.array(some_dask_array))` serves no purpose. So the coercion
has to be somewhere. You're arguing that that's up to Dask et al I think?

Putting it in Dask right now still doesn't address Juan's backwards compat
concern, but perhaps that could be bridged with a Dask bugfix release and
some short-lived pain.

I'm not convinced that this shouldn't be fixed in NumPy though. Your
concern "reliably implement "strict" overrides of NumPy's API" is a bit
abstract. Overriding the _whole_ NumPy API is definitely undesirable. If
we'd have a reference list somewhere about every function that is handled
with __array_function__, then would that address your concern? Such a list
could be auto-generated fairly easily.

>
> In contrast to putting this into NumPy, if a library like dask prefers to
> issue warnings or even keep around fallback coercion indefinitely (not that
> I would recommend it), they can do that by putting it in their
> __array_function__ implementation.
>
>
> I get the above concerns, and thanks for bringing them up, Stephan, as I'd
> only skimmed the NEP the first time around and missed them. Nevertheless,
> the fact is that the current behaviour breaks user code that was perfectly
> valid until NumPy 1.16, which seems, well, insane. So, warning for a few
> versions followed raising seems like the only way forward to me. The NEP
> explicitly states “We would like to gain experience with how
> __array_function__ is actually used before making decisions that would be
> difficult to roll back.” I think that this breakage *is* that experience,
> and the decision right now should be not to break user code with no warning
> period.
>

> I'm also wondering where the list of functions that must be implemented
> can be found, so that libraries like dask and CuPy can be sure that they
> have a complete implementation, and further typeerrors won't be raised with
> their arrays.
>

This is one of the reasons I'm working on
https://github.com/Quansight-Labs/rnumpy. It doesn't make sense for any
library to copy the whole NumPy API, it's way too large with lots of stuff
in there that's only there for backwards compat and has a better
alternative or shouldn't be in NumPy in the first place.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190702/27d624b0/attachment-0001.html>