[Numpy-discussion] __array_function related regression for 1.17.0rc1

[Stephan Hoyer]

On Tue, Jul 2, 2019 at 8:16 AM Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias <jni at fastmail.com>
> wrote:
>
>> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote:
>>
>> This is addressed in the NEP, see bullet 1 under "Partial implementation
>> of NumPy's API":
>>
>> http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api
>>
>> My concern is that fallback coercion behavior makes it difficult to
>> reliably implement "strict" overrides of NumPy's API. Fallback coercion is
>> certainly useful for interactive use, but it isn't really appropriate for
>> libraries.
>>
>>
> Do you mean "fallback coercion in NumPy itself", or "at all"? Right now
> there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users
> will keep wanting to do that. Forcing everyone to write
> `np.median(np.array(some_dask_array))` serves no purpose. So the coercion
> has to be somewhere. You're arguing that that's up to Dask et al I think?
>

Yes, I'm arguing this is up to dask to maintain backwards compatibility --
or not, as the maintainers see fit.

NumPy adding dispatching with __array_function__ did not break any existing
code, until the maintainers of other libraries started adding
__array_function__ methods. I hope that the risks of implementing such
experimental methods were self-evident.


> Putting it in Dask right now still doesn't address Juan's backwards compat
> concern, but perhaps that could be bridged with a Dask bugfix release and
> some short-lived pain.
>

I really think this is the best (only?) path forward.

I'm not convinced that this shouldn't be fixed in NumPy though. Your
> concern "reliably implement "strict" overrides of NumPy's API" is a bit
> abstract. Overriding the _whole_ NumPy API is definitely undesirable. If
> we'd have a reference list somewhere about every function that is handled
> with __array_function__, then would that address your concern? Such a list
> could be auto-generated fairly easily.
>

By "reliably implement strict overrides" I mean the ability to ensure that
every operation either uses an override or raises an informative error --
making it very clear which operation needs to be implemented or avoided.

It's true that we didn't really consider "always issuing warnings" as a
long term solution in the NEP. I can see how this would simply a backwards
compatibility story for libraries like dask, but in general, I really don't
like warnings: Using them like exceptions can easily result in code that is
partially broken or that fails later for non-obvious reasons. There's a
reason why Python's errors stop execution flow, until errors in languages
like PHP or JavaScript.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190702/ced65d8f/attachment.html>