On Tue, Jul 2, 2019 at 8:16 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:

On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias <jni@fastmail.com> wrote:
On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote:
This is addressed in the NEP, see bullet 1 under "Partial implementation of NumPy's API":

My concern is that fallback coercion behavior makes it difficult to reliably implement "strict" overrides of NumPy's API. Fallback coercion is certainly useful for interactive use, but it isn't really appropriate for libraries.

Do you mean "fallback coercion in NumPy itself", or "at all"? Right now there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users will keep wanting to do that. Forcing everyone to write  `np.median(np.array(some_dask_array))` serves no purpose. So the coercion has to be somewhere. You're arguing that that's up to Dask et al I think?

Yes, I'm arguing this is up to dask to maintain backwards compatibility -- or not, as the maintainers see fit.

NumPy adding dispatching with __array_function__ did not break any existing code, until the maintainers of other libraries started adding __array_function__ methods. I hope that the risks of implementing such experimental methods were self-evident.
Putting it in Dask right now still doesn't address Juan's backwards compat concern, but perhaps that could be bridged with a Dask bugfix release and some short-lived pain. 

I really think this is the best (only?) path forward.

I'm not convinced that this shouldn't be fixed in NumPy though. Your concern "reliably implement "strict" overrides of NumPy's API" is a bit abstract. Overriding the _whole_ NumPy API is definitely undesirable. If we'd have a reference list somewhere about every function that is handled with __array_function__, then would that address your concern? Such a list could be auto-generated fairly easily.

By "reliably implement strict overrides" I mean the ability to ensure that every operation either uses an override or raises an informative error -- making it very clear which operation needs to be implemented or avoided.

It's true that we didn't really consider "always issuing warnings" as a long term solution in the NEP. I can see how this would simply a backwards compatibility story for libraries like dask, but in general, I really don't like warnings: Using them like exceptions can easily result in code that is partially broken or that fails later for non-obvious reasons. There's a reason why Python's errors stop execution flow, until errors in languages like PHP or JavaScript.