[Numpy-discussion] __array_function related regression for 1.17.0rc1

[Ralf Gommers]

On Tue, Jul 2, 2019 at 1:15 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Tue, Jul 2, 2019 at 8:38 AM Stephan Hoyer <shoyer at gmail.com> wrote:
>
>> On Tue, Jul 2, 2019 at 8:16 AM Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Jul 2, 2019 at 1:45 AM Juan Nunez-Iglesias <jni at fastmail.com>
>>> wrote:
>>>
>>>> On Tue, 2 Jul 2019, at 4:34 PM, Stephan Hoyer wrote:
>>>>
>>>> This is addressed in the NEP, see bullet 1 under "Partial
>>>> implementation of NumPy's API":
>>>>
>>>> http://www.numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api
>>>>
>>>> My concern is that fallback coercion behavior makes it difficult to
>>>> reliably implement "strict" overrides of NumPy's API. Fallback coercion is
>>>> certainly useful for interactive use, but it isn't really appropriate for
>>>> libraries.
>>>>
>>>>
>>> Do you mean "fallback coercion in NumPy itself", or "at all"? Right now
>>> there's lots of valid code around, e.g. `np.median(some_dask_array)`. Users
>>> will keep wanting to do that. Forcing everyone to write
>>> `np.median(np.array(some_dask_array))` serves no purpose. So the coercion
>>> has to be somewhere. You're arguing that that's up to Dask et al I think?
>>>
>>
>> Yes, I'm arguing this is up to dask to maintain backwards compatibility
>> -- or not, as the maintainers see fit.
>>
>> NumPy adding dispatching with __array_function__ did not break any
>> existing code, until the maintainers of other libraries started adding
>> __array_function__ methods. I hope that the risks of implementing such
>> experimental methods were self-evident.
>>
>
> Yeah, that's a bit of a chicken-and-egg story though. We add something and
> try to be "strict". Dask adds something because they like the idea and
> generally are quick to adopt these types of things. If we make it too hard
> to be backwards compatible, then neither NumPy nor Dask may try and it ends
> up breaking scikit-image & co. I for one don't care where the fix lands,
> but it's pretty to me that breaking scikit-image is the worst of all
> options.
>
>
>>
>>> Putting it in Dask right now still doesn't address Juan's backwards
>>> compat concern, but perhaps that could be bridged with a Dask bugfix
>>> release and some short-lived pain.
>>>
>>
>> I really think this is the best (only?) path forward.
>>
>
> I think I agree (depending on how easy it is to get the Dask fix landed).
>

That's landed, and Dask is planning a bugfix release in 2 days, so before
the NumPy 1.17.0 release. So this is not a release blocker anymore for us I
think.

Cheers,
Ralf


>> I'm not convinced that this shouldn't be fixed in NumPy though. Your
>>> concern "reliably implement "strict" overrides of NumPy's API" is a bit
>>> abstract. Overriding the _whole_ NumPy API is definitely undesirable. If
>>> we'd have a reference list somewhere about every function that is handled
>>> with __array_function__, then would that address your concern? Such a list
>>> could be auto-generated fairly easily.
>>>
>>
>> By "reliably implement strict overrides" I mean the ability to ensure
>> that every operation either uses an override or raises an informative error
>> -- making it very clear which operation needs to be implemented or avoided.
>>
>
> That isn't necessarily a good goal in itself though. In many cases, an
> `asarray` call still needs to go *somewhere*. If the "reliably implement
> strict overrides" is to help library authors, then there may be other ways
> to do that. For end users it can only hurt; those TypeErrors aren't exactly
> easy to understand.
>
>
>> It's true that we didn't really consider "always issuing warnings" as a
>> long term solution in the NEP. I can see how this would simply a backwards
>> compatibility story for libraries like dask, but in general, I really don't
>> like warnings:
>>
>
> I agree.
>
> Cheers,
> Ralf
>
> Using them like exceptions can easily result in code that is partially
>> broken or that fails later for non-obvious reasons. There's a reason why
>> Python's errors stop execution flow, until errors in languages like PHP or
>> JavaScript.
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190705/167c1a4b/attachment.html>