> On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain,

My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings  attached to it.   In that case astropy might just wait before adopting it until that label was dropped.  Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely.

This would give the ecosystem an opportunity to try something and modify it after having some experience without promising to support it in a fully backwards compatible way forever.

On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
HI All,

On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13.

One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.)

All the best,

Marten

On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <mrocklin@gmail.com> wrote:
>> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them?

I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis.  Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts).  Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out.

This is different from how Numpy has operated in the past, but that might be ok.

>> 2. is it plausible that we'll find ourselves in that situation?

> Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime

Yup.  Same for Dask.  We're pretty accustomed to this.

On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <einstein.edison@gmail.com> wrote:
> On 29. Aug 2018, at 11:44, Matti Picus <matti.picus@gmail.com> wrote:
>
> On 29/08/18 10:37, Nathaniel Smith wrote:
>> it's easy to imagine scenarios where the
>> people being broken aren't the ones who had a chance to read the docs
>> – e.g. if a major package starts relying on __array_function__, then
>> it's all*their*  users who we'd be breaking, even though they had
>> nothing to do with it.
> This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages.

One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion.

Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.

>
> Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP?
> Matti
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion