[Numpy-discussion] Keep __array_function__ unexposed by default for 1.17?

[Hameer Abbasi]

On Wed, 2019-05-22 at 08:52 -0700, Stephan Hoyer wrote:
> Thanks for raising these concerns.
> The full implications of my recent __skip_array_function__ proposal
> are only now becoming evident to me now, looking at it's use in GH-
> 13585. Guaranteeing that it does not expand NumPy's API surface seems
> hard to achieve without pervasive use of __skip_array_function__
> internally.
> 
> Taking a step back, the sort of minor hacks [1] that motivated
> __skip_array_function__ for me are annoying, but really not too bad
> -- they are a small amount of additional code duplication in a
> proposal that already requires a large amount of code duplication.
> 
> So let's roll back the recent NEP change adding
> __skip_array_function__ to the public interface [2]. Inside the few
> NumPy functions where __array_function__ causes a measurable
> performance impact due to repeated calls (most notably np.block, for
> which some benchmarks are 25% slower), we can make use of the private
> __wrapped__ attribute.
> 
> I would still like to turn on __array_function__ in NumPy 1.17. At
> least, let's try that for the release candidate and see how it goes.
> The "all in" nature of __array_function__ without
> __skip_array_function__ will both limit its use to cases where it is
> strongly motivated, and also limits the API implications for NumPy.
> There is still plenty of room for expanding the protocol, but it's
> really hard to see what is necessary (and prudent!) without actual
> use.

Agreed that we should turn it on for 1.17  RC, and see if there are any
complaints.
> [1] e.g., see 
> https://github.com/google/jax/blob/62473351643cecb6c248a50601af163646ba7be6/jax/numpy/lax_numpy.py#L2440-L2459
> [2] https://github.com/numpy/numpy/pull/13305
> 
> 
> 
> 
> 
> On Tue, May 21, 2019 at 11:44 PM Juan Nunez-Iglesias <
> jni.soma at gmail.com> wrote:
> > I just want to express my general support for Marten's concerns. As
> > an "interested observer", I've been meaning to give
> > `__array_function__` a try but haven't had the chance yet. So from
> > my anecdotal experience I expect that more people need to play with
> > this before setting the API in stone.
> > 
> > At scikit-image we place a very strong emphasis on code simplicity
> > and readability, so I also share Marten's concerns about code
> > getting too complex. My impression reading the NEP was "whoa, this
> > is hard, I'm glad smarter people than me are working on this, I'm
> > sure it'll get simpler in time". But I haven't seen the simplicity
> > materialise...
> > 
> > On Wed, 22 May 2019, at 11:31 AM, Marten van Kerkwijk wrote:
> > > Hi All,
> > > 
> > > For 1.17, there has been a big effort, especially by Stephan, to
> > > make __array_function__ sufficiently usable that it can be
> > > exposed. I think this is great, and still like the idea very
> > > much, but its impact on the numpy code base has gotten so big in
> > > the most recent PR (gh-13585) that I wonder if we shouldn't
> > > reconsider the approach, and at least for 1.17 stick with the
> > > status quo. Since that seems to be a bigger question than can be
> > > usefully addressed in the PR, I thought I would raise it here.
> > > 
> > > Specifically, now not only does every numpy function have its
> > > dispatcher function, but also internally all numpy function calls
> > > are being done via the new `__skip_array_function__` attribute,
> > > to avoid further overrides. I think both changes make the code
> > > significantly less readable, thus, e.g., making it even harder
> > > than it is already to attract new contributors.
> > > 
> > > I think with this it is probably time to step back and check
> > > whether the implementation is in fact the right one. For
> > > instance, among the alternatives we originally considered was one
> > > that had the overridable versions of functions in the regular
> > > `numpy` namespace, and the once that would not themselves check
> > > in a different one. Alternatively, for some of the benefits
> > > provided by `__skip_array_function__`, there was a different
> > > suggestion to have a special return value, of
> > > `NotImplementedButCoercible`. Might these be better after all?
> > > 
> > > More generally, I think we're suffering from the fact that
> > > several of us seem to have rather different final goals in mind 
> > > In particular, I'd like to move to a state where as much of the
> > > code as possible makes use of the simplest possible
> > > implementation, with only a few true base functions, so that all
> > > but those simplest functions will generally work on any type of
> > > array. Others, however, worry much more about making
> > > implementations (even more) part of the API.
> > > 
> > > All the best,
> > > 
> > > Marten
> > > 
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at python.org
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > > 
> > 
> > _______________________________________________
> > 
> > NumPy-Discussion mailing list
> > 
> > NumPy-Discussion at python.org
> > 
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> > 
> 
> _______________________________________________NumPy-Discussion
> mailing listNumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190524/d919f5cb/attachment-0001.html>