[Numpy-discussion] Keep __array_function__ unexposed by default for 1.17?

[Hameer Abbasi]

On Thu, 2019-05-23 at 10:19 -0700, Stephan Hoyer wrote:
> On Thu, May 23, 2019 at 2:43 AM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> > On Thu, May 23, 2019 at 3:02 AM Marten van Kerkwijk <
> > m.h.vankerkwijk at gmail.com> wrote:
> > > > If we want to keep an "off" switch we might want to add some
> > > > sort of API for exposing whether NumPy is using
> > > > __array_function__ or not. Maybe
> > > > numpy.__experimental_array_function_enabled__ = True, so you
> > > > can just test `hasattr(numpy,
> > > > '__experimental_array_function_enabled__')`? This is assuming
> > > > that we are OK with adding an underscore attribute to NumPy's
> > > > namespace semi-indefinitely.
> > 
> > I don't think we want to add or document anything publicly. That
> > only adds to the configuration problem, and indeed makes it harder
> > to rely on the issue. All I was suggested was keeping some
> > (private) safety switch in the code base for a while in case of
> > real issues as a workaround. 
> > 
> 
> I was concerned that libraries dask might have different behavior
> internally depending upon whether or not __array_function__ is
> enabled, but looking more carefully dask only does this detection for
> tests. So maybe this is not needed.
> 
> Still, I'm concerned about the potential broader implications of
> making it possibly to turn this off. In general, I don't think NumPy
> should have configurable global state -- it opens up the possibility
> of a whole class of issues. Stefan van der Walt raised this point
> when this "off switch" was suggested a few months ago:
> https://mail.python.org/pipermail/numpy-discussion/2019-March/079207.html

I agree -- Global mutable state is bad in general, but keeping around
the environment variable is okay.
> That said, I'd be OK with keeping around an environment variable as
> an emergency opt-out for now, especially to support benchmarking the
> impact of __array_function__ checks.

+1 for keeping the env var for now.
> But I would definitely be opposed to keeping around this switch
> around long term, for more than a major version or two. If there will
> be an outcry when we remove checks for
> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION, then we should reconsider the
> entire __array_function__ approach.
> 
> > > Might this be overthinking it? I might use this myself on
> > > supercomputer runs were I know that I'm using arrays only. Though
> > > one should not extrapolate from oneself!
> > > 
> > > That said, it is not difficult as is. For instance, we could
> > > explain in the docs that one can tell from:
> > > ```
> > > enabled = hasattr(np.core, 'overrides') and
> > > np.core.overrides.ENABLE_ARRAY_FUNCTION
> > > ```
> > > One could even allow for eventual removal by explaining it should
> > > be,
> > > ``` 
> > > enabled = hasattr(np.core, 'overrides') and
> > > getattr(np.core.overrides, 'ENABLE_ARRAY_FUNCTION', True)
> > > ```
> > > (If I understand correctly, one cannot tell from the presence of
> > > `ndarray.__array_function__`, correct?)
> > 
> > I think a hasattr check for __array_function__ is right.
> 
> We define ndarray.__array_function__ (even on NumPy 1.16) regardless
> of whether __array_function__ is enabled or not.
> 
> In principle we could have checked the environment variable from C
> before defining the method, but it's too late for that now.

I disagree here: In principle the only people relying on this would be
the same ones relying on the functionality of this protocol, so this
would be an easy change to undo, if at all needed. I do not know of any
libraries that actually use/call the __array_function__ attribute other
than NumPy, when it isn't enabled.
> _______________________________________________NumPy-Discussion
> mailing listNumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190524/640bac98/attachment.html>