<div><div dir="auto">Perhaps I missed this but I didn’t see: what happens when both __array_ufunc__ and __array_function__ are defined? I might want to do this to for example add support for functions like concatenate or stack to a class that already has an __array_ufunc__ defines.</div><br><div class="gmail_quote"><div>On Sat, Jun 2, 2018 at 5:56 PM Stephan Hoyer <<a href="mailto:shoyer@gmail.com">shoyer@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Matthew Rocklin and I have written NEP-18, which proposes a new dispatch mechanism for NumPy's high level API: <a href="http://www.numpy.org/neps/nep-0018-array-function-protocol.html" target="_blank">http://www.numpy.org/neps/nep-0018-array-function-protocol.html</a><div><div><br></div><div>There has already been a little bit of scattered discussion on the pull request (<a href="https://github.com/numpy/numpy/pull/11189" target="_blank">https://github.com/numpy/numpy/pull/11189</a>), but per NEP-0 let's try to keep high-level discussion here on the mailing list.</div></div><div><br></div><div>The full text of the NEP is reproduced below:</div><br><div><div>==================================================</div><div>NEP: Dispatch Mechanism for NumPy's high level API</div><div>==================================================</div><div><br></div><div>:Author: Stephan Hoyer <<a href="mailto:shoyer@google.com" target="_blank">shoyer@google.com</a>></div><div>:Author: Matthew Rocklin <<a href="mailto:mrocklin@gmail.com" target="_blank">mrocklin@gmail.com</a>></div><div>:Status: Draft</div><div>:Type: Standards Track</div><div>:Created: 2018-05-29</div><div><br></div><div>Abstact</div><div>-------</div><div><br></div><div>We propose a protocol to allow arguments of numpy functions to define</div><div>how that function operates on them. This allows other libraries that</div><div>implement NumPy's high level API to reuse Numpy functions. This allows</div><div>libraries that extend NumPy's high level API to apply to more NumPy-like</div><div>libraries.</div><div><br></div><div>Detailed description</div><div>--------------------</div><div><br></div><div>Numpy's high level ndarray API has been implemented several times</div><div>outside of NumPy itself for different architectures, such as for GPU</div><div>arrays (CuPy), Sparse arrays (scipy.sparse, pydata/sparse) and parallel</div><div>arrays (Dask array) as well as various Numpy-like implementations in the</div><div>deep learning frameworks, like TensorFlow and PyTorch.</div><div><br></div><div>Similarly there are several projects that build on top of the Numpy API</div><div>for labeled and indexed arrays (XArray), automatic differentation</div><div>(Autograd, Tangent), higher order array factorizations (TensorLy), etc.</div><div>that add additional functionality on top of the Numpy API.</div><div><br></div><div>We would like to be able to use these libraries together, for example we</div><div>would like to be able to place a CuPy array within XArray, or perform</div><div>automatic differentiation on Dask array code. This would be easier to</div><div>accomplish if code written for NumPy ndarrays could also be used by</div><div>other NumPy-like projects.</div><div><br></div><div>For example, we would like for the following code example to work</div><div>equally well with any Numpy-like array object:</div><div><br></div><div>.. code:: python</div><div><br></div><div> def f(x):</div><div> y = np.tensordot(x, x.T)</div><div> return np.mean(np.exp(y))</div><div><br></div><div>Some of this is possible today with various protocol mechanisms within</div><div>Numpy.</div><div><br></div><div>- The ``np.exp`` function checks the ``__array_ufunc__`` protocol</div><div>- The ``.T`` method works using Python's method dispatch</div><div>- The ``np.mean`` function explicitly checks for a ``.mean`` method on</div><div> the argument</div><div><br></div><div>However other functions, like ``np.tensordot`` do not dispatch, and</div><div>instead are likely to coerce to a Numpy array (using the ``__array__``)</div><div>protocol, or err outright. To achieve enough coverage of the NumPy API</div><div>to support downstream projects like XArray and autograd we want to</div><div>support *almost all* functions within Numpy, which calls for a more</div><div>reaching protocol than just ``__array_ufunc__``. We would like a</div><div>protocol that allows arguments of a NumPy function to take control and</div><div>divert execution to another function (for example a GPU or parallel</div><div>implementation) in a way that is safe and consistent across projects.</div><div><br></div><div>Implementation</div><div>--------------</div><div><br></div><div>We propose adding support for a new protocol in NumPy,</div><div>``__array_function__``.</div><div><br></div><div>This protocol is intended to be a catch-all for NumPy functionality that</div><div>is not covered by existing protocols, like reductions (like ``np.sum``)</div><div>or universal functions (like ``np.exp``). The semantics are very similar</div><div>to ``__array_ufunc__``, except the operation is specified by an</div><div>arbitrary callable object rather than a ufunc instance and method.</div><div><br></div><div>The interface</div><div>~~~~~~~~~~~~~</div><div><br></div><div>We propose the following signature for implementations of</div><div>``__array_function__``:</div><div><br></div><div>.. code-block:: python</div><div><br></div><div> def __array_function__(self, func, types, args, kwargs)</div><div><br></div><div>- ``func`` is an arbitrary callable exposed by NumPy's public API,</div><div> which was called in the form ``func(*args, **kwargs)``.</div><div>- ``types`` is a list of types for all arguments to the original NumPy</div><div> function call that will be checked for an ``__array_function__``</div><div> implementation.</div><div>- The tuple ``args`` and dict ``**kwargs`` are directly passed on from the</div><div> original call.</div><div><br></div><div>Unlike ``__array_ufunc__``, there are no high-level guarantees about the</div><div>type of ``func``, or about which of ``args`` and ``kwargs`` may contain objects</div><div>implementing the array API. As a convenience for ``__array_function__``</div><div>implementors of the NumPy API, the ``types`` keyword contains a list of all</div><div>types that implement the ``__array_function__`` protocol. This allows</div><div>downstream implementations to quickly determine if they are likely able to</div><div>support the operation.</div><div><br></div><div>Still be determined: what guarantees can we offer for ``types``? Should</div><div>we promise that types are unique, and appear in the order in which they</div><div>are checked?</div><div><br></div><div>Example for a project implementing the NumPy API</div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>Most implementations of ``__array_function__`` will start with two</div><div>checks:</div><div><br></div><div>1. Is the given function something that we know how to overload?</div><div>2. Are all arguments of a type that we know how to handle?</div><div><br></div><div>If these conditions hold, ``__array_function__`` should return</div><div>the result from calling its implementation for ``func(*args, **kwargs)``.</div><div>Otherwise, it should return the sentinel value ``NotImplemented``, indicating</div><div>that the function is not implemented by these types.</div><div><br></div><div>.. code:: python</div><div><br></div><div> class MyArray:</div><div> def __array_function__(self, func, types, args, kwargs):</div><div> if func not in HANDLED_FUNCTIONS:</div><div> return NotImplemented</div><div> if not all(issubclass(t, MyArray) for t in types):</div><div> return NotImplemented</div><div> return HANDLED_FUNCTIONS[func](*args, **kwargs)</div><div><br></div><div> HANDLED_FUNCTIONS = {</div><div> np.concatenate: my_concatenate,</div><div> np.broadcast_to: my_broadcast_to,</div><div> np.sum: my_sum,</div><div> ...</div><div> }</div><div><br></div><div>Necessary changes within the Numpy codebase itself</div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>This will require two changes within the Numpy codebase:</div><div><br></div><div>1. A function to inspect available inputs, look for the</div><div> ``__array_function__`` attribute on those inputs, and call those</div><div> methods appropriately until one succeeds. This needs to be fast in the</div><div> common all-NumPy case.</div><div><br></div><div> This is one additional function of moderate complexity.</div><div>2. Calling this function within all relevant Numpy functions.</div><div><br></div><div> This affects many parts of the Numpy codebase, although with very low</div><div> complexity.</div><div><br></div><div>Finding and calling the right ``__array_function__``</div><div>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</div><div><br></div><div>Given a Numpy function, ``*args`` and ``**kwargs`` inputs, we need to</div><div>search through ``*args`` and ``**kwargs`` for all appropriate inputs</div><div>that might have the ``__array_function__`` attribute. Then we need to</div><div>select among those possible methods and execute the right one.</div><div>Negotiating between several possible implementations can be complex.</div><div><br></div><div>Finding arguments</div><div>'''''''''''''''''</div><div><br></div><div>Valid arguments may be directly in the ``*args`` and ``**kwargs``, such</div><div>as in the case for ``np.tensordot(left, right, out=out)``, or they may</div><div>be nested within lists or dictionaries, such as in the case of</div><div>``np.concatenate([x, y, z])``. This can be problematic for two reasons:</div><div><br></div><div>1. Some functions are given long lists of values, and traversing them</div><div> might be prohibitively expensive</div><div>2. Some function may have arguments that we don't want to inspect, even</div><div> if they have the ``__array_function__`` method</div><div><br></div><div>To resolve these we ask the functions to provide an explicit list of</div><div>arguments that should be traversed. This is the ``relevant_arguments=``</div><div>keyword in the examples below.</div><div><br></div><div>Trying ``__array_function__`` methods until the right one works</div><div>'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''</div><div><br></div><div>Many arguments may implement the ``__array_function__`` protocol. Some</div><div>of these may decide that, given the available inputs, they are unable to</div><div>determine the correct result. How do we call the right one? If several</div><div>are valid then which has precedence?</div><div><br></div><div>The rules for dispatch with ``__array_function__`` match those for</div><div>``__array_ufunc__`` (see</div><div>`NEP-13 <<a href="http://www.numpy.org/neps/nep-0013-ufunc-overrides.html" target="_blank">http://www.numpy.org/neps/nep-0013-ufunc-overrides.html</a>>`_).</div><div>In particular:</div><div><br></div><div>- NumPy will gather implementations of ``__array_function__`` from all</div><div> specified inputs and call them in order: subclasses before</div><div> superclasses, and otherwise left to right. Note that in some edge cases,</div><div> this differs slightly from the</div><div> `current behavior <<a href="https://bugs.python.org/issue30140" target="_blank">https://bugs.python.org/issue30140</a>>`_ of Python.</div><div>- Implementations of ``__array_function__`` indicate that they can</div><div> handle the operation by returning any value other than</div><div> ``NotImplemented``.</div><div>- If all ``__array_function__`` methods return ``NotImplemented``,</div><div> NumPy will raise ``TypeError``.</div><div><br></div><div>Changes within Numpy functions</div><div>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</div><div><br></div><div>Given a function defined above, for now call it</div><div>``do_array_function_dance``, we now need to call that function from</div><div>within every relevant Numpy function. This is a pervasive change, but of</div><div>fairly simple and innocuous code that should complete quickly and</div><div>without effect if no arguments implement the ``__array_function__``</div><div>protocol. Let us consider a few examples of NumPy functions and how they</div><div>might be affected by this change:</div><div><br></div><div>.. code:: python</div><div><br></div><div> def broadcast_to(array, shape, subok=False):</div><div> success, value = do_array_function_dance(</div><div> func=broadcast_to,</div><div> relevant_arguments=[array],</div><div> args=(array,),</div><div> kwargs=dict(shape=shape, subok=subok))</div><div> if success:</div><div> return value</div><div><br></div><div> ... # continue with the definition of broadcast_to</div><div><br></div><div> def concatenate(arrays, axis=0, out=None)</div><div> success, value = do_array_function_dance(</div><div> func=concatenate,</div><div> relevant_arguments=[arrays, out],</div><div> args=(arrays,),</div><div> kwargs=dict(axis=axis, out=out))</div><div> if success:</div><div> return value</div><div><br></div><div> ... # continue with the definition of concatenate</div><div><br></div><div>The list of objects passed to ``relevant_arguments`` are those that should</div><div>be inspected for ``__array_function__`` implementations.</div><div><br></div><div>Alternatively, we could write these overloads with a decorator, e.g.,</div><div><br></div><div>.. code:: python</div><div><br></div><div> @overload_for_array_function(['array'])</div><div> def broadcast_to(array, shape, subok=False):</div><div> ... # continue with the definition of broadcast_to</div><div><br></div><div> @overload_for_array_function(['arrays', 'out'])</div><div> def concatenate(arrays, axis=0, out=None):</div><div> ... # continue with the definition of concatenate</div><div><br></div><div>The decorator ``overload_for_array_function`` would be written in terms</div><div>of ``do_array_function_dance``.</div><div><br></div><div>The downside of this approach would be a loss of introspection capability</div><div>for NumPy functions on Python 2, since this requires the use of</div><div>``inspect.Signature`` (only available on Python 3). However, NumPy won't</div><div>be supporting Python 2 for `very much longer <<a href="http://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html" target="_blank">http://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html</a>>`_.</div><div><br></div><div>Use outside of NumPy</div><div>~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>Nothing about this protocol that is particular to NumPy itself. Should</div><div>we enourage use of the same ``__array_function__`` protocol third-party</div><div>libraries for overloading non-NumPy functions, e.g., for making</div><div>array-implementation generic functionality in SciPy?</div><div><br></div><div>This would offer significant advantages (SciPy wouldn't need to invent</div><div>its own dispatch system) and no downsides that we can think of, because</div><div>every function that dispatches with ``__array_function__`` already needs</div><div>to be explicitly recognized. Libraries like Dask, CuPy, and Autograd</div><div>already wrap a limited subset of SciPy functionality (e.g.,</div><div>``scipy.linalg``) similarly to how they wrap NumPy.</div><div><br></div><div>If we want to do this, we should consider exposing the helper function</div><div>``do_array_function_dance()`` above as a public API.</div><div><br></div><div>Non-goals</div><div>---------</div><div><br></div><div>We are aiming for basic strategy that can be relatively mechanistically</div><div>applied to almost all functions in NumPy's API in a relatively short</div><div>period of time, the development cycle of a single NumPy release.</div><div><br></div><div>We hope to get both the ``__array_function__`` protocol and all specific</div><div>overloads right on the first try, but our explicit aim here is to get</div><div>something that mostly works (and can be iterated upon), rather than to</div><div>wait for an optimal implementation. The price of moving fast is that for</div><div>now **this protocol should be considered strictly experimental**. We</div><div>reserve the right to change the details of this protocol and how</div><div>specific NumPy functions use it at any time in the future -- even in</div><div>otherwise bug-fix only releases of NumPy.</div><div><br></div><div>In particular, we don't plan to write additional NEPs that list all</div><div>specific functions to overload, with exactly how they should be</div><div>overloaded. We will leave this up to the discretion of committers on</div><div>individual pull requests, trusting that they will surface any</div><div>controversies for discussion by interested parties.</div><div><br></div><div>However, we already know several families of functions that should be</div><div>explicitly exclude from ``__array_function__``. These will need their</div><div>own protocols:</div><div><br></div><div>- universal functions, which already have their own protocol.</div><div>- ``array`` and ``asarray``, because they are explicitly intended for</div><div> coercion to actual ``numpy.ndarray`` object.</div><div>- dispatch for methods of any kind, e.g., methods on</div><div> ``np.random.RandomState`` objects.</div><div><br></div><div>As a concrete example of how we expect to break behavior in the future,</div><div>some functions such as ``np.where`` are currently not NumPy universal</div><div>functions, but conceivably could become universal functions in the</div><div>future. When/if this happens, we will change such overloads from using</div><div>``__array_function__`` to the more specialized ``__array_ufunc__``.</div><div><br></div><div><br></div><div>Backward compatibility</div><div>----------------------</div><div><br></div><div>This proposal does not change existing semantics, except for those arguments</div><div>that currently have ``__array_function__`` methods, which should be rare.</div><div><br></div><div><br></div><div>Alternatives</div><div>------------</div><div><br></div><div>Specialized protocols</div><div>~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>We could (and should) continue to develop protocols like</div><div>``__array_ufunc__`` for cohesive subsets of Numpy functionality.</div><div><br></div><div>As mentioned above, if this means that some functions that we overload</div><div>with ``__array_function__`` should switch to a new protocol instead,</div><div>that is explicitly OK for as long as ``__array_function__`` retains its</div><div>experimental status.</div><div><br></div><div>Separate namespace</div><div>~~~~~~~~~~~~~~~~~~</div><div><br></div><div>A separate namespace for overloaded functions is another possibility,</div><div>either inside or outside of NumPy.</div><div><br></div><div>This has the advantage of alleviating any possible concerns about</div><div>backwards compatibility and would provide the maximum freedom for quick</div><div>experimentation. In the long term, it would provide a clean abstration</div><div>layer, separating NumPy's high level API from default implementations on</div><div>``numpy.ndarray`` objects.</div><div><br></div><div>The downsides are that this would require an explicit opt-in from all</div><div>existing code, e.g., ``import numpy.api as np``, and in the long term</div><div>would result in the maintainence of two separate NumPy APIs. Also, many</div><div>functions from ``numpy`` itself are already overloaded (but</div><div>inadequately), so confusion about high vs. low level APIs in NumPy would</div><div>still persist.</div><div><br></div><div>Multiple dispatch</div><div>~~~~~~~~~~~~~~~~~</div><div><br></div><div>An alternative to our suggestion of the ``__array_function__`` protocol</div><div>would be implementing NumPy's core functions as</div><div>`multi-methods <<a href="https://en.wikipedia.org/wiki/Multiple_dispatch" target="_blank">https://en.wikipedia.org/wiki/Multiple_dispatch</a>>`_.</div><div>Although one of us wrote a `multiple dispatch</div><div>library <<a href="https://github.com/mrocklin/multipledispatch" target="_blank">https://github.com/mrocklin/multipledispatch</a>>`_ for Python, we</div><div>don't think this approach makes sense for NumPy in the near term.</div><div><br></div><div>The main reason is that NumPy already has a well-proven dispatching</div><div>mechanism with ``__array_ufunc__``, based on Python's own dispatching</div><div>system for arithemtic, and it would be confusing to add another</div><div>mechanism that works in a very different way. This would also be more</div><div>invasive change to NumPy itself, which would need to gain a multiple</div><div>dispatch implementation.</div><div><br></div><div>It is possible that multiple dispatch implementation for NumPy's high</div><div>level API could make sense in the future. Fortunately,</div><div>``__array_function__`` does not preclude this possibility, because it</div><div>would be straightforward to write a shim for a default</div><div>``__array_function__`` implementation in terms of multiple dispatch.</div><div><br></div><div>Implementations in terms of a limited core API</div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>The internal implemenations of some NumPy functions is extremely simple.</div><div>For example: - ``np.stack()`` is implemented in only a few lines of code</div><div>by combining indexing with ``np.newaxis``, ``np.concatenate`` and the</div><div>``shape`` attribute. - ``np.mean()`` is implemented internally in terms</div><div>of ``np.sum()``, ``np.divide()``, ``.astype()`` and ``.shape``.</div><div><br></div><div>This suggests the possibility of defining a minimal "core" ndarray</div><div>interface, and relying upon it internally in NumPy to implement the full</div><div>API. This is an attractive option, because it could significantly reduce</div><div>the work required for new array implementations.</div><div><br></div><div>However, this also comes with several downsides: 1. The details of how</div><div>NumPy implements a high-level function in terms of overloaded functions</div><div>now becomes an implicit part of NumPy's public API. For example,</div><div>refactoring ``stack`` to use ``np.block()`` instead of</div><div>``np.concatenate()`` internally would now become a breaking change. 2.</div><div>Array libraries may prefer to implement high level functions differently</div><div>than NumPy. For example, a library might prefer to implement a</div><div>fundamental operations like ``mean()`` directly rather than relying on</div><div>``sum()`` followed by division. More generally, it's not clear yet what</div><div>exactly qualifies as core functionality, and figuring this out could be</div><div>a large project. 3. We don't yet have an overloading system for</div><div>attributes and methods on array objects, e.g., for accessing ``.dtype``</div><div>and ``.shape``. This should be the subject of a future NEP, but until</div><div>then we should be reluctant to rely on these properties.</div><div><br></div><div>Given these concerns, we encourage relying on this approach only in</div><div>limited cases.</div><div><br></div><div>Coersion to a NumPy array as a catch-all fallback</div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>With the current design, classes that implement ``__array_function__``</div><div>to overload at least one function implicitly declare an intent to</div><div>implement the entire NumPy API. It's not possible to implement *only*</div><div>``np.concatenate()`` on a type, but fall back to NumPy's default</div><div>behavior of casting with ``np.asarray()`` for all other functions.</div><div><br></div><div>This could present a backwards compatibility concern that would</div><div>discourage libraries from adopting ``__array_function__`` in an</div><div>incremental fashion. For example, currently most numpy functions will</div><div>implicitly convert ``pandas.Series`` objects into NumPy arrays, behavior</div><div>that assuredly many pandas users rely on. If pandas implemented</div><div>``__array_function__`` only for ``np.concatenate``, unrelated NumPy</div><div>functions like ``np.nanmean`` would suddenly break on pandas objects by</div><div>raising TypeError.</div><div><br></div><div>With ``__array_ufunc__``, it's possible to alleviate this concern by</div><div>casting all arguments to numpy arrays and re-calling the ufunc, but the</div><div>heterogeneous function signatures supported by ``__array_function__``</div><div>make it impossible to implement this generic fallback behavior for</div><div>``__array_function__``.</div><div><br></div><div>We could resolve this issue by change the handling of return values in</div><div>``__array_function__`` in either of two possible ways: 1. Change the</div><div>meaning of all arguments returning ``NotImplemented`` to indicate that</div><div>all arguments should be coerced to NumPy arrays instead. However, many</div><div>array libraries (e.g., scipy.sparse) really don't want implicit</div><div>conversions to NumPy arrays, and often avoid implementing ``__array__``</div><div>for exactly this reason. Implicit conversions can result in silent bugs</div><div>and performance degradation. 2. Use another sentinel value of some sort</div><div>to indicate that a class implementing part of the higher level array API</div><div>is coercible as a fallback, e.g., a return value of</div><div>``np.NotImplementedButCoercible`` from ``__array_function__``.</div><div><br></div><div>If we take this second approach, we would need to define additional</div><div>rules for how coercible array arguments are coerced, e.g., - Would we</div><div>try for ``__array_function__`` overloads again after coercing coercible</div><div>arguments? - If so, would we coerce coercible arguments one-at-a-time,</div><div>or all-at-once?</div><div><br></div><div>These are slightly tricky design questions, so for now we propose to</div><div>defer this issue. We can always implement</div><div>``np.NotImplementedButCoercible`` at some later time if it proves</div><div>critical to the numpy community in the future. Importantly, we don't</div><div>think this will stop critical libraries that desire to implement most of</div><div>the high level NumPy API from adopting this proposal.</div><div><br></div><div>NOTE: If you are reading this NEP in its draft state and disagree,</div><div>please speak up on the mailing list!</div><div><br></div><div>Drawbacks of this approach</div><div>--------------------------</div><div><br></div><div>Future difficulty extending NumPy's API</div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>One downside of passing on all arguments directly on to</div><div>``__array_function__`` is that it makes it hard to extend the signatures</div><div>of overloaded NumPy functions with new arguments, because adding even an</div><div>optional keyword argument would break existing overloads.</div><div><br></div><div>This is not a new problem for NumPy. NumPy has occasionally changed the</div><div>signature for functions in the past, including functions like</div><div>``numpy.sum`` which support overloads.</div><div><br></div><div>For adding new keyword arguments that do not change default behavior, we</div><div>would only include these as keyword arguments when they have changed</div><div>from default values. This is similar to `what NumPy already has</div><div>done <<a href="https://github.com/numpy/numpy/blob/v1.14.2/numpy/core/fromnumeric.py#L1865-L1867" target="_blank">https://github.com/numpy/numpy/blob/v1.14.2/numpy/core/fromnumeric.py#L1865-L1867</a>>`_,</div><div>e.g., for the optional ``keepdims`` argument in ``sum``:</div><div><br></div><div>.. code:: python</div><div><br></div><div> def sum(array, ..., keepdims=np._NoValue):</div><div> kwargs = {}</div><div> if keepdims is not np._NoValue:</div><div> kwargs['keepdims'] = keepdims</div><div> return array.sum(..., **kwargs)</div><div><br></div><div>In other cases, such as deprecated arguments, preserving the existing</div><div>behavior of overloaded functions may not be possible. Libraries that use</div><div>``__array_function__`` should be aware of this risk: we don't propose to</div><div>freeze NumPy's API in stone any more than it already is.</div><div><br></div><div>Difficulty adding implementation specific arguments</div><div>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>Some array implementations generally follow NumPy's API, but have</div><div>additional optional keyword arguments (e.g., ``dask.array.sum()`` has</div><div>``split_every`` and ``tensorflow.reduce_sum()`` has ``name``). A generic</div><div>dispatching library could potentially pass on all unrecognized keyword</div><div>argument directly to the implementation, but extending ``np.sum()`` to</div><div>pass on ``**kwargs`` would entail public facing changes in NumPy.</div><div>Customizing the detailed behavior of array libraries will require using</div><div>library specific functions, which could be limiting in the case of</div><div>libraries that consume the NumPy API such as xarray.</div><div><br></div><div><br></div><div>Discussion</div><div>----------</div><div><br></div><div>Various alternatives to this proposal were discussed in a few Github issues:</div><div><br></div><div>1. `pydata/sparse #1 <<a href="https://github.com/pydata/sparse/issues/1" target="_blank">https://github.com/pydata/sparse/issues/1</a>>`_</div><div>2. `numpy/numpy #11129 <<a href="https://github.com/numpy/numpy/issues/11129" target="_blank">https://github.com/numpy/numpy/issues/11129</a>>`_</div><div><br></div><div>Additionally it was the subject of `a blogpost</div><div><<a href="http://matthewrocklin.com/blog/work/2018/05/27/beyond-numpy" target="_blank">http://matthewrocklin.com/blog/work/2018/05/27/beyond-numpy</a>>`_ Following this</div><div>it was discussed at a `NumPy developer sprint</div><div><<a href="https://scisprints.github.io/#may-numpy-developer-sprint" target="_blank">https://scisprints.github.io/#may-numpy-developer-sprint</a>>`_ at the `UC</div><div>Berkeley Institute for Data Science (BIDS) <<a href="https://bids.berkeley.edu/" target="_blank">https://bids.berkeley.edu/</a>>`_.</div><div><br></div><div><br></div><div>References and Footnotes</div><div>------------------------</div><div><br></div><div>.. [1] Each NEP must either be explicitly labeled as placed in the public domain (see</div><div> this NEP as an example) or licensed under the `Open Publication License`_.</div><div><br></div><div>.. _Open Publication License: <a href="http://www.opencontent.org/openpub/" target="_blank">http://www.opencontent.org/openpub/</a></div><div><br></div><div><br></div><div>Copyright</div><div>---------</div><div><br></div><div>This document has been placed in the public domain. [1]_</div></div></div>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote></div></div>