NEP: Dispatch Mechanism for NumPy’s high level API
Matthew Rocklin and I have written NEP-18, which proposes a new dispatch mechanism for NumPy's high level API: http://www.numpy.org/neps/nep-0018-array-function-protocol.html There has already been a little bit of scattered discussion on the pull request (https://github.com/numpy/numpy/pull/11189), but per NEP-0 let's try to keep high-level discussion here on the mailing list. The full text of the NEP is reproduced below: ================================================== NEP: Dispatch Mechanism for NumPy's high level API ================================================== :Author: Stephan Hoyer <shoyer@google.com> :Author: Matthew Rocklin <mrocklin@gmail.com> :Status: Draft :Type: Standards Track :Created: 2018-05-29 Abstact ------- We propose a protocol to allow arguments of numpy functions to define how that function operates on them. This allows other libraries that implement NumPy's high level API to reuse Numpy functions. This allows libraries that extend NumPy's high level API to apply to more NumPy-like libraries. Detailed description -------------------- Numpy's high level ndarray API has been implemented several times outside of NumPy itself for different architectures, such as for GPU arrays (CuPy), Sparse arrays (scipy.sparse, pydata/sparse) and parallel arrays (Dask array) as well as various Numpy-like implementations in the deep learning frameworks, like TensorFlow and PyTorch. Similarly there are several projects that build on top of the Numpy API for labeled and indexed arrays (XArray), automatic differentation (Autograd, Tangent), higher order array factorizations (TensorLy), etc. that add additional functionality on top of the Numpy API. We would like to be able to use these libraries together, for example we would like to be able to place a CuPy array within XArray, or perform automatic differentiation on Dask array code. This would be easier to accomplish if code written for NumPy ndarrays could also be used by other NumPy-like projects. For example, we would like for the following code example to work equally well with any Numpy-like array object: .. code:: python def f(x): y = np.tensordot(x, x.T) return np.mean(np.exp(y)) Some of this is possible today with various protocol mechanisms within Numpy. - The ``np.exp`` function checks the ``__array_ufunc__`` protocol - The ``.T`` method works using Python's method dispatch - The ``np.mean`` function explicitly checks for a ``.mean`` method on the argument However other functions, like ``np.tensordot`` do not dispatch, and instead are likely to coerce to a Numpy array (using the ``__array__``) protocol, or err outright. To achieve enough coverage of the NumPy API to support downstream projects like XArray and autograd we want to support *almost all* functions within Numpy, which calls for a more reaching protocol than just ``__array_ufunc__``. We would like a protocol that allows arguments of a NumPy function to take control and divert execution to another function (for example a GPU or parallel implementation) in a way that is safe and consistent across projects. Implementation -------------- We propose adding support for a new protocol in NumPy, ``__array_function__``. This protocol is intended to be a catch-all for NumPy functionality that is not covered by existing protocols, like reductions (like ``np.sum``) or universal functions (like ``np.exp``). The semantics are very similar to ``__array_ufunc__``, except the operation is specified by an arbitrary callable object rather than a ufunc instance and method. The interface ~~~~~~~~~~~~~ We propose the following signature for implementations of ``__array_function__``: .. code-block:: python def __array_function__(self, func, types, args, kwargs) - ``func`` is an arbitrary callable exposed by NumPy's public API, which was called in the form ``func(*args, **kwargs)``. - ``types`` is a list of types for all arguments to the original NumPy function call that will be checked for an ``__array_function__`` implementation. - The tuple ``args`` and dict ``**kwargs`` are directly passed on from the original call. Unlike ``__array_ufunc__``, there are no high-level guarantees about the type of ``func``, or about which of ``args`` and ``kwargs`` may contain objects implementing the array API. As a convenience for ``__array_function__`` implementors of the NumPy API, the ``types`` keyword contains a list of all types that implement the ``__array_function__`` protocol. This allows downstream implementations to quickly determine if they are likely able to support the operation. Still be determined: what guarantees can we offer for ``types``? Should we promise that types are unique, and appear in the order in which they are checked? Example for a project implementing the NumPy API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Most implementations of ``__array_function__`` will start with two checks: 1. Is the given function something that we know how to overload? 2. Are all arguments of a type that we know how to handle? If these conditions hold, ``__array_function__`` should return the result from calling its implementation for ``func(*args, **kwargs)``. Otherwise, it should return the sentinel value ``NotImplemented``, indicating that the function is not implemented by these types. .. code:: python class MyArray: def __array_function__(self, func, types, args, kwargs): if func not in HANDLED_FUNCTIONS: return NotImplemented if not all(issubclass(t, MyArray) for t in types): return NotImplemented return HANDLED_FUNCTIONS[func](*args, **kwargs) HANDLED_FUNCTIONS = { np.concatenate: my_concatenate, np.broadcast_to: my_broadcast_to, np.sum: my_sum, ... } Necessary changes within the Numpy codebase itself ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This will require two changes within the Numpy codebase: 1. A function to inspect available inputs, look for the ``__array_function__`` attribute on those inputs, and call those methods appropriately until one succeeds. This needs to be fast in the common all-NumPy case. This is one additional function of moderate complexity. 2. Calling this function within all relevant Numpy functions. This affects many parts of the Numpy codebase, although with very low complexity. Finding and calling the right ``__array_function__`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Given a Numpy function, ``*args`` and ``**kwargs`` inputs, we need to search through ``*args`` and ``**kwargs`` for all appropriate inputs that might have the ``__array_function__`` attribute. Then we need to select among those possible methods and execute the right one. Negotiating between several possible implementations can be complex. Finding arguments ''''''''''''''''' Valid arguments may be directly in the ``*args`` and ``**kwargs``, such as in the case for ``np.tensordot(left, right, out=out)``, or they may be nested within lists or dictionaries, such as in the case of ``np.concatenate([x, y, z])``. This can be problematic for two reasons: 1. Some functions are given long lists of values, and traversing them might be prohibitively expensive 2. Some function may have arguments that we don't want to inspect, even if they have the ``__array_function__`` method To resolve these we ask the functions to provide an explicit list of arguments that should be traversed. This is the ``relevant_arguments=`` keyword in the examples below. Trying ``__array_function__`` methods until the right one works ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Many arguments may implement the ``__array_function__`` protocol. Some of these may decide that, given the available inputs, they are unable to determine the correct result. How do we call the right one? If several are valid then which has precedence? The rules for dispatch with ``__array_function__`` match those for ``__array_ufunc__`` (see `NEP-13 <http://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_). In particular: - NumPy will gather implementations of ``__array_function__`` from all specified inputs and call them in order: subclasses before superclasses, and otherwise left to right. Note that in some edge cases, this differs slightly from the `current behavior <https://bugs.python.org/issue30140>`_ of Python. - Implementations of ``__array_function__`` indicate that they can handle the operation by returning any value other than ``NotImplemented``. - If all ``__array_function__`` methods return ``NotImplemented``, NumPy will raise ``TypeError``. Changes within Numpy functions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Given a function defined above, for now call it ``do_array_function_dance``, we now need to call that function from within every relevant Numpy function. This is a pervasive change, but of fairly simple and innocuous code that should complete quickly and without effect if no arguments implement the ``__array_function__`` protocol. Let us consider a few examples of NumPy functions and how they might be affected by this change: .. code:: python def broadcast_to(array, shape, subok=False): success, value = do_array_function_dance( func=broadcast_to, relevant_arguments=[array], args=(array,), kwargs=dict(shape=shape, subok=subok)) if success: return value ... # continue with the definition of broadcast_to def concatenate(arrays, axis=0, out=None) success, value = do_array_function_dance( func=concatenate, relevant_arguments=[arrays, out], args=(arrays,), kwargs=dict(axis=axis, out=out)) if success: return value ... # continue with the definition of concatenate The list of objects passed to ``relevant_arguments`` are those that should be inspected for ``__array_function__`` implementations. Alternatively, we could write these overloads with a decorator, e.g., .. code:: python @overload_for_array_function(['array']) def broadcast_to(array, shape, subok=False): ... # continue with the definition of broadcast_to @overload_for_array_function(['arrays', 'out']) def concatenate(arrays, axis=0, out=None): ... # continue with the definition of concatenate The decorator ``overload_for_array_function`` would be written in terms of ``do_array_function_dance``. The downside of this approach would be a loss of introspection capability for NumPy functions on Python 2, since this requires the use of ``inspect.Signature`` (only available on Python 3). However, NumPy won't be supporting Python 2 for `very much longer < http://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html>`_. Use outside of NumPy ~~~~~~~~~~~~~~~~~~~~ Nothing about this protocol that is particular to NumPy itself. Should we enourage use of the same ``__array_function__`` protocol third-party libraries for overloading non-NumPy functions, e.g., for making array-implementation generic functionality in SciPy? This would offer significant advantages (SciPy wouldn't need to invent its own dispatch system) and no downsides that we can think of, because every function that dispatches with ``__array_function__`` already needs to be explicitly recognized. Libraries like Dask, CuPy, and Autograd already wrap a limited subset of SciPy functionality (e.g., ``scipy.linalg``) similarly to how they wrap NumPy. If we want to do this, we should consider exposing the helper function ``do_array_function_dance()`` above as a public API. Non-goals --------- We are aiming for basic strategy that can be relatively mechanistically applied to almost all functions in NumPy's API in a relatively short period of time, the development cycle of a single NumPy release. We hope to get both the ``__array_function__`` protocol and all specific overloads right on the first try, but our explicit aim here is to get something that mostly works (and can be iterated upon), rather than to wait for an optimal implementation. The price of moving fast is that for now **this protocol should be considered strictly experimental**. We reserve the right to change the details of this protocol and how specific NumPy functions use it at any time in the future -- even in otherwise bug-fix only releases of NumPy. In particular, we don't plan to write additional NEPs that list all specific functions to overload, with exactly how they should be overloaded. We will leave this up to the discretion of committers on individual pull requests, trusting that they will surface any controversies for discussion by interested parties. However, we already know several families of functions that should be explicitly exclude from ``__array_function__``. These will need their own protocols: - universal functions, which already have their own protocol. - ``array`` and ``asarray``, because they are explicitly intended for coercion to actual ``numpy.ndarray`` object. - dispatch for methods of any kind, e.g., methods on ``np.random.RandomState`` objects. As a concrete example of how we expect to break behavior in the future, some functions such as ``np.where`` are currently not NumPy universal functions, but conceivably could become universal functions in the future. When/if this happens, we will change such overloads from using ``__array_function__`` to the more specialized ``__array_ufunc__``. Backward compatibility ---------------------- This proposal does not change existing semantics, except for those arguments that currently have ``__array_function__`` methods, which should be rare. Alternatives ------------ Specialized protocols ~~~~~~~~~~~~~~~~~~~~~ We could (and should) continue to develop protocols like ``__array_ufunc__`` for cohesive subsets of Numpy functionality. As mentioned above, if this means that some functions that we overload with ``__array_function__`` should switch to a new protocol instead, that is explicitly OK for as long as ``__array_function__`` retains its experimental status. Separate namespace ~~~~~~~~~~~~~~~~~~ A separate namespace for overloaded functions is another possibility, either inside or outside of NumPy. This has the advantage of alleviating any possible concerns about backwards compatibility and would provide the maximum freedom for quick experimentation. In the long term, it would provide a clean abstration layer, separating NumPy's high level API from default implementations on ``numpy.ndarray`` objects. The downsides are that this would require an explicit opt-in from all existing code, e.g., ``import numpy.api as np``, and in the long term would result in the maintainence of two separate NumPy APIs. Also, many functions from ``numpy`` itself are already overloaded (but inadequately), so confusion about high vs. low level APIs in NumPy would still persist. Multiple dispatch ~~~~~~~~~~~~~~~~~ An alternative to our suggestion of the ``__array_function__`` protocol would be implementing NumPy's core functions as `multi-methods <https://en.wikipedia.org/wiki/Multiple_dispatch>`_. Although one of us wrote a `multiple dispatch library <https://github.com/mrocklin/multipledispatch>`_ for Python, we don't think this approach makes sense for NumPy in the near term. The main reason is that NumPy already has a well-proven dispatching mechanism with ``__array_ufunc__``, based on Python's own dispatching system for arithemtic, and it would be confusing to add another mechanism that works in a very different way. This would also be more invasive change to NumPy itself, which would need to gain a multiple dispatch implementation. It is possible that multiple dispatch implementation for NumPy's high level API could make sense in the future. Fortunately, ``__array_function__`` does not preclude this possibility, because it would be straightforward to write a shim for a default ``__array_function__`` implementation in terms of multiple dispatch. Implementations in terms of a limited core API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The internal implemenations of some NumPy functions is extremely simple. For example: - ``np.stack()`` is implemented in only a few lines of code by combining indexing with ``np.newaxis``, ``np.concatenate`` and the ``shape`` attribute. - ``np.mean()`` is implemented internally in terms of ``np.sum()``, ``np.divide()``, ``.astype()`` and ``.shape``. This suggests the possibility of defining a minimal "core" ndarray interface, and relying upon it internally in NumPy to implement the full API. This is an attractive option, because it could significantly reduce the work required for new array implementations. However, this also comes with several downsides: 1. The details of how NumPy implements a high-level function in terms of overloaded functions now becomes an implicit part of NumPy's public API. For example, refactoring ``stack`` to use ``np.block()`` instead of ``np.concatenate()`` internally would now become a breaking change. 2. Array libraries may prefer to implement high level functions differently than NumPy. For example, a library might prefer to implement a fundamental operations like ``mean()`` directly rather than relying on ``sum()`` followed by division. More generally, it's not clear yet what exactly qualifies as core functionality, and figuring this out could be a large project. 3. We don't yet have an overloading system for attributes and methods on array objects, e.g., for accessing ``.dtype`` and ``.shape``. This should be the subject of a future NEP, but until then we should be reluctant to rely on these properties. Given these concerns, we encourage relying on this approach only in limited cases. Coersion to a NumPy array as a catch-all fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With the current design, classes that implement ``__array_function__`` to overload at least one function implicitly declare an intent to implement the entire NumPy API. It's not possible to implement *only* ``np.concatenate()`` on a type, but fall back to NumPy's default behavior of casting with ``np.asarray()`` for all other functions. This could present a backwards compatibility concern that would discourage libraries from adopting ``__array_function__`` in an incremental fashion. For example, currently most numpy functions will implicitly convert ``pandas.Series`` objects into NumPy arrays, behavior that assuredly many pandas users rely on. If pandas implemented ``__array_function__`` only for ``np.concatenate``, unrelated NumPy functions like ``np.nanmean`` would suddenly break on pandas objects by raising TypeError. With ``__array_ufunc__``, it's possible to alleviate this concern by casting all arguments to numpy arrays and re-calling the ufunc, but the heterogeneous function signatures supported by ``__array_function__`` make it impossible to implement this generic fallback behavior for ``__array_function__``. We could resolve this issue by change the handling of return values in ``__array_function__`` in either of two possible ways: 1. Change the meaning of all arguments returning ``NotImplemented`` to indicate that all arguments should be coerced to NumPy arrays instead. However, many array libraries (e.g., scipy.sparse) really don't want implicit conversions to NumPy arrays, and often avoid implementing ``__array__`` for exactly this reason. Implicit conversions can result in silent bugs and performance degradation. 2. Use another sentinel value of some sort to indicate that a class implementing part of the higher level array API is coercible as a fallback, e.g., a return value of ``np.NotImplementedButCoercible`` from ``__array_function__``. If we take this second approach, we would need to define additional rules for how coercible array arguments are coerced, e.g., - Would we try for ``__array_function__`` overloads again after coercing coercible arguments? - If so, would we coerce coercible arguments one-at-a-time, or all-at-once? These are slightly tricky design questions, so for now we propose to defer this issue. We can always implement ``np.NotImplementedButCoercible`` at some later time if it proves critical to the numpy community in the future. Importantly, we don't think this will stop critical libraries that desire to implement most of the high level NumPy API from adopting this proposal. NOTE: If you are reading this NEP in its draft state and disagree, please speak up on the mailing list! Drawbacks of this approach -------------------------- Future difficulty extending NumPy's API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ One downside of passing on all arguments directly on to ``__array_function__`` is that it makes it hard to extend the signatures of overloaded NumPy functions with new arguments, because adding even an optional keyword argument would break existing overloads. This is not a new problem for NumPy. NumPy has occasionally changed the signature for functions in the past, including functions like ``numpy.sum`` which support overloads. For adding new keyword arguments that do not change default behavior, we would only include these as keyword arguments when they have changed from default values. This is similar to `what NumPy already has done < https://github.com/numpy/numpy/blob/v1.14.2/numpy/core/fromnumeric.py#L1865-...
`_, e.g., for the optional ``keepdims`` argument in ``sum``:
.. code:: python def sum(array, ..., keepdims=np._NoValue): kwargs = {} if keepdims is not np._NoValue: kwargs['keepdims'] = keepdims return array.sum(..., **kwargs) In other cases, such as deprecated arguments, preserving the existing behavior of overloaded functions may not be possible. Libraries that use ``__array_function__`` should be aware of this risk: we don't propose to freeze NumPy's API in stone any more than it already is. Difficulty adding implementation specific arguments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some array implementations generally follow NumPy's API, but have additional optional keyword arguments (e.g., ``dask.array.sum()`` has ``split_every`` and ``tensorflow.reduce_sum()`` has ``name``). A generic dispatching library could potentially pass on all unrecognized keyword argument directly to the implementation, but extending ``np.sum()`` to pass on ``**kwargs`` would entail public facing changes in NumPy. Customizing the detailed behavior of array libraries will require using library specific functions, which could be limiting in the case of libraries that consume the NumPy API such as xarray. Discussion ---------- Various alternatives to this proposal were discussed in a few Github issues: 1. `pydata/sparse #1 <https://github.com/pydata/sparse/issues/1>`_ 2. `numpy/numpy #11129 <https://github.com/numpy/numpy/issues/11129>`_ Additionally it was the subject of `a blogpost <http://matthewrocklin.com/blog/work/2018/05/27/beyond-numpy>`_ Following this it was discussed at a `NumPy developer sprint <https://scisprints.github.io/#may-numpy-developer-sprint>`_ at the `UC Berkeley Institute for Data Science (BIDS) <https://bids.berkeley.edu/>`_. References and Footnotes ------------------------ .. [1] Each NEP must either be explicitly labeled as placed in the public domain (see this NEP as an example) or licensed under the `Open Publication License`_. .. _Open Publication License: http://www.opencontent.org/openpub/ Copyright --------- This document has been placed in the public domain. [1]_
Perhaps I missed this but I didn’t see: what happens when both __array_ufunc__ and __array_function__ are defined? I might want to do this to for example add support for functions like concatenate or stack to a class that already has an __array_ufunc__ defines. On Sat, Jun 2, 2018 at 5:56 PM Stephan Hoyer <shoyer@gmail.com> wrote:
Perhaps I missed this but I didn’t see: what happens when both __array_ufunc__ and __array_function__ are defined? I might want to do this to for example add support for functions like concatenate or stack to a class that already has an __array_ufunc__ defines. This is mentioned in the section “Non-goals”, which says that ufuncs and their methods should be excluded, along with a few other classes of functions/methods. Sent from Astro <https://www.helloastro.com> for Mac
Hi Stephan, Thanks for posting. Overall, this is great! My more general comment is one of speed: for *normal* operation performance should be impacted as minimally as possible. I think this is a serious issue and feel strongly it *has* to be possible to avoid all arguments being checked for the `__array_function__` attribute, i.e., there should be an obvious way to ensure no type checking dance is done. Some possible solutions (which I think should be in the NEP, even if as discounted options): A. Two "namespaces", one for the undecorated base functions, and one completely trivial one for the decorated ones. The idea would be that if one knows one is dealing with arrays only, one would do `import numpy.array_only as np` (i.e., the reverse of the suggestion currently in the NEP, where the decorated ones are in their own namespace - I agree with the reasons for discounting that one). Note that in this suggestion the array-only namespace serves as the one used for `ndarray.__array_function__`. B. Automatic insertion by the decorator of an `array_only=np._NoValue` (or `coerce` and perhaps `subok=...` if not present) in the function signature, so that users who know that they have arrays only could pass `array_only=True` (name to be decided). This would be most useful if there were also some type of configuration parameter that could set the default of `array_only`. Note that both A and B could also address, at least partially, the problem of sometimes wanting to just use the old coercion methods, i.e., not having to implement every possible numpy function in one go in a new `__array_function__` on one's class. Two other general comments: 1. I'm rather unclear about the use of `types`. It can help me decide what to do, but I would still have to find the argument in question (e.g., for Quantity, the unit of the relevant argument). I'd recommend passing instead a tuple of all arguments that were inspected, in the inspection order; after all, it is just a `arg.__class__` away from the type, and in your example you'd only have to replace `issubclass` by `isinstance`. 2. For subclasses, it would be very handy to have `ndarray.__array_function__`, so one can call super after changing arguments. (For `__array_ufunc__`, there was lots of question about whether this was useful, but it really is!!). [I think you already agreed with this, but want to have it in-place, as for subclasses of ndarray this is just as useful as it would be for subclasses of dask arrays.) Note that any `ndarray.__array_function__` might also help solve the problem of cases where coercion is fine: it could have an extra keyword argument (say `coerce`) that would call the function with coercion in place. Indeed, if the `ndarray.__array_function__` were used inside the "dance" function, and then the actual implementation of a given function would just be a separate, private one. Again, overall a great idea, and thanks to all those involved for taking it on. All the best, Marten On Sat, Jun 2, 2018 at 6:55 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
On Sun, Jun 3, 2018 at 8:19 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I agree that all we should try minimize the impact of dispatching on normal operations. It would be helpful to identify examples of real workflows, so we can measure the impact of doing these checks empirically. That said, I think a small degradation in performance for code that works with small arrays should be acceptable, because performance is an already an accepted limitations of using NumPy/Python for these use cases. In most cases, I suspect that the overhead of a function call and checking several arguments for "__array_function__" will be negligible, like the situation for __array_ufunc__. I'm not strongly opposed to either of your proposed solutions, but I do think it would be a little strange to insist that we need a solution for __array_function__ when __array_ufunc__ was fine.
I will mention this as a possibility. I do think there is something to be said for clear separation of overloaded and non-overloaded APIs. But f I were to choose between adding numpy.api and numpy.array_only, I would pick numpy.api, because of the virtue of preserving the existing numpy namespace as it currently exists.
Rather than adding another argument to every NumPy function, I would rather encourage writing np.asarray() explicitly.
Yes, agreed.
The virtue of a `types` argument is that we can deduplicate arguments once, rather than in each __array_function__ check. This could result in significantly more efficient code, e.g,. when np.concatenate() is called on 10,000 arrays with only two unique types, we don't need to loop through all 10,000 again objects to check that overloading is valid. Even for Quantity, I suspect you will want two layers of checks: 1. A check to verify that every argument is a Quantity (or something coercible to a Quantity). This could use `types` and return `NotImplemented` when it fails. 2. A check to verify that units match. This will have custom logic for different operations and will require checking all arguments -- not just their unique types. For many Quantity functions, the second check will indeed probably be super simple (i.e., verifying that all units match). But the first check (with `types`) really is something that basically very overload should do.
Yes, indeed.
Ufuncs actually do try to speed-up array checks - but indeed the same can (and should) be done for `__array_ufunc__`. They also do have `subok`. This currently ignored but that is mostly because looking for it in `kwargs` is so damn slow! Anyway, my main point was that it should be explicitly mentioned as a constraint that for pure ndarray input, things should be really fast.
Good point. Overall, the separate namespaces probably is not the way to do.
Good point - just as good as long as the check for all-array is very fast (which it should be - `arg.__class__ is np.ndarray` is fast!).
I think one might still want to know *where* the type occurs (e.g., as an output or index would have different implications). Possibly, a solution would rely on the same structure as used for the "dance". But as a general point, I don't see the advantage of passing types rather than arguments - less information for no benefit.
Not sure. With, Quantity I generally do not worry about other types, but rather look at units attributes, assume anything without is dimensionless, cast Quantity to array with the right unit, and then defer to `ndarray`.
On Sun, Jun 3, 2018 at 4:25 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I think one might still want to know *where* the type occurs (e.g., as an output or index would have different implications).
This in certainly true in general, but given the complete flexibility of __array_function__ there's no way we can make every check convenient. The best we can do is make it easy to handle the common cases, where the argument position does not matter.
Maybe this is premature optimization, but there will certainly be fewer unique types than arguments to check for types. I suspect this may make for a noticeable difference in performance in use cases involving a large number of argument. For example, suppose np.concatenate() is called on a list of 10,000 dask arrays. Now dask.array.Array.__array_function__ needs to check all arguments to decide whether it can use dask.array.concatenate() or needs to return NotImplemented. By using the `types` argument, it only needs to do isinstance() checks on the single argument in `types`, rather than all 10,000 overloaded function arguments.
This in certainly true in general, but given the complete flexibility of __array_function__ there's no way we can make every check convenient. The best we can do is make it easy to handle the common cases, where the argument position does not matter. I think those cases may not be as common as you think - most functions are not like `concatenate` & co... Indeed, it might be good to add some other examples to the NEP. Looing at the list of functions which do not work with Quantity currently: Maybe `np.dot`, `np.choose`, and `np.vectorize`?
One also needs to worry about the cost of contructing `types`, though I guess this could be minimal if it is a `set`. Or should it be the keys of a `dict`, with the value something meaningful that has to be calculated anyway (like a list of sequence numbers); this may all depend a bit on the implementation of "dance" - the information it gathers might as well get passed on.
It is probably a good idea to add some of these considerations to the NEP. -- Marten
The rules for dispatch with ``__array_function__`` match those for ``__array_ufunc__`` (see `NEP-13 <http://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_). In particular: - NumPy will gather implementations of ``__array_function__`` from all specified inputs and call them in order: subclasses before superclasses, and otherwise left to right. Note that in some edge cases, this differs slightly from the `current behavior <https://bugs.python.org/issue30140>`_ of Python. - Implementations of ``__array_function__`` indicate that they can handle the operation by returning any value other than ``NotImplemented``. - If all ``__array_function__`` methods return ``NotImplemented``, NumPy will raise ``TypeError``. I’d like to propose two changes to this: - ``np.NotImplementedButCoercible`` be a part of the standard from the start. - If all implementations return this, only then should it be coerced. - In the future, it might be good to mark something as coercible to coerce it to ``ndarray`` before passing to another object’s ``__array_ufunc__``. - This is necessary if libraries want to keep old behaviour for some functions, while overriding others. - Otherwise they have to implement overloads for all functions. This seems rather like an all-or-nothing choice, which I’d like to avoid. - It isn’t too hard to implement in practice. - Objects that don’t implement ``__array_function__`` should be treated as having returned ``np.NotImplementedButCoercible``. - This has the effect of coercing ``list``, etc. - At a minimum, to maintain compatibility, if all objects don’t implement ``__array_function__``, the old behaviour should stay. Also, I’m +1 on Marten’s suggestion that ``ndarray`` itself should implement ``__array_function__``.
I also am not sure there is an actual problem: In the scheme as proposed, implementations could just coerce themselves to array and call the routine again. (Or, in the scheme I proposed, call the routine again but with `coerce=True`.) Ah, I didn’t think of the first solution. `coerce=True` may not produce the desired solution in cases where some arguments can be coerced and some can’t. However, such a design may still have some benefits. For example: - ``array1.HANDLED_TYPES = [array1]`` - ``array2.HANDLED_TYPES = [array1, array2]`` - ``array1`` is coercible. - None of these is a sub/super class of the other or of ``ndarray`` - When calling ``np.func(array1(), array2())``, ``array1`` would be coerced with your solution (because of the left-to-right rule and ``array1`` choosing to coerce itself) but not with ``np.NotImplementedButCoercible``. I think that in the proposed scheme this is effectively what happens. Not really, the current scheme is unclear on what happens if none of the arguments implement ``__array_function__`` (or at least it doesn’t explicitly state it that I can see).
On Sun, Jun 3, 2018 at 11:12 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
The current proposal is to copy the behavior of __array_ufunc__. So the non-existence of an __array_function__ attribute is indeed *not* equivalent to returning NotImplemented: if no arguments implement __array_function__, then yes they will all be coerced to NumPy arrays. I do think there is elegance in defining a return value of np.NotImplementedButCoercible as equivalent to the existence of __array_function__. This resolves my design question about how coercible arguments would be coerced with NotImplementedButCoercible: we would fall back to the current behavior, which in most cases means all arguments are coerced to NumPy arrays directly. Mixed return values of NotImplementedButCoercible and NotImplemented would still result in TypeError, and there would be no second chances for overloads. This is simple enough that I am inclined to update the NEP to incorporate the suggestion (thank you!). My main question is whether we should also update __array_ufunc__ to support returning NotImplementedButCoercible for consistency. My inclination is yes: even though it's easy to implement a fallback of converting all arguments to NumPy arrays for ufuncs, it is hard to do this correctly from an __array_ufunc__ implementation, because __array_ufunc__ implementations do not know in what order they have been called. The counter-argument would be that it's not worth adding new features to __array_ufunc__ if use-cases haven't come up yet. But my guess is that most users/implementors of __array_ufunc__ are ignorant of these finer details, and not really worrying about them. Also, the list of binary operators in Python is short enough that most implementations are OK with supporting either all or none. Actually, a return value of NotImplementedButCoercible would probably be the right answer for some cases in xarray's current __array_ufunc__ method, when we encounter ufunc methods for which we haven't written an implementation (e.g., "outer" or "at").
Although I'm still not 100% convinced by NotImplementedButCoercible, I do like the idea that this is the default for items that do not implement `__array_function__`. And it might help avoid trying to find oneself in a possibly long list. -- Marten
On Sun, Jun 3, 2018 at 5:44 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Another potential consideration in favor of NotImplementedButCoercible is for subclassing: we could use it to write the default implementations of ndarray.__array_ufunc__ and ndarray.__array_function__, e.g., class ndarray: def __array_ufunc__(self, *args, **kwargs): return NotIImplementedButCoercible def __array_function__(self, *args, **kwargs): return NotIImplementedButCoercible I think (not 100% sure yet) this would result in exactly equivalent behavior to what ndarray.__array_ufunc__ currently does: http://www.numpy.org/neps/nep-0013-ufunc-overrides.html#subclass-hierarchies
Hi Stephan, Another potential consideration in favor of NotImplementedButCoercible is
As written would not work for ndarray subclasses, because the subclass will generically change itself before calling super. At least for Quantity, say if I add two quantities, the quantities will both be converted to arrays (with one scaled so that the units match) and then the super call is done with those modified arrays. This expects that the super call will actually return a result (which it now can because all inputs are arrays). But I think it would work to return `NotImplementedButCoercible` in the case that perhaps you had in mind in the first place, in which any of the *other* arguments had a `__array_ufunc__` implementation and `ndarray` thus does not know what to do. For those cases, `ndarray` currently returns a straight `NotImplemented`. Though I am still a bit worried: this gets back to `Quantity.__array_ufunc__`, but what does it do with it? It cannot just pass it on, since then it is effectively telling, incorrectly, that the *quantity* is coercible, which it is not. I guess at this point it would have to change it to `NotImplemented`. Looking at my current implementation, I see that if we made this change to `ndarray.__array_ufunc__`, the implementation would mostly raise an exception as it tried to view `NotImplementedButCoercible` as a quantity, except for comparisons, where the output is not viewed at all (being boolean and thus unit-less) and passed straight down. That said, we've said the __array_ufunc__ implementation is experimental, so I think such small annoyances are OK. Overall, it is an intriguing idea, and I think it should be mentioned at least in the NEP. It would be good, though, to have a few more examples of how it would work in practice. All the best, Marten
On Mon, Jun 4, 2018 at 7:35 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Thanks for clarifying. This is definitely trickier than I had thought. If Quantity.__array_ufunc__ implemented overrides by calling the public ufunc method again (instead of calling super), then it would still work fine with this change. But of course, in that case you would not need ndarray.__array_ufunc__ defined at all. I will say that personally, I find the complexity of the current ndarray.__array_ufunc__ implementation a little inelegant, and I would welcome simplifying it. But I also try to avoid implementation inheritance entirely [2], for exactly the same reasons why refactoring ndarray.__array_ufunc__ here would be difficult (inheritance is fragile). So I would be happy to defer to your judgment, as someone who actually uses subclassing. https://hackernoon.com/inheritance-based-on-internal-structure-is-evil-7474c...
Hi Stephan, Things would, I think, make much more sense if `ndarray.__array_ufunc__` (or `*_function__`) actually *were* the implementation for array-only. But while that is something I'd like to eventually get to, it seems out of scope for the current discussion. But we should be sure that the ndarray versions return either `NotImplemented` or a result. Given that, I think that perhaps it is also best not to do `NotImplementedButCoercible` - as I think the implementers of `__array_function__` perhaps should just do that themselves. But I may well swing the other way again... Good examples of non-trivial benefits would help. All the best, Marten
On Tue, Jun 5, 2018 at 12:35 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
If this is a desirable end-state, we should at least consider it now while we are designing the __array_function__ interface. With the current proposal, I think this would be nearly impossible. The challenge is that ndarray.__array_function__ would somehow need to call the non-overloaded version of the provided function provided that no other arguments overload __array_function__. However, currently don't expose this information in any way. Some ways this could be done (including some of your prior suggestions): - Add a coerce=True argument to all NumPy functions, which could be used by non-overloaded implementations. - A separate namespace for non-overloaded functions (e.g., numpy.array_only). - Adding another argument to the __array_function__ interface to explicitly provide the non-overloaded implementation (e.g., func_impl). I don't like any of these options and I'm not sure I agree with your goal, but the NEP should make clear that we are precluding this possibility. Given that, I think that perhaps it is also best not to do
This would also be my default stance, and of course we can always add NotImplementedButCoercible later. I can think of two main use cases: 1. Libraries that only want to overload *some* NumPy functions, but want the rest of NumPy's API by coercing arguments to NumPy arrays. 2. Library that want to eventually overload all of NumPy's high level API, but need to do so incrementally, in a way that preserves backwards compatibility. I'm not sure I agree with use case 1. Arguably, libraries that only overload a limited part of NumPy's API shouldn't encourage their users their users to rely on it. This state of affairs is pretty confusing to users. However, case 2 is valid and potentially important. Consider the case of a library with existing users that would like to start implementing __array_function__ (e.g., dask, astropy, xarray, pandas). The right strategy really depends upon whether the library considers the current behavior of NumPy functions on their objects (silent coercion to numpy arrays) a feature or a bug: - If coercion is a bug and something that the library never intended to support, then perhaps it would be OK to suddenly change all existing overloads to return the correct type. - However, if coercion is a feature (which is probably the attitude of at least some users), ideally there really should be a graceful way to enable the new overloaded behavior incrementally. For example, a library might want to start issuing FutureWarning in version X, before switching over to the new overloaded behavior in version X+1. I can't think of how to do this without NotImplementedButCoercible. For projects like dask and xarray, the benefits of __array_function__ are so large that we will accept a hard transition that breaks some user code without warning. But this may not be the case for other projects.
Hi Stephan, On `NotImplementedButCoercible`: don't forget that even a preliminary implementation of `__array_function__` has always the choice of coercing its own instances to ndarray and re-calling the function; that is really no different from (though probably a bit slower than) what would happen if one returned NIBC. It does require, however, a fairly efficient way of finding arguments of one's own class, which is partially why I think it is important for there to be a quick way to find instances of one's own type; we should try to avoid having people to reimplement the dance. It may still be that `types` is the right vehicle for this - it just depends on how much of the state of the dance it carries. On the "separate" name-space question: one thing it is not is particularly difficult, especially if one works with a decorator: effectively one already has the original function and the wrapped one; the only question is whether it would pay to keep the original one around somewhere. I do continue to think that we will get grumbling about regressions in speed and that it would help to have the undecorated versions available. Though in my ideal world those would do no coercing whatsoever, but just take arrays, i.e., they are actually faster than the current ones. All the best, Marten
On 05/06/18 14:11, Stephan Hoyer wrote:
What is the difference between the `func` provided as the first argument to `__array_function__` and `__array_ufunc__` and the "non-overloaded version of the provided function"? This NEP calls it an "arbitrary callable". In `__array_ufunc__` it turns out people count on it being exactly the `np.ufunc`. Matti
On Tue, Jun 5, 2018 at 2:47 PM Matti Picus <matti.picus@gmail.com> wrote:
The ""non-overloaded version of the provided function" is entirely hypothetical at this point. If we use a decorator to implement overloads, it would be the undecorated function, e.g., the original definition of concatenate here: @overload_for_array_function(['arrays', 'out'])def concatenate(arrays, axis=0, out=None): ... # continue with the definition of concatenate This NEP calls it an "arbitrary callable".
In `__array_ufunc__` it turns out people count on it being exactly the `np.ufunc`.
Right, I think this is good guarantee to provide. Certainly it's one that people fine useful.
Yes, the function should definitely be the same as what the user called - i.e., the decorated function. I'm only wondering if it would also be possible to have access to the undecorated one (via `coerce` or `ndarray.__array_function__` or otherwise). -- Marten
Hmm, does this mean the callable that gets passed into __array_ufunc__ will change? I'm pretty sure that will break the dispatch mechanism I'm using in my __array_ufunc__ implementation, which directly checks whether the callable is in one of several tuples of functions that have different behavior. On Tue, Jun 5, 2018 at 7:32 PM, Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Oh wait, since the decorated version of the ufunc will be the one in the public numpy API it won't break. It would only break if the callable that was passed in *wasn't* the decorated version, so it kinda *has* to pass in the decorated function to preserve backward compatibility. Apologies for the noise. On Tue, Jun 5, 2018 at 7:39 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On 6. Jun 2018 at 05:41, Nathan Goldbaum <nathan12343@gmail.com> wrote: Oh wait, since the decorated version of the ufunc will be the one in the public numpy API it won't break. It would only break if the callable that was passed in *wasn't* the decorated version, so it kinda *has* to pass in the decorated function to preserve backward compatibility. Apologies for the noise. On Tue, Jun 5, 2018 at 7:39 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Section “Non-Goals” states that Ufuncs will not be part of this protocol, __array_ufunc__ will be used to override those as usual. Sent from Astro <https://www.helloastro.com> for Mac
Hi Stephan, I think we're getting to the stage where an updated text would be useful. For that, you may want to consider an actual implementation of, e.g., a very simple function like `np.reshape` as well as a more complicated one like `np.concatenate`, and in particular how the implementation finds out where its own instances are located. All the best, Marten
On Fri, Jun 8, 2018 at 8:58 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I think we're getting to the stage where an updated text would be useful.
Yes, I plan to work on this over the weekend. Stay tuned!
Yes, I agree that actual implementation (in Python rather than C for now) would be useful.
and in particular how the implementation finds out where its own instances are located.
I think we've discussed this before, but I don't think this is feasible to solve in general given the diversity of wrapped APIs. If you want to find the arguments in which a class' own instances appear, you will need to do that in your overloaded function. That said, if merely pulling out the flat list of arguments that are checked for and/or implement __array_function__ would be enough, we can probably figure out a way to expose that information.
In the end, somewhere inside the "dance", you are checking for `__array_function` - it would seem to me that at that point you know exactly where you are, and it would not be difficult to something like ``` types[new_type] += [where_i_am] ``` (where here I assume types is a defaultdict(list)) - has the set of types in keys and locations as values. But easier to discuss whether this is easy with some sample code to look at! -- Marten
I meant whatever the state of the dance routine is, e.g., the way the arguments are enumerated by the decorator (this is partially why some example code for the dance routine is needed -- I am not 100% how this should work, just seems logical that if the dance routine can understand it, so can __array_function__ implementations). -- Marten
Mixed return values of NotImplementedButCoercible and NotImplemented would still result in TypeError, and there would be no second chances for overloads. I would like to differ with you here: It can be quite useful to have second chances for overloads. Think ``np.func(list, custom_array))``: If second rounds did not exist, custom_array would need to have a list of coercible types (which is not nice IMO). It can also help in cases where performance/feature degradation isn’t an issue, so coercing all arguments that returned ``NotImplementedButCoercible`` would allow ``__array_function__`` to succeed where it wouldn’t normally. I mean, that’s one of the major uses of this sentinel right? If done in a for loop, it wouldn’t even slow down the nominal cases. It would have the adverse effect of not allowing for a default implementation to be as simple as you stated, though. One thing we could do is manually (inside ``__array_function__``) coerce anything that didn’t implement ``__array_function__``, and that’s acceptable to me too.
On Sun, Jun 3, 2018 at 9:54 PM Hameer Abbasi <einstein.edison@gmail.com> wrote:
Even if we did this, we would still want to preserve the equivalence between: 1. Returning NotImplementedButCoercible from __array_ufunc__ or __array_function__, and 2. Not implementing __array_ufunc__ or __array_function__ at all. Changing __array_ufunc__ to do multiple rounds of checks could indeed be useful in some cases, and you're right that it would not change existing behavior (in these cases we currently raise TypeError). But I'd rather leave that for a separate discussion, because it's orthogonal to our proposal here for __array_function__. (Personally, I don't think it would be worth the additional complexity.)
Should there be discussion of typing (pep-484) or abstract base classes in this nep? Are there any requirements on the result returned by __array_function__? On Mon, Jun 4, 2018, 2:20 AM Stephan Hoyer <shoyer@gmail.com> wrote:
On Mon, Jun 4, 2018 at 5:39 AM Matthew Harrigan <harrigan.matthew@gmail.com> wrote:
This is a good question that should be addressed in the NEP. Currently, we impose no limitations on the types returned by __array_function__ (or __array_ufunc__, for that matter). Given the complexity of potential __array_function__ implementations, I think this would be hard/impossible to do in general. I think the best case scenario we could hope for is that type checkers would identify that result of NumPy functions as: - numpy.ndarray if all inputs are numpy.ndarray objects - Any if any non-numpy.ndarray inputs implement the __array_function__ Based on my understanding of proposed rules for typing protocols [1] and overloads [2], I think this could just work, e.g., @overload def func(array: np.ndarray) -> np.ndarray: ... @overload def func(array: ImplementsArrayFunction) -> Any: ... [1] https://www.python.org/dev/peps/pep-0544/ [2] https://github.com/python/typing/issues/253#issuecomment-389262904
I agree that second rounds of overloads have to be left to the implementers of `__array_function__` - obviously, though, we should be sure that these rounds are rarely necessary... The link posted by Stephan [1] has some decent discussion for `__array_ufunc__` about when an override should re-call the function rather than try to do something itself. -- Marten [1] http://www.numpy.org/neps/nep-0013-ufunc-overrides.html#subclass-hierarchies
Perhaps I missed this but I didn’t see: what happens when both __array_ufunc__ and __array_function__ are defined? I might want to do this to for example add support for functions like concatenate or stack to a class that already has an __array_ufunc__ defines. On Sat, Jun 2, 2018 at 5:56 PM Stephan Hoyer <shoyer@gmail.com> wrote:
Perhaps I missed this but I didn’t see: what happens when both __array_ufunc__ and __array_function__ are defined? I might want to do this to for example add support for functions like concatenate or stack to a class that already has an __array_ufunc__ defines. This is mentioned in the section “Non-goals”, which says that ufuncs and their methods should be excluded, along with a few other classes of functions/methods. Sent from Astro <https://www.helloastro.com> for Mac
Hi Stephan, Thanks for posting. Overall, this is great! My more general comment is one of speed: for *normal* operation performance should be impacted as minimally as possible. I think this is a serious issue and feel strongly it *has* to be possible to avoid all arguments being checked for the `__array_function__` attribute, i.e., there should be an obvious way to ensure no type checking dance is done. Some possible solutions (which I think should be in the NEP, even if as discounted options): A. Two "namespaces", one for the undecorated base functions, and one completely trivial one for the decorated ones. The idea would be that if one knows one is dealing with arrays only, one would do `import numpy.array_only as np` (i.e., the reverse of the suggestion currently in the NEP, where the decorated ones are in their own namespace - I agree with the reasons for discounting that one). Note that in this suggestion the array-only namespace serves as the one used for `ndarray.__array_function__`. B. Automatic insertion by the decorator of an `array_only=np._NoValue` (or `coerce` and perhaps `subok=...` if not present) in the function signature, so that users who know that they have arrays only could pass `array_only=True` (name to be decided). This would be most useful if there were also some type of configuration parameter that could set the default of `array_only`. Note that both A and B could also address, at least partially, the problem of sometimes wanting to just use the old coercion methods, i.e., not having to implement every possible numpy function in one go in a new `__array_function__` on one's class. Two other general comments: 1. I'm rather unclear about the use of `types`. It can help me decide what to do, but I would still have to find the argument in question (e.g., for Quantity, the unit of the relevant argument). I'd recommend passing instead a tuple of all arguments that were inspected, in the inspection order; after all, it is just a `arg.__class__` away from the type, and in your example you'd only have to replace `issubclass` by `isinstance`. 2. For subclasses, it would be very handy to have `ndarray.__array_function__`, so one can call super after changing arguments. (For `__array_ufunc__`, there was lots of question about whether this was useful, but it really is!!). [I think you already agreed with this, but want to have it in-place, as for subclasses of ndarray this is just as useful as it would be for subclasses of dask arrays.) Note that any `ndarray.__array_function__` might also help solve the problem of cases where coercion is fine: it could have an extra keyword argument (say `coerce`) that would call the function with coercion in place. Indeed, if the `ndarray.__array_function__` were used inside the "dance" function, and then the actual implementation of a given function would just be a separate, private one. Again, overall a great idea, and thanks to all those involved for taking it on. All the best, Marten On Sat, Jun 2, 2018 at 6:55 PM, Stephan Hoyer <shoyer@gmail.com> wrote:
On Sun, Jun 3, 2018 at 8:19 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I agree that all we should try minimize the impact of dispatching on normal operations. It would be helpful to identify examples of real workflows, so we can measure the impact of doing these checks empirically. That said, I think a small degradation in performance for code that works with small arrays should be acceptable, because performance is an already an accepted limitations of using NumPy/Python for these use cases. In most cases, I suspect that the overhead of a function call and checking several arguments for "__array_function__" will be negligible, like the situation for __array_ufunc__. I'm not strongly opposed to either of your proposed solutions, but I do think it would be a little strange to insist that we need a solution for __array_function__ when __array_ufunc__ was fine.
I will mention this as a possibility. I do think there is something to be said for clear separation of overloaded and non-overloaded APIs. But f I were to choose between adding numpy.api and numpy.array_only, I would pick numpy.api, because of the virtue of preserving the existing numpy namespace as it currently exists.
Rather than adding another argument to every NumPy function, I would rather encourage writing np.asarray() explicitly.
Yes, agreed.
The virtue of a `types` argument is that we can deduplicate arguments once, rather than in each __array_function__ check. This could result in significantly more efficient code, e.g,. when np.concatenate() is called on 10,000 arrays with only two unique types, we don't need to loop through all 10,000 again objects to check that overloading is valid. Even for Quantity, I suspect you will want two layers of checks: 1. A check to verify that every argument is a Quantity (or something coercible to a Quantity). This could use `types` and return `NotImplemented` when it fails. 2. A check to verify that units match. This will have custom logic for different operations and will require checking all arguments -- not just their unique types. For many Quantity functions, the second check will indeed probably be super simple (i.e., verifying that all units match). But the first check (with `types`) really is something that basically very overload should do.
Yes, indeed.
Ufuncs actually do try to speed-up array checks - but indeed the same can (and should) be done for `__array_ufunc__`. They also do have `subok`. This currently ignored but that is mostly because looking for it in `kwargs` is so damn slow! Anyway, my main point was that it should be explicitly mentioned as a constraint that for pure ndarray input, things should be really fast.
Good point. Overall, the separate namespaces probably is not the way to do.
Good point - just as good as long as the check for all-array is very fast (which it should be - `arg.__class__ is np.ndarray` is fast!).
I think one might still want to know *where* the type occurs (e.g., as an output or index would have different implications). Possibly, a solution would rely on the same structure as used for the "dance". But as a general point, I don't see the advantage of passing types rather than arguments - less information for no benefit.
Not sure. With, Quantity I generally do not worry about other types, but rather look at units attributes, assume anything without is dimensionless, cast Quantity to array with the right unit, and then defer to `ndarray`.
On Sun, Jun 3, 2018 at 4:25 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I think one might still want to know *where* the type occurs (e.g., as an output or index would have different implications).
This in certainly true in general, but given the complete flexibility of __array_function__ there's no way we can make every check convenient. The best we can do is make it easy to handle the common cases, where the argument position does not matter.
Maybe this is premature optimization, but there will certainly be fewer unique types than arguments to check for types. I suspect this may make for a noticeable difference in performance in use cases involving a large number of argument. For example, suppose np.concatenate() is called on a list of 10,000 dask arrays. Now dask.array.Array.__array_function__ needs to check all arguments to decide whether it can use dask.array.concatenate() or needs to return NotImplemented. By using the `types` argument, it only needs to do isinstance() checks on the single argument in `types`, rather than all 10,000 overloaded function arguments.
This in certainly true in general, but given the complete flexibility of __array_function__ there's no way we can make every check convenient. The best we can do is make it easy to handle the common cases, where the argument position does not matter. I think those cases may not be as common as you think - most functions are not like `concatenate` & co... Indeed, it might be good to add some other examples to the NEP. Looing at the list of functions which do not work with Quantity currently: Maybe `np.dot`, `np.choose`, and `np.vectorize`?
One also needs to worry about the cost of contructing `types`, though I guess this could be minimal if it is a `set`. Or should it be the keys of a `dict`, with the value something meaningful that has to be calculated anyway (like a list of sequence numbers); this may all depend a bit on the implementation of "dance" - the information it gathers might as well get passed on.
It is probably a good idea to add some of these considerations to the NEP. -- Marten
The rules for dispatch with ``__array_function__`` match those for ``__array_ufunc__`` (see `NEP-13 <http://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_). In particular: - NumPy will gather implementations of ``__array_function__`` from all specified inputs and call them in order: subclasses before superclasses, and otherwise left to right. Note that in some edge cases, this differs slightly from the `current behavior <https://bugs.python.org/issue30140>`_ of Python. - Implementations of ``__array_function__`` indicate that they can handle the operation by returning any value other than ``NotImplemented``. - If all ``__array_function__`` methods return ``NotImplemented``, NumPy will raise ``TypeError``. I’d like to propose two changes to this: - ``np.NotImplementedButCoercible`` be a part of the standard from the start. - If all implementations return this, only then should it be coerced. - In the future, it might be good to mark something as coercible to coerce it to ``ndarray`` before passing to another object’s ``__array_ufunc__``. - This is necessary if libraries want to keep old behaviour for some functions, while overriding others. - Otherwise they have to implement overloads for all functions. This seems rather like an all-or-nothing choice, which I’d like to avoid. - It isn’t too hard to implement in practice. - Objects that don’t implement ``__array_function__`` should be treated as having returned ``np.NotImplementedButCoercible``. - This has the effect of coercing ``list``, etc. - At a minimum, to maintain compatibility, if all objects don’t implement ``__array_function__``, the old behaviour should stay. Also, I’m +1 on Marten’s suggestion that ``ndarray`` itself should implement ``__array_function__``.
I also am not sure there is an actual problem: In the scheme as proposed, implementations could just coerce themselves to array and call the routine again. (Or, in the scheme I proposed, call the routine again but with `coerce=True`.) Ah, I didn’t think of the first solution. `coerce=True` may not produce the desired solution in cases where some arguments can be coerced and some can’t. However, such a design may still have some benefits. For example: - ``array1.HANDLED_TYPES = [array1]`` - ``array2.HANDLED_TYPES = [array1, array2]`` - ``array1`` is coercible. - None of these is a sub/super class of the other or of ``ndarray`` - When calling ``np.func(array1(), array2())``, ``array1`` would be coerced with your solution (because of the left-to-right rule and ``array1`` choosing to coerce itself) but not with ``np.NotImplementedButCoercible``. I think that in the proposed scheme this is effectively what happens. Not really, the current scheme is unclear on what happens if none of the arguments implement ``__array_function__`` (or at least it doesn’t explicitly state it that I can see).
On Sun, Jun 3, 2018 at 11:12 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
The current proposal is to copy the behavior of __array_ufunc__. So the non-existence of an __array_function__ attribute is indeed *not* equivalent to returning NotImplemented: if no arguments implement __array_function__, then yes they will all be coerced to NumPy arrays. I do think there is elegance in defining a return value of np.NotImplementedButCoercible as equivalent to the existence of __array_function__. This resolves my design question about how coercible arguments would be coerced with NotImplementedButCoercible: we would fall back to the current behavior, which in most cases means all arguments are coerced to NumPy arrays directly. Mixed return values of NotImplementedButCoercible and NotImplemented would still result in TypeError, and there would be no second chances for overloads. This is simple enough that I am inclined to update the NEP to incorporate the suggestion (thank you!). My main question is whether we should also update __array_ufunc__ to support returning NotImplementedButCoercible for consistency. My inclination is yes: even though it's easy to implement a fallback of converting all arguments to NumPy arrays for ufuncs, it is hard to do this correctly from an __array_ufunc__ implementation, because __array_ufunc__ implementations do not know in what order they have been called. The counter-argument would be that it's not worth adding new features to __array_ufunc__ if use-cases haven't come up yet. But my guess is that most users/implementors of __array_ufunc__ are ignorant of these finer details, and not really worrying about them. Also, the list of binary operators in Python is short enough that most implementations are OK with supporting either all or none. Actually, a return value of NotImplementedButCoercible would probably be the right answer for some cases in xarray's current __array_ufunc__ method, when we encounter ufunc methods for which we haven't written an implementation (e.g., "outer" or "at").
Although I'm still not 100% convinced by NotImplementedButCoercible, I do like the idea that this is the default for items that do not implement `__array_function__`. And it might help avoid trying to find oneself in a possibly long list. -- Marten
On Sun, Jun 3, 2018 at 5:44 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Another potential consideration in favor of NotImplementedButCoercible is for subclassing: we could use it to write the default implementations of ndarray.__array_ufunc__ and ndarray.__array_function__, e.g., class ndarray: def __array_ufunc__(self, *args, **kwargs): return NotIImplementedButCoercible def __array_function__(self, *args, **kwargs): return NotIImplementedButCoercible I think (not 100% sure yet) this would result in exactly equivalent behavior to what ndarray.__array_ufunc__ currently does: http://www.numpy.org/neps/nep-0013-ufunc-overrides.html#subclass-hierarchies
Hi Stephan, Another potential consideration in favor of NotImplementedButCoercible is
As written would not work for ndarray subclasses, because the subclass will generically change itself before calling super. At least for Quantity, say if I add two quantities, the quantities will both be converted to arrays (with one scaled so that the units match) and then the super call is done with those modified arrays. This expects that the super call will actually return a result (which it now can because all inputs are arrays). But I think it would work to return `NotImplementedButCoercible` in the case that perhaps you had in mind in the first place, in which any of the *other* arguments had a `__array_ufunc__` implementation and `ndarray` thus does not know what to do. For those cases, `ndarray` currently returns a straight `NotImplemented`. Though I am still a bit worried: this gets back to `Quantity.__array_ufunc__`, but what does it do with it? It cannot just pass it on, since then it is effectively telling, incorrectly, that the *quantity* is coercible, which it is not. I guess at this point it would have to change it to `NotImplemented`. Looking at my current implementation, I see that if we made this change to `ndarray.__array_ufunc__`, the implementation would mostly raise an exception as it tried to view `NotImplementedButCoercible` as a quantity, except for comparisons, where the output is not viewed at all (being boolean and thus unit-less) and passed straight down. That said, we've said the __array_ufunc__ implementation is experimental, so I think such small annoyances are OK. Overall, it is an intriguing idea, and I think it should be mentioned at least in the NEP. It would be good, though, to have a few more examples of how it would work in practice. All the best, Marten
On Mon, Jun 4, 2018 at 7:35 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Thanks for clarifying. This is definitely trickier than I had thought. If Quantity.__array_ufunc__ implemented overrides by calling the public ufunc method again (instead of calling super), then it would still work fine with this change. But of course, in that case you would not need ndarray.__array_ufunc__ defined at all. I will say that personally, I find the complexity of the current ndarray.__array_ufunc__ implementation a little inelegant, and I would welcome simplifying it. But I also try to avoid implementation inheritance entirely [2], for exactly the same reasons why refactoring ndarray.__array_ufunc__ here would be difficult (inheritance is fragile). So I would be happy to defer to your judgment, as someone who actually uses subclassing. https://hackernoon.com/inheritance-based-on-internal-structure-is-evil-7474c...
Hi Stephan, Things would, I think, make much more sense if `ndarray.__array_ufunc__` (or `*_function__`) actually *were* the implementation for array-only. But while that is something I'd like to eventually get to, it seems out of scope for the current discussion. But we should be sure that the ndarray versions return either `NotImplemented` or a result. Given that, I think that perhaps it is also best not to do `NotImplementedButCoercible` - as I think the implementers of `__array_function__` perhaps should just do that themselves. But I may well swing the other way again... Good examples of non-trivial benefits would help. All the best, Marten
On Tue, Jun 5, 2018 at 12:35 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
If this is a desirable end-state, we should at least consider it now while we are designing the __array_function__ interface. With the current proposal, I think this would be nearly impossible. The challenge is that ndarray.__array_function__ would somehow need to call the non-overloaded version of the provided function provided that no other arguments overload __array_function__. However, currently don't expose this information in any way. Some ways this could be done (including some of your prior suggestions): - Add a coerce=True argument to all NumPy functions, which could be used by non-overloaded implementations. - A separate namespace for non-overloaded functions (e.g., numpy.array_only). - Adding another argument to the __array_function__ interface to explicitly provide the non-overloaded implementation (e.g., func_impl). I don't like any of these options and I'm not sure I agree with your goal, but the NEP should make clear that we are precluding this possibility. Given that, I think that perhaps it is also best not to do
This would also be my default stance, and of course we can always add NotImplementedButCoercible later. I can think of two main use cases: 1. Libraries that only want to overload *some* NumPy functions, but want the rest of NumPy's API by coercing arguments to NumPy arrays. 2. Library that want to eventually overload all of NumPy's high level API, but need to do so incrementally, in a way that preserves backwards compatibility. I'm not sure I agree with use case 1. Arguably, libraries that only overload a limited part of NumPy's API shouldn't encourage their users their users to rely on it. This state of affairs is pretty confusing to users. However, case 2 is valid and potentially important. Consider the case of a library with existing users that would like to start implementing __array_function__ (e.g., dask, astropy, xarray, pandas). The right strategy really depends upon whether the library considers the current behavior of NumPy functions on their objects (silent coercion to numpy arrays) a feature or a bug: - If coercion is a bug and something that the library never intended to support, then perhaps it would be OK to suddenly change all existing overloads to return the correct type. - However, if coercion is a feature (which is probably the attitude of at least some users), ideally there really should be a graceful way to enable the new overloaded behavior incrementally. For example, a library might want to start issuing FutureWarning in version X, before switching over to the new overloaded behavior in version X+1. I can't think of how to do this without NotImplementedButCoercible. For projects like dask and xarray, the benefits of __array_function__ are so large that we will accept a hard transition that breaks some user code without warning. But this may not be the case for other projects.
Hi Stephan, On `NotImplementedButCoercible`: don't forget that even a preliminary implementation of `__array_function__` has always the choice of coercing its own instances to ndarray and re-calling the function; that is really no different from (though probably a bit slower than) what would happen if one returned NIBC. It does require, however, a fairly efficient way of finding arguments of one's own class, which is partially why I think it is important for there to be a quick way to find instances of one's own type; we should try to avoid having people to reimplement the dance. It may still be that `types` is the right vehicle for this - it just depends on how much of the state of the dance it carries. On the "separate" name-space question: one thing it is not is particularly difficult, especially if one works with a decorator: effectively one already has the original function and the wrapped one; the only question is whether it would pay to keep the original one around somewhere. I do continue to think that we will get grumbling about regressions in speed and that it would help to have the undecorated versions available. Though in my ideal world those would do no coercing whatsoever, but just take arrays, i.e., they are actually faster than the current ones. All the best, Marten
On 05/06/18 14:11, Stephan Hoyer wrote:
What is the difference between the `func` provided as the first argument to `__array_function__` and `__array_ufunc__` and the "non-overloaded version of the provided function"? This NEP calls it an "arbitrary callable". In `__array_ufunc__` it turns out people count on it being exactly the `np.ufunc`. Matti
On Tue, Jun 5, 2018 at 2:47 PM Matti Picus <matti.picus@gmail.com> wrote:
The ""non-overloaded version of the provided function" is entirely hypothetical at this point. If we use a decorator to implement overloads, it would be the undecorated function, e.g., the original definition of concatenate here: @overload_for_array_function(['arrays', 'out'])def concatenate(arrays, axis=0, out=None): ... # continue with the definition of concatenate This NEP calls it an "arbitrary callable".
In `__array_ufunc__` it turns out people count on it being exactly the `np.ufunc`.
Right, I think this is good guarantee to provide. Certainly it's one that people fine useful.
Yes, the function should definitely be the same as what the user called - i.e., the decorated function. I'm only wondering if it would also be possible to have access to the undecorated one (via `coerce` or `ndarray.__array_function__` or otherwise). -- Marten
Hmm, does this mean the callable that gets passed into __array_ufunc__ will change? I'm pretty sure that will break the dispatch mechanism I'm using in my __array_ufunc__ implementation, which directly checks whether the callable is in one of several tuples of functions that have different behavior. On Tue, Jun 5, 2018 at 7:32 PM, Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
Oh wait, since the decorated version of the ufunc will be the one in the public numpy API it won't break. It would only break if the callable that was passed in *wasn't* the decorated version, so it kinda *has* to pass in the decorated function to preserve backward compatibility. Apologies for the noise. On Tue, Jun 5, 2018 at 7:39 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On 6. Jun 2018 at 05:41, Nathan Goldbaum <nathan12343@gmail.com> wrote: Oh wait, since the decorated version of the ufunc will be the one in the public numpy API it won't break. It would only break if the callable that was passed in *wasn't* the decorated version, so it kinda *has* to pass in the decorated function to preserve backward compatibility. Apologies for the noise. On Tue, Jun 5, 2018 at 7:39 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Section “Non-Goals” states that Ufuncs will not be part of this protocol, __array_ufunc__ will be used to override those as usual. Sent from Astro <https://www.helloastro.com> for Mac
Hi Stephan, I think we're getting to the stage where an updated text would be useful. For that, you may want to consider an actual implementation of, e.g., a very simple function like `np.reshape` as well as a more complicated one like `np.concatenate`, and in particular how the implementation finds out where its own instances are located. All the best, Marten
On Fri, Jun 8, 2018 at 8:58 AM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:
I think we're getting to the stage where an updated text would be useful.
Yes, I plan to work on this over the weekend. Stay tuned!
Yes, I agree that actual implementation (in Python rather than C for now) would be useful.
and in particular how the implementation finds out where its own instances are located.
I think we've discussed this before, but I don't think this is feasible to solve in general given the diversity of wrapped APIs. If you want to find the arguments in which a class' own instances appear, you will need to do that in your overloaded function. That said, if merely pulling out the flat list of arguments that are checked for and/or implement __array_function__ would be enough, we can probably figure out a way to expose that information.
In the end, somewhere inside the "dance", you are checking for `__array_function` - it would seem to me that at that point you know exactly where you are, and it would not be difficult to something like ``` types[new_type] += [where_i_am] ``` (where here I assume types is a defaultdict(list)) - has the set of types in keys and locations as values. But easier to discuss whether this is easy with some sample code to look at! -- Marten
I meant whatever the state of the dance routine is, e.g., the way the arguments are enumerated by the decorator (this is partially why some example code for the dance routine is needed -- I am not 100% how this should work, just seems logical that if the dance routine can understand it, so can __array_function__ implementations). -- Marten
Mixed return values of NotImplementedButCoercible and NotImplemented would still result in TypeError, and there would be no second chances for overloads. I would like to differ with you here: It can be quite useful to have second chances for overloads. Think ``np.func(list, custom_array))``: If second rounds did not exist, custom_array would need to have a list of coercible types (which is not nice IMO). It can also help in cases where performance/feature degradation isn’t an issue, so coercing all arguments that returned ``NotImplementedButCoercible`` would allow ``__array_function__`` to succeed where it wouldn’t normally. I mean, that’s one of the major uses of this sentinel right? If done in a for loop, it wouldn’t even slow down the nominal cases. It would have the adverse effect of not allowing for a default implementation to be as simple as you stated, though. One thing we could do is manually (inside ``__array_function__``) coerce anything that didn’t implement ``__array_function__``, and that’s acceptable to me too.
On Sun, Jun 3, 2018 at 9:54 PM Hameer Abbasi <einstein.edison@gmail.com> wrote:
Even if we did this, we would still want to preserve the equivalence between: 1. Returning NotImplementedButCoercible from __array_ufunc__ or __array_function__, and 2. Not implementing __array_ufunc__ or __array_function__ at all. Changing __array_ufunc__ to do multiple rounds of checks could indeed be useful in some cases, and you're right that it would not change existing behavior (in these cases we currently raise TypeError). But I'd rather leave that for a separate discussion, because it's orthogonal to our proposal here for __array_function__. (Personally, I don't think it would be worth the additional complexity.)
Should there be discussion of typing (pep-484) or abstract base classes in this nep? Are there any requirements on the result returned by __array_function__? On Mon, Jun 4, 2018, 2:20 AM Stephan Hoyer <shoyer@gmail.com> wrote:
On Mon, Jun 4, 2018 at 5:39 AM Matthew Harrigan <harrigan.matthew@gmail.com> wrote:
This is a good question that should be addressed in the NEP. Currently, we impose no limitations on the types returned by __array_function__ (or __array_ufunc__, for that matter). Given the complexity of potential __array_function__ implementations, I think this would be hard/impossible to do in general. I think the best case scenario we could hope for is that type checkers would identify that result of NumPy functions as: - numpy.ndarray if all inputs are numpy.ndarray objects - Any if any non-numpy.ndarray inputs implement the __array_function__ Based on my understanding of proposed rules for typing protocols [1] and overloads [2], I think this could just work, e.g., @overload def func(array: np.ndarray) -> np.ndarray: ... @overload def func(array: ImplementsArrayFunction) -> Any: ... [1] https://www.python.org/dev/peps/pep-0544/ [2] https://github.com/python/typing/issues/253#issuecomment-389262904
I agree that second rounds of overloads have to be left to the implementers of `__array_function__` - obviously, though, we should be sure that these rounds are rarely necessary... The link posted by Stephan [1] has some decent discussion for `__array_ufunc__` about when an override should re-call the function rather than try to do something itself. -- Marten [1] http://www.numpy.org/neps/nep-0013-ufunc-overrides.html#subclass-hierarchies
participants (6)
-
Hameer Abbasi -
Marten van Kerkwijk -
Matthew Harrigan -
Matti Picus -
Nathan Goldbaum -
Stephan Hoyer