[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

Juan Nunez-Iglesias jni at fastmail.com
Wed Oct 9 20:59:50 EDT 2019

Hi all, and thank you for all your hard work with this. 

I wanted to provide more of an "end user" perspective than I think has been present in this discussion so far. Over the past month, I've quickly skimmed some emails on this thread and skipped others altogether. I am far from a NumPy novice, but essentially *all* of the discussion went over my head. For a while my attitude was "Oh well, far smarter people than me are dealing with this, I'll let them figure it out." Looking at the participants in the thread, I worry that this is the attitude almost everyone has taken, and that the solution proposed will not be easy enough to deal with for any meaningful adoption. Certainly with `__array_function__` I only took interest when our tests broke with 1.17rc1.

Today I was particularly interested because I'm working to improve scikit-image support for pyopencl.Array inputs. I went back and read the original NEP and the latest iteration. Thank you again for the discussion, because the latest is indeed a vast improvement over the original.

I think the very motivation has the wrong focus. I would summarise it as "we've been coming up with all kinds of ways to do multiple dispatch for array-likes, and we've found that we need more ways, so let's come up with the One True Way." I think the focus should be on the users and community. Something along the lines of: "New implementations of array computing are cropping up left, right, and centre in Python (not to speak of other languages!). There are good reasons for this (GPUs, distributed computing, sparse data, etc), but it leaves users and library authors in a pickle: how can they ensure that their functions, written with NumPy array inputs and outputs in mind, work well in this ecosystem?"

With this new motivation in mind, I think that the user story below is (a) the best part of the NEP, but (b) underdeveloped. The NEP is all about "if I want my array implementation to work with this fancy dispatch system, what do I need to do?". But there should be more of "in user situations X, Y, and Z, what is the desired behaviour?"

> The way we propose the overrides will be used by end users is::
>  # On the library side
>  import numpy.overridable as unp
>  def library_function(array):
>  array = unp.asarray(array)
>  # Code using unumpy as usual
>  return array
>  # On the user side:
>  import numpy.overridable as unp
>  import uarray as ua
>  import dask.array as da
>  ua.register_backend(da)
>  library_function(dask_array) # works and returns dask_array
>  with unp.set_backend(da):
>  library_function([1, 2, 3, 4]) # actually returns a Dask array.
> Here, ``backend`` can be any compatible object defined either by NumPy or an
> external library, such as Dask or CuPy. Ideally, it should be the module
> ``dask.array`` or ``cupy`` itself.

Some questions about the above:

- What happens if I call `library_function(dask_array)` without registering `da` as a backend first? Will `unp.asarray` try to instantiate a potentially 100GB array? This seems bad.
- To get `library_function`, I presumably have to do `from fancy_array_library import library_function`. Can the code in `fancy_array_library` itself register backends, and if so, should/would fancy array libraries that want to maximise compatibility pre-register a bunch of backends so that users don't have to?

Here are a couple of code snippets that I would *want* to "just work". Maybe it's unreasonable, but imho the NEP should provide these as use cases (specifically: how library_function should be written so that they work, and what dask.array and pytorch would need to do so that they work, OR, why the NEP doesn't solve them).

from dask import array as da
from fancy_array_library import library_function # hopefully skimage one day ;)

data = da.from_zarr('myfile.zarr')
result = library_function(data) # result should still be dask, all things being equal

from dask import array as da
from magic_library import pytorch_predict

data = da.from_zarr('myfile.zarr')
result = pytorch_predict(data) # normally here I would use e.g. data.map_overlap, but could this be done magically?

There's probably a whole bunch of other "user stories" one can concoct, and no doubt many from the authors of the NEP themselves, but they don't come through in the NEP text. My apologies that I haven't read *all* the references: I understand that it is frustrating if the above are addressed there, but I think it's important to have this kind of context in the NEP itself.

Thank you again, and I hope the above is helpful rather than feels like more unnecessary churn.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20191009/4a32d210/attachment.html>

More information about the NumPy-Discussion mailing list