[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

Ralf Gommers ralf.gommers at gmail.com
Fri Sep 6 17:45:11 EDT 2019


On Fri, Sep 6, 2019 at 1:32 AM Hameer Abbasi <einstein.edison at gmail.com>
wrote:

> That's a lot of very good questions! Let me see if I can answer them
> one-by-one.
>
> On 06.09.19 09:49, Nathaniel Smith wrote:
>
> But those are general questions about unumpy, and I'm guessing no-one
> knows all the answers yet... and these question actually aren't super
> relevant to the NEP. The NEP isn't inventing unumpy. IIUC, the main
> thing the NEP is proposes is simply to make  "numpy.overridable" an
> alias for "unumpy".
>
> It's not clear to me what problem this alias is solving. If all
> downstream users have to update their imports anyway, then they can
> write "import unumpy as np" just as easily as they can write "import
> numpy.overridable as np". I guess the main reason this is a NEP is
> because the unumpy project is hoping to get an "official stamp of
> approval" from numpy?
>
> Also because we have NEP 30 for yet another protocol, and there's likely
another NEP to follow after that for array creation. Those use cases are
covered by unumpy, so it makes sense to have a NEP for that as well, so
they can be considered side-by-side.

> That's part of it. The concrete problems it's solving are threefold:
>
>    1. Array creation functions can be overridden.
>    2. Array coercion is now covered.
>    3. "Default implementations" will allow you to re-write your NumPy
>    array more easily, when such efficient implementations exist in terms of
>    other NumPy functions. That will also help achieve similar semantics, but
>    as I said, they're just "default"...
>
> There may be another very concrete one (that's not yet in the NEP):
allowing other libraries that consume ndarrays to use overrides. An example
is numpy.fft: currently both mkl_fft and pyfftw monkeypatch NumPy,
something we don't like all that much (in particular for mkl_fft, because
it's the default in Anaconda). `__array_function__` isn't able to help
here, because it will always choose NumPy's own implementation for ndarray
input. With unumpy you can support multiple libraries that consume ndarrays.

Another example is einsum: if you want to use opt_einsum for all inputs
(including ndarrays), then you cannot use np.einsum. And yet another is
using bottleneck (https://kwgoodman.github.io/bottleneck-doc/reference.html)
for nan-functions and partition. There's likely more of these.

The point is: sometimes the array protocols are preferred (e.g.
Dask/Xarray-style meta-arrays), sometimes unumpy-style dispatch works
better. It's also not necessarily an either or, they can be complementary.

Actually, after writing this I just realized something. With 1.17.x we have:

```
In [1]: import dask.array as da


In [2]: d = da.from_array(np.linspace(0, 1))


In [3]: np.fft.fft(d)

Out[3]: dask.array<fft, shape=(50,), dtype=complex128, chunksize=(50,)>
```

In Anaconda `np.fft.fft` *is* `mkl_fft._numpy_fft.fft`, so this won't work.
We have no bug report yet because 1.17.x hasn't landed in conda defaults
yet (perhaps this is a/the reason why?), but it will be a problem.

The import numpy.overridable part is meant to help garner adoption, and to
> prefer the unumpy module if it is available (which will continue to be
> developed separately). That way it isn't so tightly coupled to the release
> cycle. One alternative Sebastian Berg mentioned (and I am on board with) is
> just moving unumpy into the NumPy organisation. What we fear keeping it
> separate is that the simple act of a pip install unumpy will keep people
> from using it or trying it out.
>
Note that this is not the most critical aspect. I pushed for vendoring as
numpy.overridable because I want to not derail the comparison with NEP 30
et al. with a "should we add a dependency" discussion. The interesting part
to decide on first is: do we need the unumpy override mechanism? Vendoring
opt-in vs. making it default vs. adding a dependency is of secondary
interest right now.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190906/4b528a17/attachment-0001.html>


More information about the NumPy-Discussion mailing list