[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

Ralf Gommers ralf.gommers at gmail.com
Sat Sep 7 02:04:02 EDT 2019


On Fri, Sep 6, 2019 at 4:51 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Sep 6, 2019 at 2:45 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> > There may be another very concrete one (that's not yet in the NEP):
> allowing other libraries that consume ndarrays to use overrides. An example
> is numpy.fft: currently both mkl_fft and pyfftw monkeypatch NumPy,
> something we don't like all that much (in particular for mkl_fft, because
> it's the default in Anaconda). `__array_function__` isn't able to help
> here, because it will always choose NumPy's own implementation for ndarray
> input. With unumpy you can support multiple libraries that consume ndarrays.
>
> unumpy doesn't help with this either though, does it? unumpy is
> double-opt-in: the code using np.fft has to switch to using unumpy.fft
> instead, and then someone has to enable the backend.


Very good point. It would make a lot of sense to at least make unumpy
default on fft/linalg/random, even if we want to keep it opt-in for the
functions in the main namespace.

But MKL/pyfftw
> started out as opt-in – you could `import mkl_fft` or `import pyfftw`
> – and the whole reason they switched to monkeypatching is that they
> decided that opt-in wasn't good enough for them.
>

No, that's not correct. The MKL team has asked for a proper backend system,
so they can plug into numpy rather than monkeypatch it. Oleksey, Chuck and
I discussed that two years ago already at the NumFOCUS Summit 2017.

This has been explicitly on the NumPy roadmap for quite a while:
"A backend system for numpy.fft (so that e.g. fft-mkl doesn’t need to
monkeypatch numpy)" (see
https://numpy.org/neps/roadmap.html#other-functionality)

And if Anaconda would like to default to it, that's possible - because one
registered backend needs to be chosen as the default, that could be
mkl-fft. That is still a major improvement over the situation today.


> >> The import numpy.overridable part is meant to help garner adoption, and
> to prefer the unumpy module if it is available (which will continue to be
> developed separately). That way it isn't so tightly coupled to the release
> cycle. One alternative Sebastian Berg mentioned (and I am on board with) is
> just moving unumpy into the NumPy organisation. What we fear keeping it
> separate is that the simple act of a pip install unumpy will keep people
> from using it or trying it out.
> >
> > Note that this is not the most critical aspect. I pushed for vendoring
> as numpy.overridable because I want to not derail the comparison with NEP
> 30 et al. with a "should we add a dependency" discussion. The interesting
> part to decide on first is: do we need the unumpy override mechanism?
> Vendoring opt-in vs. making it default vs. adding a dependency is of
> secondary interest right now.
>
> Wait, but I thought the only reason we would have a dependency is if
> we're exporting it as part of the numpy namespace. If we keep the
> import as `import unumpy`, then it works just as well, without any
> dependency *or* vendoring in numpy, right?
>

 Vendoring means "include the code". So no dependency on an external
package. If we don't vendor, it's going to be either unused, or end up as a
dependency for the whole SciPy/PyData stack.

Actually, now that we've discussed the fft issue, I'd suggest to change the
NEP to: vendor, and make default for fft, random, and linalg.

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190906/bd3ae39c/attachment.html>


More information about the NumPy-Discussion mailing list