On Fri, Sep 6, 2019 at 4:51 PM Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Sep 6, 2019 at 2:45 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:

There may be another very concrete one (that's not yet in the NEP): allowing other libraries that consume ndarrays to use overrides. An example is numpy.fft: currently both mkl_fft and pyfftw monkeypatch NumPy, something we don't like all that much (in particular for mkl_fft, because it's the default in Anaconda). `__array_function__` isn't able to help here, because it will always choose NumPy's own implementation for ndarray input. With unumpy you can support multiple libraries that consume ndarrays.

unumpy doesn't help with this either though, does it? unumpy is double-opt-in: the code using np.fft has to switch to using unumpy.fft instead, and then someone has to enable the backend.

Very good point. It would make a lot of sense to at least make unumpy default on fft/linalg/random, even if we want to keep it opt-in for the functions in the main namespace. But MKL/pyfftw

started out as opt-in – you could `import mkl_fft` or `import pyfftw` – and the whole reason they switched to monkeypatching is that they decided that opt-in wasn't good enough for them.

No, that's not correct. The MKL team has asked for a proper backend system, so they can plug into numpy rather than monkeypatch it. Oleksey, Chuck and I discussed that two years ago already at the NumFOCUS Summit 2017. This has been explicitly on the NumPy roadmap for quite a while: "A backend system for numpy.fft (so that e.g. fft-mkl doesn’t need to monkeypatch numpy)" (see https://numpy.org/neps/roadmap.html#other-functionality) And if Anaconda would like to default to it, that's possible - because one registered backend needs to be chosen as the default, that could be mkl-fft. That is still a major improvement over the situation today.

The import numpy.overridable part is meant to help garner adoption, and to prefer the unumpy module if it is available (which will continue to be developed separately). That way it isn't so tightly coupled to the release cycle. One alternative Sebastian Berg mentioned (and I am on board with) is just moving unumpy into the NumPy organisation. What we fear keeping it separate is that the simple act of a pip install unumpy will keep people from using it or trying it out.

Note that this is not the most critical aspect. I pushed for vendoring as numpy.overridable because I want to not derail the comparison with NEP 30 et al. with a "should we add a dependency" discussion. The interesting part to decide on first is: do we need the unumpy override mechanism? Vendoring opt-in vs. making it default vs. adding a dependency is of secondary interest right now.

Wait, but I thought the only reason we would have a dependency is if we're exporting it as part of the numpy namespace. If we keep the import as `import unumpy`, then it works just as well, without any dependency *or* vendoring in numpy, right?

Vendoring means "include the code". So no dependency on an external package. If we don't vendor, it's going to be either unused, or end up as a dependency for the whole SciPy/PyData stack. Actually, now that we've discussed the fft issue, I'd suggest to change the NEP to: vendor, and make default for fft, random, and linalg. Cheers, Ralf