[Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API

Peter Bell PeterBell10 at live.co.uk
Sat Sep 7 05:32:41 EDT 2019

>> There may be another very concrete one (that's not yet in the NEP): allowing other libraries that consume ndarrays to use overrides. An example is numpy.fft: currently both mkl_fft and pyfftw monkeypatch NumPy, something we don't like all that much (in particular for mkl_fft, because it's the default in Anaconda). `__array_function__` isn't able to help here, because it will always choose NumPy's own implementation for ndarray input. With unumpy you can support multiple libraries that consume ndarrays.

> unumpy doesn't help with this either though, does it? unumpy is double-opt-in: the code using np.fft has to switch to using unumpy.fft instead, and then someone has to enable the backend. But MKL/pyfftw started out as opt-in – you could `import mkl_fft` or `import pyfftw` – and the whole reason they switched to monkeypatching is that they decided that opt-in wasn't good enough for them.

Because numpy functions are used to write many library functions, the end user isn't always able to opt-in by changing imports. So, for library functions, monkey patching is not simply convenient but actually necessary. Take for example scipy.signal.fftconvolve: SciPy can't change to pyfftw for licensing reasons so with SciPy < 1.4 your only option is to monkey patch scipy.fftpack and numpy.fft. However in SciPy >= 1.4, thanks to the uarray-based backend support in scipy.fft, I can write

    from scipy import fft, signal
    import pyfftw.interfaces.scipy_fft as pyfftw_fft

    x = np.random.randn(1024, 1024)
    with fft.set_backend(pyfftw_fft):
        y = signal.fftconvolve(x, x)  # Calls pyfftw's rfft, irfft

Yes, we had to opt-in in the library function (signal moved from scipy.fftpack to scipy.fft). But because there can be distance between the set_backend call and the FFT calls, the library is now much more configurable. Generally speaking, any library written to use unumpy would be configurable: (i) by the user, (ii) at runtime, (iii) without changing library code and (iv) without monkey patching. 

In scipy.fft I actually did it slightly differently than unumpy: the scipy.fft interface itself has the uarray dispatch and I set SciPy's version of pocketfft as the default global backend. This means that normal users don't need to set a backend, and thus don't need to opt-in in any way. For NumPy to follow this pattern as well would require more change to NumPy's code base than the current NEP's suggestion, mainly in separating the interface from the implementation that would become the default backend.

- Peter 

More information about the NumPy-Discussion mailing list