Mailman 3 February 2020 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

help translating into Russian
by Inessa Pawson May 18, 2020

May 18, 2020

Our collaboration with the students and faculty from the Master’s program in Survey Methodology at the University of Michigan and the University of Maryland is underway. We are looking for a volunteer to translate the survey questionnaire into Russian. If you are available, or you know someone who would be interested to help, please leave a comment here: https://github.com/numpy/numpy-surveys/issues/1. -- Every good wish, *Inessa Pawson * Executive Director Albus Code

1 1

NEP 37: A dispatch protocol for NumPy-like modules
by Stephan Hoyer April 12, 2020

April 12, 2020

I am pleased to present a new NumPy Enhancement Proposal for discussion: "NEP-37: A dispatch protocol for NumPy-like modules." Feedback would be very welcome! The full text follows. The rendered proposal can also be found online at https://numpy.org/neps/nep-0037-array-module.html Best, Stephan Hoyer =================================================== NEP 37 — A dispatch protocol for NumPy-like modules =================================================== :Author: Stephan Hoyer <shoyer(a)google.com> :Author: Hameer Abbasi :Author: Sebastian Berg :Status: Draft :Type: Standards Track :Created: 2019-12-29 Abstract -------- NEP-18's ``__array_function__`` has been a mixed success. Some projects (e.g., dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. Others (e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose a new protocol, ``__array_module__``, that we expect could eventually subsume most use-cases for ``__array_function__``. The protocol requires explicit adoption by both users and library authors, which ensures backwards compatibility, and is also significantly simpler than ``__array_function__``, both of which we expect will make it easier to adopt. Why ``__array_function__`` hasn't been enough --------------------------------------------- There are two broad ways in which NEP-18 has fallen short of its goals: 1. **Maintainability concerns**. `__array_function__` has significant implications for libraries that use it: - Projects like `PyTorch <https://github.com/pytorch/pytorch/issues/22402>`_, `JAX <https://github.com/google/jax/issues/1565>`_ and even `scipy.sparse <https://github.com/scipy/scipy/issues/10362>`_ have been reluctant to implement `__array_function__` in part because they are concerned about **breaking existing code**: users expect NumPy functions like ``np.concatenate`` to return NumPy arrays. This is a fundamental limitation of the ``__array_function__`` design, which we chose to allow overriding the existing ``numpy`` namespace. - ``__array_function__`` currently requires an "all or nothing" approach to implementing NumPy's API. There is no good pathway for **incremental adoption**, which is particularly problematic for established projects for which adopting ``__array_function__`` would result in breaking changes. - It is no longer possible to use **aliases to NumPy functions** within modules that support overrides. For example, both CuPy and JAX set ``result_type = np.result_type``. - Implementing **fall-back mechanisms** for unimplemented NumPy functions by using NumPy's implementation is hard to get right (but see the `version from dask <https://github.com/dask/dask/pull/5043>`_), because ``__array_function__`` does not present a consistent interface. Converting all arguments of array type requires recursing into generic arguments of the form ``*args, **kwargs``. 2. **Limitations on what can be overridden.** ``__array_function__`` has some important gaps, most notably array creation and coercion functions: - **Array creation** routines (e.g., ``np.arange`` and those in ``np.random``) need some other mechanism for indicating what type of arrays to create. `NEP 36 <https://github.com/numpy/numpy/pull/14715>`_ proposed adding optional ``like=`` arguments to functions without existing array arguments. However, we still lack any mechanism to override methods on objects, such as those needed by ``np.random.RandomState``. - **Array conversion** can't reuse the existing coercion functions like ``np.asarray``, because ``np.asarray`` sometimes means "convert to an exact ``np.ndarray``" and other times means "convert to something _like_ a NumPy array." This led to the `NEP 30 <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_ proposal for a separate ``np.duckarray`` function, but this still does not resolve how to cast one duck array into a type matching another duck array. ``get_array_module`` and the ``__array_module__`` protocol ---------------------------------------------------------- We propose a new user-facing mechanism for dispatching to a duck-array implementation, ``numpy.get_array_module``. ``get_array_module`` performs the same type resolution as ``__array_function__`` and returns a module with an API promised to match the standard interface of ``numpy`` that can implement operations on all provided array types. The protocol itself is both simpler and more powerful than ``__array_function__``, because it doesn't need to worry about actually implementing functions. We believe it resolves most of the maintainability and functionality limitations of ``__array_function__``. The new protocol is opt-in, explicit and with local control; see :ref:`appendix-design-choices` for discussion on the importance of these design features. The array module contract ========================= Modules returned by ``get_array_module``/``__array_module__`` should make a best effort to implement NumPy's core functionality on new array types(s). Unimplemented functionality should simply be omitted (e.g., accessing an unimplemented function should raise ``AttributeError``). In the future, we anticipate codifying a protocol for requesting restricted subsets of ``numpy``; see :ref:`requesting-restricted-subsets` for more details. How to use ``get_array_module`` =============================== Code that wants to support generic duck arrays should explicitly call ``get_array_module`` to determine an appropriate array module from which to call functions, rather than using the ``numpy`` namespace directly. For example: .. code:: python # calls the appropriate version of np.something for x and y module = np.get_array_module(x, y) module.something(x, y) Both array creation and array conversion are supported, because dispatching is handled by ``get_array_module`` rather than via the types of function arguments. For example, to use random number generation functions or methods, we can simply pull out the appropriate submodule: .. code:: python def duckarray_add_random(array): module = np.get_array_module(array) noise = module.random.randn(*array.shape) return array + noise We can also write the duck-array ``stack`` function from `NEP 30 <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_, without the need for a new ``np.duckarray`` function: .. code:: python def duckarray_stack(arrays): module = np.get_array_module(*arrays) arrays = [module.asarray(arr) for arr in arrays] shapes = {arr.shape for arr in arrays} if len(shapes) != 1: raise ValueError('all input arrays must have the same shape') expanded_arrays = [arr[module.newaxis, ...] for arr in arrays] return module.concatenate(expanded_arrays, axis=0) By default, ``get_array_module`` will return the ``numpy`` module if no arguments are arrays. This fall-back can be explicitly controlled by providing the ``module`` keyword-only argument. It is also possible to indicate that an exception should be raised instead of returning a default array module by setting ``module=None``. How to implement ``__array_module__`` ===================================== Libraries implementing a duck array type that want to support ``get_array_module`` need to implement the corresponding protocol, ``__array_module__``. This new protocol is based on Python's dispatch protocol for arithmetic, and is essentially a simpler version of ``__array_function__``. Only one argument is passed into ``__array_module__``, a Python collection of unique array types passed into ``get_array_module``, i.e., all arguments with an ``__array_module__`` attribute. The special method should either return an namespace with an API matching ``numpy``, or ``NotImplemented``, indicating that it does not know how to handle the operation: .. code:: python class MyArray: def __array_module__(self, types): if not all(issubclass(t, MyArray) for t in types): return NotImplemented return my_array_module Returning custom objects from ``__array_module__`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``my_array_module`` will typically, but need not always, be a Python module. Returning a custom objects (e.g., with functions implemented via ``__getattr__``) may be useful for some advanced use cases. For example, custom objects could allow for partial implementations of duck array modules that fall-back to NumPy (although this is not recommended in general because such fall-back behavior can be error prone): .. code:: python class MyArray: def __array_module__(self, types): if all(issubclass(t, MyArray) for t in types): return ArrayModule() else: return NotImplemented class ArrayModule: def __getattr__(self, name): import base_module return getattr(base_module, name, getattr(numpy, name)) Subclassing from ``numpy.ndarray`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the same guidance about well-defined type casting hierarchies from NEP-18 still applies. ``numpy.ndarray`` itself contains a matching implementation of ``__array_module__``, which is convenient for subclasses: .. code:: python class ndarray: def __array_module__(self, types): if all(issubclass(t, ndarray) for t in types): return numpy else: return NotImplemented NumPy's internal machinery ========================== The type resolution rules of ``get_array_module`` follow the same model as Python and NumPy's existing dispatch protocols: subclasses are called before super-classes, and otherwise left to right. ``__array_module__`` is guaranteed to be called only a single time on each unique type. The actual implementation of `get_array_module` will be in C, but should be equivalent to this Python code: .. code:: python def get_array_module(*arrays, default=numpy): implementing_arrays, types = _implementing_arrays_and_types(arrays) if not implementing_arrays and default is not None: return default for array in implementing_arrays: module = array.__array_module__(types) if module is not NotImplemented: return module raise TypeError("no common array module found") def _implementing_arrays_and_types(relevant_arrays): types = [] implementing_arrays = [] for array in relevant_arrays: t = type(array) if t not in types and hasattr(t, '__array_module__'): types.append(t) # Subclasses before superclasses, otherwise left to right index = len(implementing_arrays) for i, old_array in enumerate(implementing_arrays): if issubclass(t, type(old_array)): index = i break implementing_arrays.insert(index, array) return implementing_arrays, types Relationship with ``__array_ufunc__`` and ``__array_function__`` ---------------------------------------------------------------- These older protocols have distinct use-cases and should remain =============================================================== ``__array_module__`` is intended to resolve limitations of ``__array_function__``, so it is natural to consider whether it could entirely replace ``__array_function__``. This would offer dual benefits: (1) simplifying the user-story about how to override NumPy and (2) removing the slowdown associated with checking for dispatch when calling every NumPy function. However, ``__array_module__`` and ``__array_function__`` are pretty different from a user perspective: it requires explicit calls to ``get_array_function``, rather than simply reusing original ``numpy`` functions. This is probably fine for *libraries* that rely on duck-arrays, but may be frustratingly verbose for interactive use. Some of the dispatching use-cases for ``__array_ufunc__`` are also solved by ``__array_module__``, but not all of them. For example, it is still useful to be able to define non-NumPy ufuncs (e.g., from Numba or SciPy) in a generic way on non-NumPy arrays (e.g., with dask.array). Given their existing adoption and distinct use cases, we don't think it makes sense to remove or deprecate ``__array_function__`` and ``__array_ufunc__`` at this time. Mixin classes to implement ``__array_function__`` and ``__array_ufunc__`` ========================================================================= Despite the user-facing differences, ``__array_module__`` and a module implementing NumPy's API still contain sufficient functionality needed to implement dispatching with the existing duck array protocols. For example, the following mixin classes would provide sensible defaults for these special methods in terms of ``get_array_module`` and ``__array_module__``: .. code:: python class ArrayUfuncFromModuleMixin: def __array_ufunc__(self, ufunc, method, *inputs, **kwargs): arrays = inputs + kwargs.get('out', ()) try: array_module = np.get_array_module(*arrays) except TypeError: return NotImplemented try: # Note this may have false positive matches, if ufunc.__name__ # matches the name of a ufunc defined by NumPy. Unfortunately # there is no way to determine in which module a ufunc was # defined. new_ufunc = getattr(array_module, ufunc.__name__) except AttributeError: return NotImplemented try: callable = getattr(new_ufunc, method) except AttributeError: return NotImplemented return callable(*inputs, **kwargs) class ArrayFunctionFromModuleMixin: def __array_function__(self, func, types, args, kwargs): array_module = self.__array_module__(types) if array_module is NotImplemented: return NotImplemented # Traverse submodules to find the appropriate function modules = func.__module__.split('.') assert modules[0] == 'numpy' for submodule in modules[1:]: module = getattr(module, submodule, None) new_func = getattr(module, func.__name__, None) if new_func is None: return NotImplemented return new_func(*args, **kwargs) To make it easier to write duck arrays, we could also add these mixin classes into ``numpy.lib.mixins`` (but the examples above may suffice). Alternatives considered ----------------------- Naming ====== We like the name ``__array_module__`` because it mirrors the existing ``__array_function__`` and ``__array_ufunc__`` protocols. Another reasonable choice could be ``__array_namespace__``. It is less clear what the NumPy function that calls this protocol should be called (``get_array_module`` in this proposal). Some possible alternatives: ``array_module``, ``common_array_module``, ``resolve_array_module``, ``get_namespace``, ``get_numpy``, ``get_numpylike_module``, ``get_duck_array_module``. .. _requesting-restricted-subsets: Requesting restricted subsets of NumPy's API ============================================ Over time, NumPy has accumulated a very large API surface, with over 600 attributes in the top level ``numpy`` module alone. It is unlikely that any duck array library could or would want to implement all of these functions and classes, because the frequently used subset of NumPy is much smaller. We think it would be useful exercise to define "minimal" subset(s) of NumPy's API, omitting rarely used or non-recommended functionality. For example, minimal NumPy might include ``stack``, but not the other stacking functions ``column_stack``, ``dstack``, ``hstack`` and ``vstack``. This could clearly indicate to duck array authors and users want functionality is core and what functionality they can skip. Support for requesting a restricted subset of NumPy's API would be a natural feature to include in ``get_array_function`` and ``__array_module__``, e.g., .. code:: python # array_module is only guaranteed to contain "minimal" NumPy array_module = np.get_array_module(*arrays, request='minimal') To facilitate testing with NumPy and use with any valid duck array library, NumPy itself would return restricted versions of the ``numpy`` module when ``get_array_module`` is called only on NumPy arrays. Omitted functions would simply not exist. Unfortunately, we have not yet figured out what these restricted subsets should be, so it doesn't make sense to do this yet. When/if we do, we could either add new keyword arguments to ``get_array_module`` or add new top level functions, e.g., ``get_minimal_array_module``. We would also need to add either a new protocol patterned off of ``__array_module__`` (e.g., ``__array_module_minimal__``), or could add an optional second argument to ``__array_module__`` (catching errors with ``try``/``except``). A new namespace for implicit dispatch ===================================== Instead of supporting overrides in the main `numpy` namespace with ``__array_function__``, we could create a new opt-in namespace, e.g., ``numpy.api``, with versions of NumPy functions that support dispatching. These overrides would need new opt-in protocols, e.g., ``__array_function_api__`` patterned off of ``__array_function__``. This would resolve the biggest limitations of ``__array_function__`` by being opt-in and would also allow for unambiguously overriding functions like ``asarray``, because ``np.api.asarray`` would always mean "convert an array-like object." But it wouldn't solve all the dispatching needs met by ``__array_module__``, and would leave us with supporting a considerably more complex protocol both for array users and implementors. We could potentially implement such a new namespace *via* the ``__array_module__`` protocol. Certainly some users would find this convenient, because it is slightly less boilerplate. But this would leave users with a confusing choice: when should they use `get_array_module` vs. `np.api.something`. Also, we would have to add and maintain a whole new module, which is considerably more expensive than merely adding a function. Dispatching on both types and arrays instead of only types ========================================================== Instead of supporting dispatch only via unique array types, we could also support dispatch via array objects, e.g., by passing an ``arrays`` argument as part of the ``__array_module__`` protocol. This could potentially be useful for dispatch for arrays with metadata, such provided by Dask and Pint, but would impose costs in terms of type safety and complexity. For example, a library that supports arrays on both CPUs and GPUs might decide on which device to create a new arrays from functions like ``ones`` based on input arguments: .. code:: python class Array: def __array_module__(self, types, arrays): useful_arrays = tuple(a in arrays if isinstance(a, Array)) if not useful_arrays: return NotImplemented prefer_gpu = any(a.prefer_gpu for a in useful_arrays) return ArrayModule(prefer_gpu) class ArrayModule: def __init__(self, prefer_gpu): self.prefer_gpu = prefer_gpu def __getattr__(self, name): import base_module base_func = getattr(base_module, name) return functools.partial(base_func, prefer_gpu=self.prefer_gpu) This might be useful, but it's not clear if we really need it. Pint seems to get along OK without any explicit array creation routines (favoring multiplication by units, e.g., ``np.ones(5) * ureg.m``), and for the most part Dask is also OK with existing ``__array_function__`` style overides (e.g., favoring ``np.ones_like`` over ``np.ones``). Choosing whether to place an array on the CPU or GPU could be solved by `making array creation lazy <https://github.com/google/jax/pull/1668>`_. .. _appendix-design-choices: Appendix: design choices for API overrides ------------------------------------------ There is a large range of possible design choices for overriding NumPy's API. Here we discuss three major axes of the design decision that guided our design for ``__array_module__``. Opt-in vs. opt-out for users ============================ The ``__array_ufunc__`` and ``__array_function__`` protocols provide a mechanism for overriding NumPy functions *within NumPy's existing namespace*. This means that users need to explicitly opt-out if they do not want any overridden behavior, e.g., by casting arrays with ``np.asarray()``. In theory, this approach lowers the barrier for adopting these protocols in user code and libraries, because code that uses the standard NumPy namespace is automatically compatible. But in practice, this hasn't worked out. For example, most well-maintained libraries that use NumPy follow the best practice of casting all inputs with ``np.asarray()``, which they would have to explicitly relax to use ``__array_function__``. Our experience has been that making a library compatible with a new duck array type typically requires at least a small amount of work to accommodate differences in the data model and operations that can be implemented efficiently. These opt-out approaches also considerably complicate backwards compatibility for libraries that adopt these protocols, because by opting in as a library they also opt-in their users, whether they expect it or not. For winning over libraries that have been unable to adopt ``__array_function__``, an opt-in approach seems like a must. Explicit vs. implicit choice of implementation ============================================== Both ``__array_ufunc__`` and ``__array_function__`` have implicit control over dispatching: the dispatched functions are determined via the appropriate protocols in every function call. This generalizes well to handling many different types of objects, as evidenced by its use for implementing arithmetic operators in Python, but it has two downsides: 1. *Speed*: it imposes additional overhead in every function call, because each function call needs to inspect each of its arguments for overrides. This is why arithmetic on builtin Python numbers is slow. 2. *Readability*: it is not longer immediately evident to readers of code what happens when a function is called, because the function's implementation could be overridden by any of its arguments. In contrast, importing a new library (e.g., ``import dask.array as da``) with an API matching NumPy is entirely explicit. There is no overhead from dispatch or ambiguity about which implementation is being used. Explicit and implicit choice of implementations are not mutually exclusive options. Indeed, most implementations of NumPy API overrides via ``__array_function__`` that we are familiar with (namely, dask, CuPy and sparse, but not Pint) also include an explicit way to use their version of NumPy's API by importing a module directly (``dask.array``, ``cupy`` or ``sparse``, respectively). Local vs. non-local vs. global control ====================================== The final design axis is how users control the choice of API: - **Local control**, as exemplified by multiple dispatch and Python protocols for arithmetic, determines which implementation to use either by checking types or calling methods on the direct arguments of a function. - **Non-local control** such as `np.errstate <https://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html >`_ overrides behavior with global-state via function decorators or context-managers. Control is determined hierarchically, via the inner-most context. - **Global control** provides a mechanism for users to set default behavior, either via function calls or configuration files. For example, matplotlib allows setting a global choice of plotting backend. Local control is generally considered a best practice for API design, because control flow is entirely explicit, which makes it the easiest to understand. Non-local and global control are occasionally used, but generally either due to ignorance or a lack of better alternatives. In the case of duck typing for NumPy's public API, we think non-local or global control would be mistakes, mostly because they **don't compose well**. If one library sets/needs one set of overrides and then internally calls a routine that expects another set of overrides, the resulting behavior may be very surprising. Higher order functions are especially problematic, because the context in which functions are evaluated may not be the context in which they are defined. One class of override use cases where we think non-local and global control are appropriate is for choosing a backend system that is guaranteed to have an entirely consistent interface, such as a faster alternative implementation of ``numpy.fft`` on NumPy arrays. However, these are out of scope for the current proposal, which is focused on duck arrays.

6 27

New DTypes: Are scalars a central concept in NumPy or not?
by Sebastian Berg April 8, 2020

April 8, 2020

Hi all, When we create new datatypes, we have the option to make new choices for the new datatypes [0] (not the existing ones). The question is: Should every NumPy datatype have a scalar associated and should operations like indexing return a scalar or a 0-D array? This is in my opinion a complex, almost philosophical, question, and we do not have to settle anything for a long time. But, if we do not decide a direction before we have many new datatypes the decision will make itself... So happy about any ideas, even if its just a gut feeling :). There are various points. I would like to mostly ignore the technical ones, but I am listing them anyway here: * Scalars are faster (although that can be optimized likely) * Scalars have a lower memory footprint * The current implementation incurs a technical debt in NumPy. (I do not think that is a general issue, though. We could automatically create scalars for each new datatype probably.) Advantages of having no scalars: * No need to keep track of scalars to preserve them in ufuncs, or libraries using `np.asarray`, do they need `np.asarray_or_scalar`? (or decide they return always arrays, although ufuncs may not) * Seems simpler in many ways, you always know the output will be an array if it has to do with NumPy. Advantages of having scalars: * Scalars are immutable and we are used to them from Python. A 0-D array cannot be used as a dictionary key consistently [1]. I.e. without scalars as first class citizen `dict[arr1d[0]]` cannot work, `dict[arr1d[0].item()]` may (if `.item()` is defined, and e.g. `dict[arr1d[0].frozen()]` could make a copy to work. [2] * Object arrays as we have them now make sense, `arr1d[0]` can reasonably return a Python object. I.e. arrays feel more like container if you can take elements out easily. Could go both ways: * Scalar math `scalar = arr1d[0]; scalar += 1` modifies the array without scalars. With scalars `arr1d[0, ...]` clarifies the meaning. (In principle it is good to never use `arr2d[0]` to get a 1D slice, probably more-so if scalars exist.) Note: array-scalars (the current NumPy scalars) are not useful in my opinion [3]. A scalar should not be indexed or have a shape. I do not believe in scalars pretending to be arrays. I personally tend towards liking scalars. If Python was a language where the array (array-programming) concept was ingrained into the language itself, I would lean the other way. But users are used to scalars, and they "put" scalars into arrays. Array objects are in some ways strange in Python, and I feel not having scalars detaches them further. Having scalars, however also means we should preserve them. I feel in principle that is actually fairly straight forward. E.g. for ufuncs: * np.add(scalar, scalar) -> scalar * np.add.reduce(arr, axis=None) -> scalar * np.add.reduce(arr, axis=1) -> array (even if arr is 1d) * np.add.reduce(scalar, axis=()) -> array Of course libraries that do `np.asarray` would/could basically chose to not preserve scalars: Their signature is defined as taking strictly array input. Cheers, Sebastian [0] At best this can be a vision to decide which way they may evolve. [1] E.g. PyTorch uses `hash(tensor) == id(tensor)` which is arguably strange. E.g. Quantity defines hash correctly, but does not fully ensure immutability for 0-D Quantities. Ensuring immutability in a world where "views" are a central concept requires a write-only copy. [2] Arguably `.item()` would always return a scalar, but it would be a second class citizen. (Although if it returns a scalar, at least we already have a scalar implementation.) [3] They are necessary due to technical debt for NumPy datatypes though.

8 14

Proposal to accept NEP #44: Restructuring the NumPy Documentation
by Melissa Mendonça March 2, 2020

March 2, 2020

Hi all, I am proposing the acceptance of NEP 44 - Restructuring the NumPy Documentation. https://numpy.org/neps/nep-0044-restructuring-numpy-docs.html There were some comments about reorganizing the text to make it clearer, and some rewording regarding Reference Guides, developer "under-the-hoods" documentation and the possible mechanism for community contributions in the form of new tutorials or how-tos. Overall I believe we have answered all comments and there were no other points of concern. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details: see http://www.numpy.org/neps/nep-0000.html Thanks for the comments! Looking forward to start working on this :) -- Melissa Weber Mendonça

3 3

[ANN] pvlib participating in GSoC
by Dr. Mark Alexander Mikofski PhD Feb. 27, 2020

Feb. 27, 2020

Excited to announce that pvlib python <https://pvlib-python.readthedocs.io/en/latest/> is participating in its first ever Google Summer of Code GSoC under the NumFOCUS umbrella. If you are a student interested in modeling renewable solar energy please apply: https://summerofcode.withgoogle.com/organizations/4727917315096576/ For project ideas, take a look at the pvlib github wiki: https://summerofcode.withgoogle.com/organizations/4727917315096576/ and reach out on our Google Groups forum: https://groups.google.com/forum/#!forum/pvlib-python We look forward to hearing from you! -- Mark Mikofski, PhD (2005) *Fiat Lux*

1 1

Output type of round is inconsistent with python built-in
by Hameer Abbasi Feb. 27, 2020

Feb. 27, 2020

Hello, Currently, the built-in Python round (which is different from np.round) when called on a np.float64 returns a np.float64, due to its __round__ method. A congruous statement is true for np.float32. However, since Python 3, the default behavior of round is to return a Python int when it operates on a Python float. This is a mismatch according to the Liskov Substitution Principle<https://en.wikipedia.org/wiki/Liskov_substitution_principle>, as both these types subclass Python’s float. This has been brought up in gh-15297<https://github.com/numpy/numpy/issues/15297>. Here is the problem summed up in code: >>> type(round(np.float64(5))) <class 'numpy.float64'> >>> type(round(np.float32(5))) <class 'numpy.float32'> >>> type(round(float(5))) <class 'int'> This problem manifests itself most prominently when trying to index into collections: >>> np.arange(6)[round(float(5))] 5 >>> np.arange(6)[round(np.float64(5))] Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices There still remains the question, do we return Python ints or np.int64s? * Python ints have the advantage of not overflowing. * If we decide to add __round__ to arrays in the future, Python ints may become inconsistent with our design, as such a method will return an int64 array. This was issue was discussed in the weekly triage meeting today, and the following plan of action was proposed: * change scalar floats to return integers for __round__ (which integer type was not discussed, I propose np.int64) * not change anything else: not 0d arrays and not other numpy functionality Does anyone have any thoughts on the proposal? Best regards, Hameer Abbasi

4 16

Re: [Numpy-discussion] Output type of round is inconsistent with python built-in
by Stefano Miccoli Feb. 27, 2020

Feb. 27, 2020

There several mixed issues here. 1. PEP 3141 <https://www.python.org/dev/peps/pep-3141/> compliance. Numpy scalars are `numbers.Real` instances, and have to respect the `__round__` semantics defined by PEP 3141: def __round__(self, ndigits:Integral=None): """Rounds self to ndigits decimal places, defaulting to 0. If ndigits is omitted or None, returns an Integral, otherwise returns a Real, preferably of the same type as self. Types may choose which direction to round half. For example, float rounds half toward even. """ This means that if Real -> Real rounding is desired one should call `round(x, 0)` or `np.around(x)`. This semantics only dictates that the return type should be Integral, so for `round(x)` and `round(x, None)` np.float32 -> np.int32 np.float32 -> np.int64 np.float64 -> np.int64 np.floatXX -> int are all OK. I think also that it is perfectly OK to raise an overflow on `round(x)` 2. Liskov substitution principle `np.float64` floats are also `float` instances (but `np.float32` are not.) This means that strictly respecting LSP means that `np.float64` should round to python `int`, since `round(x)` never overflows for python `float`. Here we have several options. - round `np.float64` -> `int` and respect LSP. - relax LSP, and round `np.float64` -> `np.int64`. Who cares about `round(1e300)`? - decide that there is no reason for having `np.float64` a subclass of `float`, so that LSP does not apply. This all said, I think that these are the two most sensible choices for `round(x)`: np.float32 -> np.int32 np.float64 -> np.int64 drop np.float64 subclassing python float or np.float32 -> int np.float64 -> int keep np.float64 subclassing python float The second one seems to me the less disruptive one. Bye Stefano

1 0