Mailman 3 March 2020 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 19 Feb '23

19 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 23 Jul '21

23 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

help translating into Russian
by Inessa Pawson 19 May '20

19 May '20

Our collaboration with the students and faculty from the Master’s program in Survey Methodology at the University of Michigan and the University of Maryland is underway. We are looking for a volunteer to translate the survey questionnaire into Russian. If you are available, or you know someone who would be interested to help, please leave a comment here: https://github.com/numpy/numpy-surveys/issues/1. -- Every good wish, *Inessa Pawson * Executive Director Albus Code

1 1

NEP 37: A dispatch protocol for NumPy-like modules
by Stephan Hoyer 12 Apr '20

12 Apr '20

I am pleased to present a new NumPy Enhancement Proposal for discussion: "NEP-37: A dispatch protocol for NumPy-like modules." Feedback would be very welcome! The full text follows. The rendered proposal can also be found online at https://numpy.org/neps/nep-0037-array-module.html Best, Stephan Hoyer =================================================== NEP 37 — A dispatch protocol for NumPy-like modules =================================================== :Author: Stephan Hoyer <shoyer(a)google.com> :Author: Hameer Abbasi :Author: Sebastian Berg :Status: Draft :Type: Standards Track :Created: 2019-12-29 Abstract -------- NEP-18's ``__array_function__`` has been a mixed success. Some projects (e.g., dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. Others (e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose a new protocol, ``__array_module__``, that we expect could eventually subsume most use-cases for ``__array_function__``. The protocol requires explicit adoption by both users and library authors, which ensures backwards compatibility, and is also significantly simpler than ``__array_function__``, both of which we expect will make it easier to adopt. Why ``__array_function__`` hasn't been enough --------------------------------------------- There are two broad ways in which NEP-18 has fallen short of its goals: 1. **Maintainability concerns**. `__array_function__` has significant implications for libraries that use it: - Projects like `PyTorch <https://github.com/pytorch/pytorch/issues/22402>`_, `JAX <https://github.com/google/jax/issues/1565>`_ and even `scipy.sparse <https://github.com/scipy/scipy/issues/10362>`_ have been reluctant to implement `__array_function__` in part because they are concerned about **breaking existing code**: users expect NumPy functions like ``np.concatenate`` to return NumPy arrays. This is a fundamental limitation of the ``__array_function__`` design, which we chose to allow overriding the existing ``numpy`` namespace. - ``__array_function__`` currently requires an "all or nothing" approach to implementing NumPy's API. There is no good pathway for **incremental adoption**, which is particularly problematic for established projects for which adopting ``__array_function__`` would result in breaking changes. - It is no longer possible to use **aliases to NumPy functions** within modules that support overrides. For example, both CuPy and JAX set ``result_type = np.result_type``. - Implementing **fall-back mechanisms** for unimplemented NumPy functions by using NumPy's implementation is hard to get right (but see the `version from dask <https://github.com/dask/dask/pull/5043>`_), because ``__array_function__`` does not present a consistent interface. Converting all arguments of array type requires recursing into generic arguments of the form ``*args, **kwargs``. 2. **Limitations on what can be overridden.** ``__array_function__`` has some important gaps, most notably array creation and coercion functions: - **Array creation** routines (e.g., ``np.arange`` and those in ``np.random``) need some other mechanism for indicating what type of arrays to create. `NEP 36 <https://github.com/numpy/numpy/pull/14715>`_ proposed adding optional ``like=`` arguments to functions without existing array arguments. However, we still lack any mechanism to override methods on objects, such as those needed by ``np.random.RandomState``. - **Array conversion** can't reuse the existing coercion functions like ``np.asarray``, because ``np.asarray`` sometimes means "convert to an exact ``np.ndarray``" and other times means "convert to something _like_ a NumPy array." This led to the `NEP 30 <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_ proposal for a separate ``np.duckarray`` function, but this still does not resolve how to cast one duck array into a type matching another duck array. ``get_array_module`` and the ``__array_module__`` protocol ---------------------------------------------------------- We propose a new user-facing mechanism for dispatching to a duck-array implementation, ``numpy.get_array_module``. ``get_array_module`` performs the same type resolution as ``__array_function__`` and returns a module with an API promised to match the standard interface of ``numpy`` that can implement operations on all provided array types. The protocol itself is both simpler and more powerful than ``__array_function__``, because it doesn't need to worry about actually implementing functions. We believe it resolves most of the maintainability and functionality limitations of ``__array_function__``. The new protocol is opt-in, explicit and with local control; see :ref:`appendix-design-choices` for discussion on the importance of these design features. The array module contract ========================= Modules returned by ``get_array_module``/``__array_module__`` should make a best effort to implement NumPy's core functionality on new array types(s). Unimplemented functionality should simply be omitted (e.g., accessing an unimplemented function should raise ``AttributeError``). In the future, we anticipate codifying a protocol for requesting restricted subsets of ``numpy``; see :ref:`requesting-restricted-subsets` for more details. How to use ``get_array_module`` =============================== Code that wants to support generic duck arrays should explicitly call ``get_array_module`` to determine an appropriate array module from which to call functions, rather than using the ``numpy`` namespace directly. For example: .. code:: python # calls the appropriate version of np.something for x and y module = np.get_array_module(x, y) module.something(x, y) Both array creation and array conversion are supported, because dispatching is handled by ``get_array_module`` rather than via the types of function arguments. For example, to use random number generation functions or methods, we can simply pull out the appropriate submodule: .. code:: python def duckarray_add_random(array): module = np.get_array_module(array) noise = module.random.randn(*array.shape) return array + noise We can also write the duck-array ``stack`` function from `NEP 30 <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_, without the need for a new ``np.duckarray`` function: .. code:: python def duckarray_stack(arrays): module = np.get_array_module(*arrays) arrays = [module.asarray(arr) for arr in arrays] shapes = {arr.shape for arr in arrays} if len(shapes) != 1: raise ValueError('all input arrays must have the same shape') expanded_arrays = [arr[module.newaxis, ...] for arr in arrays] return module.concatenate(expanded_arrays, axis=0) By default, ``get_array_module`` will return the ``numpy`` module if no arguments are arrays. This fall-back can be explicitly controlled by providing the ``module`` keyword-only argument. It is also possible to indicate that an exception should be raised instead of returning a default array module by setting ``module=None``. How to implement ``__array_module__`` ===================================== Libraries implementing a duck array type that want to support ``get_array_module`` need to implement the corresponding protocol, ``__array_module__``. This new protocol is based on Python's dispatch protocol for arithmetic, and is essentially a simpler version of ``__array_function__``. Only one argument is passed into ``__array_module__``, a Python collection of unique array types passed into ``get_array_module``, i.e., all arguments with an ``__array_module__`` attribute. The special method should either return an namespace with an API matching ``numpy``, or ``NotImplemented``, indicating that it does not know how to handle the operation: .. code:: python class MyArray: def __array_module__(self, types): if not all(issubclass(t, MyArray) for t in types): return NotImplemented return my_array_module Returning custom objects from ``__array_module__`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``my_array_module`` will typically, but need not always, be a Python module. Returning a custom objects (e.g., with functions implemented via ``__getattr__``) may be useful for some advanced use cases. For example, custom objects could allow for partial implementations of duck array modules that fall-back to NumPy (although this is not recommended in general because such fall-back behavior can be error prone): .. code:: python class MyArray: def __array_module__(self, types): if all(issubclass(t, MyArray) for t in types): return ArrayModule() else: return NotImplemented class ArrayModule: def __getattr__(self, name): import base_module return getattr(base_module, name, getattr(numpy, name)) Subclassing from ``numpy.ndarray`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the same guidance about well-defined type casting hierarchies from NEP-18 still applies. ``numpy.ndarray`` itself contains a matching implementation of ``__array_module__``, which is convenient for subclasses: .. code:: python class ndarray: def __array_module__(self, types): if all(issubclass(t, ndarray) for t in types): return numpy else: return NotImplemented NumPy's internal machinery ========================== The type resolution rules of ``get_array_module`` follow the same model as Python and NumPy's existing dispatch protocols: subclasses are called before super-classes, and otherwise left to right. ``__array_module__`` is guaranteed to be called only a single time on each unique type. The actual implementation of `get_array_module` will be in C, but should be equivalent to this Python code: .. code:: python def get_array_module(*arrays, default=numpy): implementing_arrays, types = _implementing_arrays_and_types(arrays) if not implementing_arrays and default is not None: return default for array in implementing_arrays: module = array.__array_module__(types) if module is not NotImplemented: return module raise TypeError("no common array module found") def _implementing_arrays_and_types(relevant_arrays): types = [] implementing_arrays = [] for array in relevant_arrays: t = type(array) if t not in types and hasattr(t, '__array_module__'): types.append(t) # Subclasses before superclasses, otherwise left to right index = len(implementing_arrays) for i, old_array in enumerate(implementing_arrays): if issubclass(t, type(old_array)): index = i break implementing_arrays.insert(index, array) return implementing_arrays, types Relationship with ``__array_ufunc__`` and ``__array_function__`` ---------------------------------------------------------------- These older protocols have distinct use-cases and should remain =============================================================== ``__array_module__`` is intended to resolve limitations of ``__array_function__``, so it is natural to consider whether it could entirely replace ``__array_function__``. This would offer dual benefits: (1) simplifying the user-story about how to override NumPy and (2) removing the slowdown associated with checking for dispatch when calling every NumPy function. However, ``__array_module__`` and ``__array_function__`` are pretty different from a user perspective: it requires explicit calls to ``get_array_function``, rather than simply reusing original ``numpy`` functions. This is probably fine for *libraries* that rely on duck-arrays, but may be frustratingly verbose for interactive use. Some of the dispatching use-cases for ``__array_ufunc__`` are also solved by ``__array_module__``, but not all of them. For example, it is still useful to be able to define non-NumPy ufuncs (e.g., from Numba or SciPy) in a generic way on non-NumPy arrays (e.g., with dask.array). Given their existing adoption and distinct use cases, we don't think it makes sense to remove or deprecate ``__array_function__`` and ``__array_ufunc__`` at this time. Mixin classes to implement ``__array_function__`` and ``__array_ufunc__`` ========================================================================= Despite the user-facing differences, ``__array_module__`` and a module implementing NumPy's API still contain sufficient functionality needed to implement dispatching with the existing duck array protocols. For example, the following mixin classes would provide sensible defaults for these special methods in terms of ``get_array_module`` and ``__array_module__``: .. code:: python class ArrayUfuncFromModuleMixin: def __array_ufunc__(self, ufunc, method, *inputs, **kwargs): arrays = inputs + kwargs.get('out', ()) try: array_module = np.get_array_module(*arrays) except TypeError: return NotImplemented try: # Note this may have false positive matches, if ufunc.__name__ # matches the name of a ufunc defined by NumPy. Unfortunately # there is no way to determine in which module a ufunc was # defined. new_ufunc = getattr(array_module, ufunc.__name__) except AttributeError: return NotImplemented try: callable = getattr(new_ufunc, method) except AttributeError: return NotImplemented return callable(*inputs, **kwargs) class ArrayFunctionFromModuleMixin: def __array_function__(self, func, types, args, kwargs): array_module = self.__array_module__(types) if array_module is NotImplemented: return NotImplemented # Traverse submodules to find the appropriate function modules = func.__module__.split('.') assert modules[0] == 'numpy' for submodule in modules[1:]: module = getattr(module, submodule, None) new_func = getattr(module, func.__name__, None) if new_func is None: return NotImplemented return new_func(*args, **kwargs) To make it easier to write duck arrays, we could also add these mixin classes into ``numpy.lib.mixins`` (but the examples above may suffice). Alternatives considered ----------------------- Naming ====== We like the name ``__array_module__`` because it mirrors the existing ``__array_function__`` and ``__array_ufunc__`` protocols. Another reasonable choice could be ``__array_namespace__``. It is less clear what the NumPy function that calls this protocol should be called (``get_array_module`` in this proposal). Some possible alternatives: ``array_module``, ``common_array_module``, ``resolve_array_module``, ``get_namespace``, ``get_numpy``, ``get_numpylike_module``, ``get_duck_array_module``. .. _requesting-restricted-subsets: Requesting restricted subsets of NumPy's API ============================================ Over time, NumPy has accumulated a very large API surface, with over 600 attributes in the top level ``numpy`` module alone. It is unlikely that any duck array library could or would want to implement all of these functions and classes, because the frequently used subset of NumPy is much smaller. We think it would be useful exercise to define "minimal" subset(s) of NumPy's API, omitting rarely used or non-recommended functionality. For example, minimal NumPy might include ``stack``, but not the other stacking functions ``column_stack``, ``dstack``, ``hstack`` and ``vstack``. This could clearly indicate to duck array authors and users want functionality is core and what functionality they can skip. Support for requesting a restricted subset of NumPy's API would be a natural feature to include in ``get_array_function`` and ``__array_module__``, e.g., .. code:: python # array_module is only guaranteed to contain "minimal" NumPy array_module = np.get_array_module(*arrays, request='minimal') To facilitate testing with NumPy and use with any valid duck array library, NumPy itself would return restricted versions of the ``numpy`` module when ``get_array_module`` is called only on NumPy arrays. Omitted functions would simply not exist. Unfortunately, we have not yet figured out what these restricted subsets should be, so it doesn't make sense to do this yet. When/if we do, we could either add new keyword arguments to ``get_array_module`` or add new top level functions, e.g., ``get_minimal_array_module``. We would also need to add either a new protocol patterned off of ``__array_module__`` (e.g., ``__array_module_minimal__``), or could add an optional second argument to ``__array_module__`` (catching errors with ``try``/``except``). A new namespace for implicit dispatch ===================================== Instead of supporting overrides in the main `numpy` namespace with ``__array_function__``, we could create a new opt-in namespace, e.g., ``numpy.api``, with versions of NumPy functions that support dispatching. These overrides would need new opt-in protocols, e.g., ``__array_function_api__`` patterned off of ``__array_function__``. This would resolve the biggest limitations of ``__array_function__`` by being opt-in and would also allow for unambiguously overriding functions like ``asarray``, because ``np.api.asarray`` would always mean "convert an array-like object." But it wouldn't solve all the dispatching needs met by ``__array_module__``, and would leave us with supporting a considerably more complex protocol both for array users and implementors. We could potentially implement such a new namespace *via* the ``__array_module__`` protocol. Certainly some users would find this convenient, because it is slightly less boilerplate. But this would leave users with a confusing choice: when should they use `get_array_module` vs. `np.api.something`. Also, we would have to add and maintain a whole new module, which is considerably more expensive than merely adding a function. Dispatching on both types and arrays instead of only types ========================================================== Instead of supporting dispatch only via unique array types, we could also support dispatch via array objects, e.g., by passing an ``arrays`` argument as part of the ``__array_module__`` protocol. This could potentially be useful for dispatch for arrays with metadata, such provided by Dask and Pint, but would impose costs in terms of type safety and complexity. For example, a library that supports arrays on both CPUs and GPUs might decide on which device to create a new arrays from functions like ``ones`` based on input arguments: .. code:: python class Array: def __array_module__(self, types, arrays): useful_arrays = tuple(a in arrays if isinstance(a, Array)) if not useful_arrays: return NotImplemented prefer_gpu = any(a.prefer_gpu for a in useful_arrays) return ArrayModule(prefer_gpu) class ArrayModule: def __init__(self, prefer_gpu): self.prefer_gpu = prefer_gpu def __getattr__(self, name): import base_module base_func = getattr(base_module, name) return functools.partial(base_func, prefer_gpu=self.prefer_gpu) This might be useful, but it's not clear if we really need it. Pint seems to get along OK without any explicit array creation routines (favoring multiplication by units, e.g., ``np.ones(5) * ureg.m``), and for the most part Dask is also OK with existing ``__array_function__`` style overides (e.g., favoring ``np.ones_like`` over ``np.ones``). Choosing whether to place an array on the CPU or GPU could be solved by `making array creation lazy <https://github.com/google/jax/pull/1668>`_. .. _appendix-design-choices: Appendix: design choices for API overrides ------------------------------------------ There is a large range of possible design choices for overriding NumPy's API. Here we discuss three major axes of the design decision that guided our design for ``__array_module__``. Opt-in vs. opt-out for users ============================ The ``__array_ufunc__`` and ``__array_function__`` protocols provide a mechanism for overriding NumPy functions *within NumPy's existing namespace*. This means that users need to explicitly opt-out if they do not want any overridden behavior, e.g., by casting arrays with ``np.asarray()``. In theory, this approach lowers the barrier for adopting these protocols in user code and libraries, because code that uses the standard NumPy namespace is automatically compatible. But in practice, this hasn't worked out. For example, most well-maintained libraries that use NumPy follow the best practice of casting all inputs with ``np.asarray()``, which they would have to explicitly relax to use ``__array_function__``. Our experience has been that making a library compatible with a new duck array type typically requires at least a small amount of work to accommodate differences in the data model and operations that can be implemented efficiently. These opt-out approaches also considerably complicate backwards compatibility for libraries that adopt these protocols, because by opting in as a library they also opt-in their users, whether they expect it or not. For winning over libraries that have been unable to adopt ``__array_function__``, an opt-in approach seems like a must. Explicit vs. implicit choice of implementation ============================================== Both ``__array_ufunc__`` and ``__array_function__`` have implicit control over dispatching: the dispatched functions are determined via the appropriate protocols in every function call. This generalizes well to handling many different types of objects, as evidenced by its use for implementing arithmetic operators in Python, but it has two downsides: 1. *Speed*: it imposes additional overhead in every function call, because each function call needs to inspect each of its arguments for overrides. This is why arithmetic on builtin Python numbers is slow. 2. *Readability*: it is not longer immediately evident to readers of code what happens when a function is called, because the function's implementation could be overridden by any of its arguments. In contrast, importing a new library (e.g., ``import dask.array as da``) with an API matching NumPy is entirely explicit. There is no overhead from dispatch or ambiguity about which implementation is being used. Explicit and implicit choice of implementations are not mutually exclusive options. Indeed, most implementations of NumPy API overrides via ``__array_function__`` that we are familiar with (namely, dask, CuPy and sparse, but not Pint) also include an explicit way to use their version of NumPy's API by importing a module directly (``dask.array``, ``cupy`` or ``sparse``, respectively). Local vs. non-local vs. global control ====================================== The final design axis is how users control the choice of API: - **Local control**, as exemplified by multiple dispatch and Python protocols for arithmetic, determines which implementation to use either by checking types or calling methods on the direct arguments of a function. - **Non-local control** such as `np.errstate <https://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html >`_ overrides behavior with global-state via function decorators or context-managers. Control is determined hierarchically, via the inner-most context. - **Global control** provides a mechanism for users to set default behavior, either via function calls or configuration files. For example, matplotlib allows setting a global choice of plotting backend. Local control is generally considered a best practice for API design, because control flow is entirely explicit, which makes it the easiest to understand. Non-local and global control are occasionally used, but generally either due to ignorance or a lack of better alternatives. In the case of duck typing for NumPy's public API, we think non-local or global control would be mistakes, mostly because they **don't compose well**. If one library sets/needs one set of overrides and then internally calls a routine that expects another set of overrides, the resulting behavior may be very surprising. Higher order functions are especially problematic, because the context in which functions are evaluated may not be the context in which they are defined. One class of override use cases where we think non-local and global control are appropriate is for choosing a backend system that is guaranteed to have an entirely consistent interface, such as a faster alternative implementation of ``numpy.fft`` on NumPy arrays. However, these are out of scope for the current proposal, which is focused on duck arrays.

6 27

New DTypes: Are scalars a central concept in NumPy or not?
by Sebastian Berg 09 Apr '20

09 Apr '20

Hi all, When we create new datatypes, we have the option to make new choices for the new datatypes [0] (not the existing ones). The question is: Should every NumPy datatype have a scalar associated and should operations like indexing return a scalar or a 0-D array? This is in my opinion a complex, almost philosophical, question, and we do not have to settle anything for a long time. But, if we do not decide a direction before we have many new datatypes the decision will make itself... So happy about any ideas, even if its just a gut feeling :). There are various points. I would like to mostly ignore the technical ones, but I am listing them anyway here: * Scalars are faster (although that can be optimized likely) * Scalars have a lower memory footprint * The current implementation incurs a technical debt in NumPy. (I do not think that is a general issue, though. We could automatically create scalars for each new datatype probably.) Advantages of having no scalars: * No need to keep track of scalars to preserve them in ufuncs, or libraries using `np.asarray`, do they need `np.asarray_or_scalar`? (or decide they return always arrays, although ufuncs may not) * Seems simpler in many ways, you always know the output will be an array if it has to do with NumPy. Advantages of having scalars: * Scalars are immutable and we are used to them from Python. A 0-D array cannot be used as a dictionary key consistently [1]. I.e. without scalars as first class citizen `dict[arr1d[0]]` cannot work, `dict[arr1d[0].item()]` may (if `.item()` is defined, and e.g. `dict[arr1d[0].frozen()]` could make a copy to work. [2] * Object arrays as we have them now make sense, `arr1d[0]` can reasonably return a Python object. I.e. arrays feel more like container if you can take elements out easily. Could go both ways: * Scalar math `scalar = arr1d[0]; scalar += 1` modifies the array without scalars. With scalars `arr1d[0, ...]` clarifies the meaning. (In principle it is good to never use `arr2d[0]` to get a 1D slice, probably more-so if scalars exist.) Note: array-scalars (the current NumPy scalars) are not useful in my opinion [3]. A scalar should not be indexed or have a shape. I do not believe in scalars pretending to be arrays. I personally tend towards liking scalars. If Python was a language where the array (array-programming) concept was ingrained into the language itself, I would lean the other way. But users are used to scalars, and they "put" scalars into arrays. Array objects are in some ways strange in Python, and I feel not having scalars detaches them further. Having scalars, however also means we should preserve them. I feel in principle that is actually fairly straight forward. E.g. for ufuncs: * np.add(scalar, scalar) -> scalar * np.add.reduce(arr, axis=None) -> scalar * np.add.reduce(arr, axis=1) -> array (even if arr is 1d) * np.add.reduce(scalar, axis=()) -> array Of course libraries that do `np.asarray` would/could basically chose to not preserve scalars: Their signature is defined as taking strictly array input. Cheers, Sebastian [0] At best this can be a vision to decide which way they may evolve. [1] E.g. PyTorch uses `hash(tensor) == id(tensor)` which is arguably strange. E.g. Quantity defines hash correctly, but does not fully ensure immutability for 0-D Quantities. Ensuring immutability in a world where "views" are a central concept requires a write-only copy. [2] Arguably `.item()` would always return a scalar, but it would be a second class citizen. (Although if it returns a scalar, at least we already have a scalar implementation.) [3] They are necessary due to technical debt for NumPy datatypes though.

8 14

Proposal: NEP 41 -- First step towards a new Datatype System
by Sebastian Berg 09 Apr '20

09 Apr '20

Hi all, I am pleased to propose NEP 41: First step towards a new Datatype System https://numpy.org/neps/nep-0041-improved-dtype-support.html This NEP motivates the larger restructure of the datatype machinery in NumPy and defines a few fundamental design aspects. The long term user impact will be allowing easier and more rich featured user defined datatypes. As this is a large restructure, the NEP represents only the first steps with some additional information in further NEPs being drafted [1] (this may be helpful to look at depending on the level of detail you are interested in). The NEP itself does not propose to add significant new public API. Instead it proposes to move forward with an incremental internal refactor and lays the foundation for this process. The main user facing change at this time is that datatypes will become classes (e.g. ``type(np.dtype("float64"))`` will be a float64 specific class. For most users, the main impact should be many new datatypes in the long run (see the user impact section). However, for those interested in API design within NumPy or with respect to implementing new datatypes, this and the following NEPs are important decisions in the future roadmap for NumPy. The current full text is reproduced below, although the above link is probably a better way to read it. Cheers Sebastian [1] NEP 40 gives some background information about the current systems and issues with it: https://github.com/numpy/numpy/blob/1248cf7a8765b7b53d883f9e7061173817533aa… and NEP 42 being a first draft of how the new API may look like: https://github.com/numpy/numpy/blob/f07e25cdff3967a19c4cc45c6e1a94a38f53cee… (links to current rendered versions, check https://github.com/numpy/numpy/pull/15505 and https://github.com/numpy/numpy/pull/15507 for updates) ---------------------------------------------------------------------- ================================================= NEP 41 — First step towards a new Datatype System ================================================= :title: Improved Datatype Support :Author: Sebastian Berg :Author: Stéfan van der Walt :Author: Matti Picus :Status: Draft :Type: Standard Track :Created: 2020-02-03 .. note:: This NEP is part of a series of NEPs encompassing first information about the previous dtype implementation and issues with it in NEP 40. NEP 41 (this document) then provides an overview and generic design choices for the refactor. Further NEPs 42 and 43 go into the technical details of the datatype and universal function related internal and external API changes. In some cases it may be necessary to consult the other NEPs for a full picture of the desired changes and why these changes are necessary. Abstract -------- `Datatypes <data-type-objects-dtype>` in NumPy describe how to interpret each element in arrays. NumPy provides ``int``, ``float``, and ``complex`` numerical types, as well as string, datetime, and structured datatype capabilities. The growing Python community, however, has need for more diverse datatypes. Examples are datatypes with unit information attached (such as meters) or categorical datatypes (fixed set of possible values). However, the current NumPy datatype API is too limited to allow the creation of these. This NEP is the first step to enable such growth; it will lead to a simpler development path for new datatypes. In the long run the new datatype system will also support the creation of datatypes directly from Python rather than C. Refactoring the datatype API will improve maintainability and facilitate development of both user-defined external datatypes, as well as new features for existing datatypes internal to NumPy. Motivation and Scope -------------------- .. seealso:: The user impact section includes examples of what kind of new datatypes will be enabled by the proposed changes in the long run. It may thus help to read these section out of order. Motivation ^^^^^^^^^^ One of the main issues with the current API is the definition of typical functions such as addition and multiplication for parametric datatypes (see also NEP 40) which require additional steps to determine the output type. For example when adding two strings of length 4, the result is a string of length 8, which is different from the input. Similarly, a datatype which embeds a physical unit must calculate the new unit information: dividing a distance by a time results in a speed. A related difficulty is that the :ref:`current casting rules <_ufuncs.casting>` -- the conversion between different datatypes -- cannot describe casting for such parametric datatypes implemented outside of NumPy. This additional functionality for supporting parametric datatypes introduces increased complexity within NumPy itself, and furthermore is not available to external user-defined datatypes. In general the concerns of different datatypes are not well well-encapsulated. This burden is exacerbated by the exposure of internal C structures, limiting the addition of new fields (for example to support new sorting methods [new_sort]_). Currently there are many factors which limit the creation of new user-defined datatypes: * Creating casting rules for parametric user-defined dtypes is either impossible or so complex that it has never been attempted. * Type promotion, e.g. the operation deciding that adding float and integer values should return a float value, is very valuable for numeric datatypes but is limited in scope for user-defined and especially parametric datatypes. * Much of the logic (e.g. promotion) is written in single functions instead of being split as methods on the datatype itself. * In the current design datatypes cannot have methods that do not generalize to other datatypes. For example a unit datatype cannot have a ``.to_si()`` method to easily find the datatype which would represent the same values in SI units. The large need to solve these issues has driven the scientific community to create work-arounds in multiple projects implementing physical units as an array-like class instead of a datatype, which would generalize better across multiple array-likes (Dask, pandas, etc.). Already, Pandas has made a push into the same direction with its extension arrays [pandas_extension_arrays]_ and undoubtedly the community would be best served if such new features could be common between NumPy, Pandas, and other projects. Scope ^^^^^ The proposed refactoring of the datatype system is a large undertaking and thus is proposed to be split into various phases, roughly: * Phase I: Restructure and extend the datatype infrastructure (This NEP 41) * Phase II: Incrementally define or rework API (Detailed largely in NEPs 42/43) * Phase III: Growth of NumPy and Scientific Python Ecosystem capabilities. For a more detailed accounting of the various phases, see "Plan to Approach the Full Refactor" in the Implementation section below. This NEP proposes to move ahead with the necessary creation of new dtype subclasses (Phase I), and start working on implementing current functionality. Within the context of this NEP all development will be fully private API or use preliminary underscored names which must be changed in the future. Most of the internal and public API choices are part of a second Phase and will be discussed in more detail in the following NEPs 42 and 43. The initial implementation of this NEP will have little or no effect on users, but provides the necessary ground work for incrementally addressing the full rework. The implementation of this NEP and the following, implied large rework of how datatypes are defined in NumPy is expected to create small incompatibilities (see backward compatibility section). However, a transition requiring large code adaption is not anticipated and not within scope. Specifically, this NEP makes the following design choices which are discussed in more details in the detailed description section: 1. Each datatype will be an instance of a subclass of ``np.dtype``, with most of the datatype-specific logic being implemented as special methods on the class. In the C-API, these correspond to specific slots. In short, for ``f = np.dtype("f8")``, ``isinstance(f, np.dtype)`` will remain true, but ``type(f)`` will be a subclass of ``np.dtype`` rather than just ``np.dtype`` itself. The ``PyArray_ArrFuncs`` which are currently stored as a pointer on the instance (as ``PyArray_Descr->f``), should instead be stored on the class as typically done in Python. In the future these may correspond to python side dunder methods. Storage information such as itemsize and byteorder can differ between different dtype instances (e.g. "S3" vs. "S8") and will remain part of the instance. This means that in the long run the current lowlevel access to dtype methods will be removed (see ``PyArray_ArrFuncs`` in NEP 40). 2. The current NumPy scalars will *not* change, they will not be instances of datatypes. This will also be true for new datatypes, scalars will not be instances of a dtype (although ``isinstance(scalar, dtype)`` may be made to return ``True`` when appropriate). Detailed technical decisions to follow in NEP 42. Further, the public API will be designed in a way that is extensible in the future: 3. All new C-API functions provided to the user will hide implementation details as much as possible. The public API should be an identical, but limited, version of the C-API used for the internal NumPy datatypes. The changes to the datatype system in Phase II must include a large refactor of the UFunc machinery, which will be further defined in NEP 43: 4. To enable all of the desired functionality for new user-defined datatypes, the UFunc machinery will be changed to replace the current dispatching and type resolution system. The old system should be *mostly* supported as a legacy version for some time. Additionally, as a general design principle, the addition of new user-defined datatypes will *not* change the behaviour of programs. For example ``common_dtype(a, b)`` must not be ``c`` unless ``a`` or ``b`` know that ``c`` exists. User Impact ----------- The current ecosystem has very few user-defined datatypes using NumPy, the two most prominent being: ``rational`` and ``quaternion``. These represent fairly simple datatypes which are not strongly impacted by the current limitations. However, we have identified a need for datatypes such as: * bfloat16, used in deep learning * categorical types * physical units (such as meters) * datatypes for tracing/automatic differentiation * high, fixed precision math * specialized integer types such as int2, int24 * new, better datetime representations * extending e.g. integer dtypes to have a sentinel NA value * geometrical objects [pygeos]_ Some of these are partially solved; for example unit capability is provided in ``astropy.units``, ``unyt``, or ``pint``, as `numpy.ndarray` subclasses. Most of these datatypes, however, simply cannot be reasonably defined right now. An advantage of having such datatypes in NumPy is that they should integrate seamlessly with other array or array-like packages such as Pandas, ``xarray`` [xarray_dtype_issue]_, or ``Dask``. The long term user impact of implementing this NEP will be to allow both the growth of the whole ecosystem by having such new datatypes, as well as consolidating implementation of such datatypes within NumPy to achieve better interoperability. Examples ^^^^^^^^ The following examples represent future user-defined datatypes we wish to enable. These datatypes are not part the NEP and choices (e.g. choice of casting rules) are possibilities we wish to enable and do not represent recommendations. Simple Numerical Types """""""""""""""""""""" Mainly used where memory is a consideration, lower-precision numeric types such as :ref:```bfloat16`` <https://en.wikipedia.org/wiki/Bfloat16_floating-point_format>` are common in other computational frameworks. For these types the definitions of things such as ``np.common_type`` and ``np.can_cast`` are some of the most important interfaces. Once they support ``np.common_type``, it is (for the most part) possible to find the correct ufunc loop to call, since most ufuncs -- such as add -- effectively only require ``np.result_type``:: >>> np.add(arr1, arr2).dtype == np.result_type(arr1, arr2) and `~numpy.result_type` is largely identical to `~numpy.common_type`. Fixed, high precision math """""""""""""""""""""""""" Allowing arbitrary precision or higher precision math is important in simulations. For instance ``mpmath`` defines a precision:: >>> import mpmath as mp >>> print(mp.dps) # the current (default) precision 15 NumPy should be able to construct a native, memory-efficient array from a list of ``mpmath.mpf`` floating point objects:: >>> arr_15_dps = np.array(mp.arange(3)) # (mp.arange returns a list) >>> print(arr_15_dps) # Must find the correct precision from the objects: array(['0.0', '1.0', '2.0'], dtype=mpf[dps=15]) We should also be able to specify the desired precision when creating the datatype for an array. Here, we use ``np.dtype[mp.mpf]`` to find the DType class (the notation is not part of this NEP), which is then instantiated with the desired parameter. This could also be written as ``MpfDType`` class:: >>> arr_100_dps = np.array([1, 2, 3], dtype=np.dtype[mp.mpf](dps=100)) >>> print(arr_15_dps + arr_100_dps) array(['0.0', '2.0', '4.0'], dtype=mpf[dps=100]) The ``mpf`` datatype can decide that the result of the operation should be the higher precision one of the two, so uses a precision of 100. Furthermore, we should be able to define casting, for example as in:: >>> np.can_cast(arr_15_dps.dtype, arr_100_dps.dtype, casting="safe") True >>> np.can_cast(arr_100_dps.dtype, arr_15_dps.dtype, casting="safe") False # loses precision >>> np.can_cast(arr_100_dps.dtype, arr_100_dps.dtype, casting="same_kind") True Casting from float is a probably always at least a ``same_kind`` cast, but in general, it is not safe:: >>> np.can_cast(np.float64, np.dtype[mp.mpf](dps=4), casting="safe") False since a float64 has a higer precision than the ``mpf`` datatype with ``dps=4``. Alternatively, we can say that:: >>> np.common_type(np.dtype[mp.mpf](dps=5), np.dtype[mp.mpf](dps=10)) np.dtype[mp.mpf](dps=10) And possibly even:: >>> np.common_type(np.dtype[mp.mpf](dps=5), np.float64) np.dtype[mp.mpf](dps=16) # equivalent precision to float64 (I believe) since ``np.float64`` can be cast to a ``np.dtype[mp.mpf](dps=16)`` safely. Categoricals """""""""""" Categoricals are interesting in that they can have fixed, predefined values, or can be dynamic with the ability to modify categories when necessary. The fixed categories (defined ahead of time) is the most straight forward categorical definition. Categoricals are *hard*, since there are many strategies to implement them, suggesting NumPy should only provide the scaffolding for user-defined categorical types. For instance:: >>> cat = Categorical(["eggs", "spam", "toast"]) >>> breakfast = array(["eggs", "spam", "eggs", "toast"], dtype=cat) could store the array very efficiently, since it knows that there are only 3 categories. Since a categorical in this sense knows almost nothing about the data stored in it, few operations makes, sense, although equality does: >>> breakfast2 = array(["eggs", "eggs", "eggs", "eggs"], dtype=cat) >>> breakfast == breakfast2 array[True, False, True, False]) The categorical datatype could work like a dictionary: no two items names can be equal (checked on dtype creation), so that the equality operation above can be performed very efficiently. If the values define an order, the category labels (internally integers) could be ordered the same way to allow efficient sorting and comparison. Whether or not casting is defined from one categorical with less to one with strictly more values defined, is something that the Categorical datatype would need to decide. Both options should be available. Unit on the Datatype """""""""""""""""""" There are different ways to define Units, depending on how the internal machinery would be organized, one way is to have a single Unit datatype for every existing numerical type. This will be written as ``Unit[float64]``, the unit itself is part of the DType instance ``Unit[float64]("m")`` is a ``float64`` with meters attached:: >>> from astropy import units >>> meters = np.array([1, 2, 3], dtype=np.float64) * units.m # meters >>> print(meters) array([1.0, 2.0, 3.0], dtype=Unit[float64]("m")) Note that units are a bit tricky. It is debatable, whether:: >>> np.array([1.0, 2.0, 3.0], dtype=Unit[float64]("m")) should be valid syntax (coercing the float scalars without a unit to meters). Once the array is created, math will work without any issue:: >>> meters / (2 * unit.seconds) array([0.5, 1.0, 1.5], dtype=Unit[float64]("m/s")) Casting is not valid from one unit to the other, but can be valid between different scales of the same dimensionality (although this may be "unsafe"):: >>> meters.astype(Unit[float64]("s")) TypeError: Cannot cast meters to seconds. >>> meters.astype(Unit[float64]("km")) >>> # Convert to centimeter-gram-second (cgs) units: >>> meters.astype(meters.dtype.to_cgs()) The above notation is somewhat clumsy. Functions could be used instead to convert between units. There may be ways to make these more convenient, but those must be left for future discussions:: >>> units.convert(meters, "km") >>> units.to_cgs(meters) There are some open questions. For example, whether additional methods on the array object could exist to simplify some of the notions, and how these would percolate from the datatype to the ``ndarray``. The interaction with other scalars would likely be defined through:: >>> np.common_type(np.float64, Unit) Unit[np.float64](dimensionless) Ufunc output datatype determination can be more involved than for simple numerical dtypes since there is no "universal" output type:: >>> np.multiply(meters, seconds).dtype != np.result_type(meters, seconds) In fact ``np.result_type(meters, seconds)`` must error without context of the operation being done. This example highlights how the specific ufunc loop (loop with known, specific DTypes as inputs), has to be able to to make certain decisions before the actual calculation can start. Implementation -------------- Plan to Approach the Full Refactor ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To address these issues in NumPy and enable new datatypes, multiple development stages are required: * Phase I: Restructure and extend the datatype infrastructure (This NEP) * Organize Datatypes like normal Python classes [`PR 15508`]_ * Phase II: Incrementally define or rework API * Create a new and easily extensible API for defining new datatypes and related functionality. (NEP 42) * Incrementally define all necessary functionality through the new API (NEP 42): * Defining operations such as ``np.common_type``. * Allowing to define casting between datatypes. * Add functionality necessary to create a numpy array from Python scalars (i.e. ``np.array(...)``). * … * Restructure how universal functions work (NEP 43), in order to: * make it possible to allow a `~numpy.ufunc` such as ``np.add`` to be extended by user-defined datatypes such as Units. * allow efficient lookup for the correct implementation for user-defined datatypes. * enable reuse of existing code. Units should be able to use the normal math loops and add additional logic to determine output type. * Phase III: Growth of NumPy and Scientific Python Ecosystem capabilities: * Cleanup of legacy behaviour where it is considered buggy or undesirable. * Provide a path to define new datatypes from Python. * Assist the community in creating types such as Units or Categoricals * Allow strings to be used in functions such as ``np.equal`` or ``np.add``. * Remove legacy code paths within NumPy to improve long term maintainability This document serves as a basis for phase I and provides the vision and motivation for the full project. Phase I does not introduce any new user-facing features, but is concerned with the necessary conceptual cleanup of the current datatype system. It provides a more "pythonic" datatype Python type object, with a clear class hierarchy. The second phase is the incremental creation of all APIs necessary to define fully featured datatypes and reorganization of the NumPy datatype system. This phase will thus be primarily concerned with defining an, initially preliminary, stable public API. Some of the benefits of a large refactor may only become evident after the full deprecation of the current legacy implementation (i.e. larger code removals). However, these steps are necessary for improvements to many parts of the core NumPy API, and are expected to make the implementation generally easier to understand. The following figure illustrates the proposed design at a high level, and roughly delineates the components of the overall design. Note that this NEP only regards Phase I (shaded area), the rest encompasses Phase II and the design choices are up for discussion, however, it highlights that the DType datatype class is the central, necessary concept: .. image:: _static/nep-0041-mindmap.svg First steps directly related to this NEP ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The required changes necessary to NumPy are large and touch many areas of the code base but many of these changes can be addressed incrementally. To enable an incremental approach we will start by creating a C defined ``PyArray_DTypeMeta`` class with its instances being the ``DType`` classes, subclasses of ``np.dtype``. This is necessary to add the ability of storing custom slots on the DType in C. This ``DTypeMeta`` will be implemented first to then enable incremental restructuring of current code. The addition of ``DType`` will then enable addressing other changes incrementally, some of which may begin before the settling the full internal API: 1. New machinery for array coercion, with the goal of enabling user DTypes with appropriate class methods. 2. The replacement or wrapping of the current casting machinery. 3. Incremental redefinition of the current ``PyArray_ArrFuncs`` slots into DType method slots. At this point, no or only very limited new public API will be added and the internal API is considered to be in flux. Any new public API may be set up give warnings and will have leading underscores to indicate that it is not finalized and can be changed without warning. Backward compatibility ---------------------- While the actual backward compatibility impact of implementing Phase I and II are not yet fully clear, we anticipate, and accept the following changes: * **Python API**: * ``type(np.dtype("f8"))`` will be a subclass of ``np.dtype``, while right now ``type(np.dtype("f8")) is np.dtype``. Code should use ``isinstance`` checks, and in very rare cases may have to be adapted to use it. * **C-API**: * In old versions of NumPy ``PyArray_DescrCheck`` is a macro which uses ``type(dtype) is np.dtype``. When compiling against an old NumPy version, the macro may have to be replaced with the corresponding ``PyObject_IsInstance`` call. (If this is a problem, we could backport fixing the macro) * The UFunc machinery changes will break *limited* parts of the current implementation. Replacing e.g. the default ``TypeResolver`` is expected to remain supported for a time, although optimized masked inner loop iteration (which is not even used *within* NumPy) will no longer be supported. * All functions currently defined on the dtypes, such as ``PyArray_Descr->f->nonzero``, will be defined and accessed differently. This means that in the long run lowlevel access code will have to be changed to use the new API. Such changes are expected to be necessary in very few project. * **dtype implementors (C-API)**: * The array which is currently provided to some functions (such as cast functions), will no longer be provided. For example ``PyArray_Descr->f->nonzero`` or ``PyArray_Descr->f->copyswapn``, may instead receive a dummy array object with only some fields (mainly the dtype), being valid. At least in some code paths, a similar mechanism is already used. * The ``scalarkind`` slot and registration of scalar casting will be removed/ignored without replacement. It currently allows partial value-based casting. The ``PyArray_ScalarKind`` function will continue to work for builtin types, but will not be used internally and be deprecated. * Currently user dtypes are defined as instances of ``np.dtype``. The creation works by the user providing a prototype instance. NumPy will need to modify at least the type during registration. This has no effect for either ``rational`` or ``quaternion`` and mutation of the structure seems unlikely after registration. Since there is a fairly large API surface concerning datatypes, further changes or the limitation certain function to currently existing datatypes is likely to occur. For example functions which use the type number as input should be replaced with functions taking DType classes instead. Although public, large parts of this C-API seem to be used rarely, possibly never, by downstream projects. Detailed Description -------------------- This section details the design decisions covered by this NEP. The subsections correspond to the list of design choices presented in the Scope section. Datatypes as Python Classes (1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The current NumPy datatypes are not full scale python classes. They are instead (prototype) instances of a single ``np.dtype`` class. Changing this means that any special handling, e.g. for ``datetime`` can be moved to the Datetime DType class instead, away from monolithic general code (e.g. current ``PyArray_AdjustFlexibleDType``). The main consequence of this change with respect to the API is that special methods move from the dtype instances to methods on the new DType class. This is the typical design pattern used in Python. Organizing these methods and information in a more Pythonic way provides a solid foundation for refining and extending the API in the future. The current API cannot be extended due to how it is exposed publically. This means for example that the methods currently stored in ``PyArray_ArrFuncs`` on each datatype (see NEP 40) will be defined differently in the future and deprecated in the long run. The most prominent visible side effect of this will be that ``type(np.dtype(np.float64))`` will not be ``np.dtype`` anymore. Instead it will be a subclass of ``np.dtype`` meaning that ``isinstance(np.dtype(np.float64), np.dtype)`` will remain true. This will also add the ability to use ``isinstance(dtype, np.dtype[float64])`` thus removing the need to use ``dtype.kind``, ``dtype.char``, or ``dtype.type`` to do this check. With the design decision of DTypes as full-scale Python classes, the question of subclassing arises. Inheritance, however, appears problematic and a complexity best avoided (at least initially) for container datatypes. Further, subclasses may be more interesting for interoperability for example with GPU backends (CuPy) storing additional methods related to the GPU rather than as a mechanism to define new datatypes. A class hierarchy does provides value, this may be achieved by allowing the creation of *abstract* datatypes. An example for an abstract datatype would be the datatype equivalent of ``np.floating``, representing any floating point number. These can serve the same purpose as Python's abstract base classes. Scalars should not be instances of the datatypes (2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For simple datatypes such as ``float64`` (see also below), it seems tempting that the instance of a ``np.dtype("float64")`` can be the scalar. This idea may be even more appealing due to the fact that scalars, rather than datatypes, currently define a useful type hierarchy. However, we have specifically decided against this for a number of reasons. First, the new datatypes described herein would be instances of DType classes. Making these instances themselves classes, while possible, adds additional complexity that users need to understand. It would also mean that scalars must have storage information (such as byteorder) which is generally unnecessary and currently is not used. Second, while the simple NumPy scalars such as ``float64`` may be such instances, it should be possible to create datatypes for Python objects without enforcing NumPy as a dependency. However, Python objects that do not depend on NumPy cannot be instances of a NumPy DType. Third, there is a mismatch between the methods and attributes which are useful for scalars and datatypes. For instance ``to_float()`` makes sense for a scalar but not for a datatype and ``newbyteorder`` is not useful on a scalar (or has a different meaning). Overall, it seem rather than reducing the complexity, i.e. by merging the two distinct type hierarchies, making scalars instances of DTypes would increase the complexity of both the design and implementation. A possible future path may be to instead simplify the current NumPy scalars to be much simpler objects which largely derive their behaviour from the datatypes. C-API for creating new Datatypes (3) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The current C-API with which users can create new datatypes is limited in scope, and requires use of "private" structures. This means the API is not extensible: no new members can be added to the structure without losing binary compatibility. This has already limited the inclusion of new sorting methods into NumPy [new_sort]_. The new version shall thus replace the current ``PyArray_ArrFuncs`` structure used to define new datatypes. Datatypes that currently exist and are defined using these slots will be supported during a deprecation period. The most likely solution is to hide the implementation from the user and thus make it extensible in the future is to model the API after Python's stable API [PEP-384]_: .. code-block:: C static struct PyArrayMethodDef slots[] = { {NPY_dt_method, method_implementation}, ..., {0, NULL} } typedef struct{ PyTypeObject *typeobj; /* type of python scalar */ ...; PyType_Slot *slots; } PyArrayDTypeMeta_Spec; PyObject* PyArray_InitDTypeMetaFromSpec( PyArray_DTypeMeta *user_dtype, PyArrayDTypeMeta_Spec *dtype_spec); The C-side slots should be designed to mirror Python side methods such as ``dtype.__dtype_method__``, although the exposure to Python is a later step in the implementation to reduce the complexity of the initial implementation. C-API Changes to the UFunc Machinery (4) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Proposed changes to the UFunc machinery will be part of NEP 43. However, the following changes will be necessary (see NEP 40 for a detailed description of the current implementation and its issues): * The current UFunc type resolution must be adapted to allow better control for user-defined dtypes as well as resolve current inconsistencies. * The inner-loop used in UFuncs must be expanded to include a return value. Further, error reporting must be improved, and passing in dtype-specific information enabled. This requires the modification of the inner-loop function signature and addition of new hooks called before and after the inner-loop is used. An important goal for any changes to the universal functions will be to allow the reuse of existing loops. It should be easy for a new units datatype to fall back to existing math functions after handling the unit related computations. Discussion ---------- See NEP 40 for a list of previous meetings and discussions. References ---------- .. [pandas_extension_arrays] https://pandas.pydata.org/pandas-docs/stable/development/extending.html#ext… .. _xarray_dtype_issue: https://github.com/pydata/xarray/issues/1262 .. [pygeos] https://github.com/caspervdw/pygeos .. [new_sort] https://github.com/numpy/numpy/pull/12945 .. [PEP-384] https://www.python.org/dev/peps/pep-0384/ .. [PR 15508] https://github.com/numpy/numpy/pull/15508 Copyright --------- This document has been placed in the public domain. Acknowledgments --------------- The effort to create new datatypes for NumPy has been discussed for several years in many different contexts and settings, making it impossible to list everyone involved. We would like to thank especially Stephan Hoyer, Nathaniel Smith, and Eric Wieser for repeated in-depth discussion about datatype design. We are very grateful for the community input in reviewing and revising this NEP and would like to thank especially Ross Barnowski and Ralf Gommers.

7 15

ANN: SfePy 2020.1
by Robert Cimrman 01 Apr '20

01 Apr '20

I am pleased to announce release 2020.1 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - reading/writing of additional mesh formats by using meshio [2] - Python 3 only from now on For full release notes see [1]. Cheers, Robert Cimrman [1] http://docs.sfepy.org/doc/release_notes.html#id1 [2] https://github.com/nschloe/meshio --- Contributors to this release in alphabetical order: Robert Cimrman Lubos Kejzlar Vladimir Lukes

1 0

(Two hours later!) NumPy Community Meeting Wednesday, March 18
by Sebastian Berg 01 Apr '20

01 Apr '20

Hi all, There will be a NumPy Community meeting Wednesday April 1 at 1(!)pm Pacific Time. Everyone is invited and encouraged to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian PS: Remember that we will try doing the meeting two hours later than previously to accommodate different timezones. I realize that this will be very late for those in Europe, but it seemed still possible in the poll (and with Matti was suggesting it), so it may be be the best time. I see this as a trial balloon for now, please complain here or to me if the time is bad for you and we can move it back.

1 0

Deprecating ndarray.tostring()
by Eric Wieser 31 Mar '20

31 Mar '20

Hi all, Just a heads up that in https://github.com/numpy/numpy/pull/15867 I plan to deprecate ndarray.tostring(), which is just a confusing way to spell ndarray.tobytes(). This function has been documented as a compatibility alias since NumPy 1.9, but never emitted a warning upon use. Given array.array.tostring() has been warning as far back as Python 3.1, it seems like we should follow suit. In order to reduce the impact of such a deprecation, I’ve filed the necessary scipy PR: https://github.com/scipy/scipy/pull/11755. It’s unlikely we’ll remove this function entirely any time soon, but the act of deprecating it may cause a few failing CI runs in projects where warnings are turned into errors. Eric

3 2

Put type annotations in NumPy proper?
by Stephan Hoyer 30 Mar '20

30 Mar '20

When we started numpy-stubs [1] a few years ago, putting type annotations in NumPy itself seemed premature. We still supported Python 2, which meant that we would need to use awkward comments for type annotations. Over the past few years, using type annotations has become increasingly popular, even in the scientific Python stack. For example, off-hand I know that at least SciPy, pandas and xarray have at least part of their APIs type annotated. Even without annotations for shapes or dtypes, it would be valuable to have near complete annotations for NumPy, the project at the bottom of the scientific stack. Unfortunately, numpy-stubs never really took off. I can think of a few reasons for that: 1. Missing high level guidance on how to write type annotations, particularly for how (or if) to annotate particularly dynamic parts of NumPy (e.g., consider __array_function__), and whether we should prioritize strictness or faithfulness [2]. 2. We didn't have a good experience for new contributors. Due to the relatively low level of interest in the project, when a contributor would occasionally drop in, I often didn't even notice their PR for a few weeks. 3. Developing type annotations separately from the main codebase makes them a little harder to keep in sync. This means that type annotations couldn't serve their typical purpose of self-documenting code. Part of this may be necessary for NumPy (due to our use of C extensions), but large parts of NumPy's user facing APIs are written in Python. We no longer support Python 2, so at least we no longer need to worry about putting annotations in comments. We eventually could probably use a formal NEP (or several) on how we want to use type annotations in NumPy, but I think a good first step would be to think about how to start moving the annotations from numpy-stubs into numpy proper. Any thoughts? Anyone interested in taking the lead on this? Cheers, Stephan [1] https://github.com/numpy/numpy-stubs [2] https://github.com/numpy/numpy-stubs/issues/12

7 6

NumPy-Discussion March 2020