Mailman 3 February 2020 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How … [View More]

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program … [View More]makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil [View Less]

10 10

help translating into Russian
by Inessa Pawson May 19, 2020

May 19, 2020

Our collaboration with the students and faculty from the Master’s program in Survey Methodology at the University of Michigan and the University of Maryland is underway. We are looking for a volunteer to translate the survey questionnaire into Russian. If you are available, or you know someone who would be interested to help, please leave a comment here: https://github.com/numpy/numpy-surveys/issues/1. -- Every good wish, *Inessa Pawson * Executive Director Albus Code

1 1

NEP 37: A dispatch protocol for NumPy-like modules
by Stephan Hoyer April 12, 2020

April 12, 2020

I am pleased to present a new NumPy Enhancement Proposal for discussion: "NEP-37: A dispatch protocol for NumPy-like modules." Feedback would be very welcome! The full text follows. The rendered proposal can also be found online at https://numpy.org/neps/nep-0037-array-module.html Best, Stephan Hoyer =================================================== NEP 37 — A dispatch protocol for NumPy-like modules =================================================== :Author: Stephan Hoyer <shoyer(a)… [View More]google.com> :Author: Hameer Abbasi :Author: Sebastian Berg :Status: Draft :Type: Standards Track :Created: 2019-12-29 Abstract -------- NEP-18's ``__array_function__`` has been a mixed success. Some projects (e.g., dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. Others (e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose a new protocol, ``__array_module__``, that we expect could eventually subsume most use-cases for ``__array_function__``. The protocol requires explicit adoption by both users and library authors, which ensures backwards compatibility, and is also significantly simpler than ``__array_function__``, both of which we expect will make it easier to adopt. Why ``__array_function__`` hasn't been enough --------------------------------------------- There are two broad ways in which NEP-18 has fallen short of its goals: 1. **Maintainability concerns**. `__array_function__` has significant implications for libraries that use it: - Projects like `PyTorch <https://github.com/pytorch/pytorch/issues/22402>`_, `JAX <https://github.com/google/jax/issues/1565>`_ and even `scipy.sparse <https://github.com/scipy/scipy/issues/10362>`_ have been reluctant to implement `__array_function__` in part because they are concerned about **breaking existing code**: users expect NumPy functions like ``np.concatenate`` to return NumPy arrays. This is a fundamental limitation of the ``__array_function__`` design, which we chose to allow overriding the existing ``numpy`` namespace. - ``__array_function__`` currently requires an "all or nothing" approach to implementing NumPy's API. There is no good pathway for **incremental adoption**, which is particularly problematic for established projects for which adopting ``__array_function__`` would result in breaking changes. - It is no longer possible to use **aliases to NumPy functions** within modules that support overrides. For example, both CuPy and JAX set ``result_type = np.result_type``. - Implementing **fall-back mechanisms** for unimplemented NumPy functions by using NumPy's implementation is hard to get right (but see the `version from dask <https://github.com/dask/dask/pull/5043>`_), because ``__array_function__`` does not present a consistent interface. Converting all arguments of array type requires recursing into generic arguments of the form ``*args, **kwargs``. 2. **Limitations on what can be overridden.** ``__array_function__`` has some important gaps, most notably array creation and coercion functions: - **Array creation** routines (e.g., ``np.arange`` and those in ``np.random``) need some other mechanism for indicating what type of arrays to create. `NEP 36 <https://github.com/numpy/numpy/pull/14715>`_ proposed adding optional ``like=`` arguments to functions without existing array arguments. However, we still lack any mechanism to override methods on objects, such as those needed by ``np.random.RandomState``. - **Array conversion** can't reuse the existing coercion functions like ``np.asarray``, because ``np.asarray`` sometimes means "convert to an exact ``np.ndarray``" and other times means "convert to something _like_ a NumPy array." This led to the `NEP 30 <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_ proposal for a separate ``np.duckarray`` function, but this still does not resolve how to cast one duck array into a type matching another duck array. ``get_array_module`` and the ``__array_module__`` protocol ---------------------------------------------------------- We propose a new user-facing mechanism for dispatching to a duck-array implementation, ``numpy.get_array_module``. ``get_array_module`` performs the same type resolution as ``__array_function__`` and returns a module with an API promised to match the standard interface of ``numpy`` that can implement operations on all provided array types. The protocol itself is both simpler and more powerful than ``__array_function__``, because it doesn't need to worry about actually implementing functions. We believe it resolves most of the maintainability and functionality limitations of ``__array_function__``. The new protocol is opt-in, explicit and with local control; see :ref:`appendix-design-choices` for discussion on the importance of these design features. The array module contract ========================= Modules returned by ``get_array_module``/``__array_module__`` should make a best effort to implement NumPy's core functionality on new array types(s). Unimplemented functionality should simply be omitted (e.g., accessing an unimplemented function should raise ``AttributeError``). In the future, we anticipate codifying a protocol for requesting restricted subsets of ``numpy``; see :ref:`requesting-restricted-subsets` for more details. How to use ``get_array_module`` =============================== Code that wants to support generic duck arrays should explicitly call ``get_array_module`` to determine an appropriate array module from which to call functions, rather than using the ``numpy`` namespace directly. For example: .. code:: python # calls the appropriate version of np.something for x and y module = np.get_array_module(x, y) module.something(x, y) Both array creation and array conversion are supported, because dispatching is handled by ``get_array_module`` rather than via the types of function arguments. For example, to use random number generation functions or methods, we can simply pull out the appropriate submodule: .. code:: python def duckarray_add_random(array): module = np.get_array_module(array) noise = module.random.randn(*array.shape) return array + noise We can also write the duck-array ``stack`` function from `NEP 30 <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_, without the need for a new ``np.duckarray`` function: .. code:: python def duckarray_stack(arrays): module = np.get_array_module(*arrays) arrays = [module.asarray(arr) for arr in arrays] shapes = {arr.shape for arr in arrays} if len(shapes) != 1: raise ValueError('all input arrays must have the same shape') expanded_arrays = [arr[module.newaxis, ...] for arr in arrays] return module.concatenate(expanded_arrays, axis=0) By default, ``get_array_module`` will return the ``numpy`` module if no arguments are arrays. This fall-back can be explicitly controlled by providing the ``module`` keyword-only argument. It is also possible to indicate that an exception should be raised instead of returning a default array module by setting ``module=None``. How to implement ``__array_module__`` ===================================== Libraries implementing a duck array type that want to support ``get_array_module`` need to implement the corresponding protocol, ``__array_module__``. This new protocol is based on Python's dispatch protocol for arithmetic, and is essentially a simpler version of ``__array_function__``. Only one argument is passed into ``__array_module__``, a Python collection of unique array types passed into ``get_array_module``, i.e., all arguments with an ``__array_module__`` attribute. The special method should either return an namespace with an API matching ``numpy``, or ``NotImplemented``, indicating that it does not know how to handle the operation: .. code:: python class MyArray: def __array_module__(self, types): if not all(issubclass(t, MyArray) for t in types): return NotImplemented return my_array_module Returning custom objects from ``__array_module__`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``my_array_module`` will typically, but need not always, be a Python module. Returning a custom objects (e.g., with functions implemented via ``__getattr__``) may be useful for some advanced use cases. For example, custom objects could allow for partial implementations of duck array modules that fall-back to NumPy (although this is not recommended in general because such fall-back behavior can be error prone): .. code:: python class MyArray: def __array_module__(self, types): if all(issubclass(t, MyArray) for t in types): return ArrayModule() else: return NotImplemented class ArrayModule: def __getattr__(self, name): import base_module return getattr(base_module, name, getattr(numpy, name)) Subclassing from ``numpy.ndarray`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the same guidance about well-defined type casting hierarchies from NEP-18 still applies. ``numpy.ndarray`` itself contains a matching implementation of ``__array_module__``, which is convenient for subclasses: .. code:: python class ndarray: def __array_module__(self, types): if all(issubclass(t, ndarray) for t in types): return numpy else: return NotImplemented NumPy's internal machinery ========================== The type resolution rules of ``get_array_module`` follow the same model as Python and NumPy's existing dispatch protocols: subclasses are called before super-classes, and otherwise left to right. ``__array_module__`` is guaranteed to be called only a single time on each unique type. The actual implementation of `get_array_module` will be in C, but should be equivalent to this Python code: .. code:: python def get_array_module(*arrays, default=numpy): implementing_arrays, types = _implementing_arrays_and_types(arrays) if not implementing_arrays and default is not None: return default for array in implementing_arrays: module = array.__array_module__(types) if module is not NotImplemented: return module raise TypeError("no common array module found") def _implementing_arrays_and_types(relevant_arrays): types = [] implementing_arrays = [] for array in relevant_arrays: t = type(array) if t not in types and hasattr(t, '__array_module__'): types.append(t) # Subclasses before superclasses, otherwise left to right index = len(implementing_arrays) for i, old_array in enumerate(implementing_arrays): if issubclass(t, type(old_array)): index = i break implementing_arrays.insert(index, array) return implementing_arrays, types Relationship with ``__array_ufunc__`` and ``__array_function__`` ---------------------------------------------------------------- These older protocols have distinct use-cases and should remain =============================================================== ``__array_module__`` is intended to resolve limitations of ``__array_function__``, so it is natural to consider whether it could entirely replace ``__array_function__``. This would offer dual benefits: (1) simplifying the user-story about how to override NumPy and (2) removing the slowdown associated with checking for dispatch when calling every NumPy function. However, ``__array_module__`` and ``__array_function__`` are pretty different from a user perspective: it requires explicit calls to ``get_array_function``, rather than simply reusing original ``numpy`` functions. This is probably fine for *libraries* that rely on duck-arrays, but may be frustratingly verbose for interactive use. Some of the dispatching use-cases for ``__array_ufunc__`` are also solved by ``__array_module__``, but not all of them. For example, it is still useful to be able to define non-NumPy ufuncs (e.g., from Numba or SciPy) in a generic way on non-NumPy arrays (e.g., with dask.array). Given their existing adoption and distinct use cases, we don't think it makes sense to remove or deprecate ``__array_function__`` and ``__array_ufunc__`` at this time. Mixin classes to implement ``__array_function__`` and ``__array_ufunc__`` ========================================================================= Despite the user-facing differences, ``__array_module__`` and a module implementing NumPy's API still contain sufficient functionality needed to implement dispatching with the existing duck array protocols. For example, the following mixin classes would provide sensible defaults for these special methods in terms of ``get_array_module`` and ``__array_module__``: .. code:: python class ArrayUfuncFromModuleMixin: def __array_ufunc__(self, ufunc, method, *inputs, **kwargs): arrays = inputs + kwargs.get('out', ()) try: array_module = np.get_array_module(*arrays) except TypeError: return NotImplemented try: # Note this may have false positive matches, if ufunc.__name__ # matches the name of a ufunc defined by NumPy. Unfortunately # there is no way to determine in which module a ufunc was # defined. new_ufunc = getattr(array_module, ufunc.__name__) except AttributeError: return NotImplemented try: callable = getattr(new_ufunc, method) except AttributeError: return NotImplemented return callable(*inputs, **kwargs) class ArrayFunctionFromModuleMixin: def __array_function__(self, func, types, args, kwargs): array_module = self.__array_module__(types) if array_module is NotImplemented: return NotImplemented # Traverse submodules to find the appropriate function modules = func.__module__.split('.') assert modules[0] == 'numpy' for submodule in modules[1:]: module = getattr(module, submodule, None) new_func = getattr(module, func.__name__, None) if new_func is None: return NotImplemented return new_func(*args, **kwargs) To make it easier to write duck arrays, we could also add these mixin classes into ``numpy.lib.mixins`` (but the examples above may suffice). Alternatives considered ----------------------- Naming ====== We like the name ``__array_module__`` because it mirrors the existing ``__array_function__`` and ``__array_ufunc__`` protocols. Another reasonable choice could be ``__array_namespace__``. It is less clear what the NumPy function that calls this protocol should be called (``get_array_module`` in this proposal). Some possible alternatives: ``array_module``, ``common_array_module``, ``resolve_array_module``, ``get_namespace``, ``get_numpy``, ``get_numpylike_module``, ``get_duck_array_module``. .. _requesting-restricted-subsets: Requesting restricted subsets of NumPy's API ============================================ Over time, NumPy has accumulated a very large API surface, with over 600 attributes in the top level ``numpy`` module alone. It is unlikely that any duck array library could or would want to implement all of these functions and classes, because the frequently used subset of NumPy is much smaller. We think it would be useful exercise to define "minimal" subset(s) of NumPy's API, omitting rarely used or non-recommended functionality. For example, minimal NumPy might include ``stack``, but not the other stacking functions ``column_stack``, ``dstack``, ``hstack`` and ``vstack``. This could clearly indicate to duck array authors and users want functionality is core and what functionality they can skip. Support for requesting a restricted subset of NumPy's API would be a natural feature to include in ``get_array_function`` and ``__array_module__``, e.g., .. code:: python # array_module is only guaranteed to contain "minimal" NumPy array_module = np.get_array_module(*arrays, request='minimal') To facilitate testing with NumPy and use with any valid duck array library, NumPy itself would return restricted versions of the ``numpy`` module when ``get_array_module`` is called only on NumPy arrays. Omitted functions would simply not exist. Unfortunately, we have not yet figured out what these restricted subsets should be, so it doesn't make sense to do this yet. When/if we do, we could either add new keyword arguments to ``get_array_module`` or add new top level functions, e.g., ``get_minimal_array_module``. We would also need to add either a new protocol patterned off of ``__array_module__`` (e.g., ``__array_module_minimal__``), or could add an optional second argument to ``__array_module__`` (catching errors with ``try``/``except``). A new namespace for implicit dispatch ===================================== Instead of supporting overrides in the main `numpy` namespace with ``__array_function__``, we could create a new opt-in namespace, e.g., ``numpy.api``, with versions of NumPy functions that support dispatching. These overrides would need new opt-in protocols, e.g., ``__array_function_api__`` patterned off of ``__array_function__``. This would resolve the biggest limitations of ``__array_function__`` by being opt-in and would also allow for unambiguously overriding functions like ``asarray``, because ``np.api.asarray`` would always mean "convert an array-like object." But it wouldn't solve all the dispatching needs met by ``__array_module__``, and would leave us with supporting a considerably more complex protocol both for array users and implementors. We could potentially implement such a new namespace *via* the ``__array_module__`` protocol. Certainly some users would find this convenient, because it is slightly less boilerplate. But this would leave users with a confusing choice: when should they use `get_array_module` vs. `np.api.something`. Also, we would have to add and maintain a whole new module, which is considerably more expensive than merely adding a function. Dispatching on both types and arrays instead of only types ========================================================== Instead of supporting dispatch only via unique array types, we could also support dispatch via array objects, e.g., by passing an ``arrays`` argument as part of the ``__array_module__`` protocol. This could potentially be useful for dispatch for arrays with metadata, such provided by Dask and Pint, but would impose costs in terms of type safety and complexity. For example, a library that supports arrays on both CPUs and GPUs might decide on which device to create a new arrays from functions like ``ones`` based on input arguments: .. code:: python class Array: def __array_module__(self, types, arrays): useful_arrays = tuple(a in arrays if isinstance(a, Array)) if not useful_arrays: return NotImplemented prefer_gpu = any(a.prefer_gpu for a in useful_arrays) return ArrayModule(prefer_gpu) class ArrayModule: def __init__(self, prefer_gpu): self.prefer_gpu = prefer_gpu def __getattr__(self, name): import base_module base_func = getattr(base_module, name) return functools.partial(base_func, prefer_gpu=self.prefer_gpu) This might be useful, but it's not clear if we really need it. Pint seems to get along OK without any explicit array creation routines (favoring multiplication by units, e.g., ``np.ones(5) * ureg.m``), and for the most part Dask is also OK with existing ``__array_function__`` style overides (e.g., favoring ``np.ones_like`` over ``np.ones``). Choosing whether to place an array on the CPU or GPU could be solved by `making array creation lazy <https://github.com/google/jax/pull/1668>`_. .. _appendix-design-choices: Appendix: design choices for API overrides ------------------------------------------ There is a large range of possible design choices for overriding NumPy's API. Here we discuss three major axes of the design decision that guided our design for ``__array_module__``. Opt-in vs. opt-out for users ============================ The ``__array_ufunc__`` and ``__array_function__`` protocols provide a mechanism for overriding NumPy functions *within NumPy's existing namespace*. This means that users need to explicitly opt-out if they do not want any overridden behavior, e.g., by casting arrays with ``np.asarray()``. In theory, this approach lowers the barrier for adopting these protocols in user code and libraries, because code that uses the standard NumPy namespace is automatically compatible. But in practice, this hasn't worked out. For example, most well-maintained libraries that use NumPy follow the best practice of casting all inputs with ``np.asarray()``, which they would have to explicitly relax to use ``__array_function__``. Our experience has been that making a library compatible with a new duck array type typically requires at least a small amount of work to accommodate differences in the data model and operations that can be implemented efficiently. These opt-out approaches also considerably complicate backwards compatibility for libraries that adopt these protocols, because by opting in as a library they also opt-in their users, whether they expect it or not. For winning over libraries that have been unable to adopt ``__array_function__``, an opt-in approach seems like a must. Explicit vs. implicit choice of implementation ============================================== Both ``__array_ufunc__`` and ``__array_function__`` have implicit control over dispatching: the dispatched functions are determined via the appropriate protocols in every function call. This generalizes well to handling many different types of objects, as evidenced by its use for implementing arithmetic operators in Python, but it has two downsides: 1. *Speed*: it imposes additional overhead in every function call, because each function call needs to inspect each of its arguments for overrides. This is why arithmetic on builtin Python numbers is slow. 2. *Readability*: it is not longer immediately evident to readers of code what happens when a function is called, because the function's implementation could be overridden by any of its arguments. In contrast, importing a new library (e.g., ``import dask.array as da``) with an API matching NumPy is entirely explicit. There is no overhead from dispatch or ambiguity about which implementation is being used. Explicit and implicit choice of implementations are not mutually exclusive options. Indeed, most implementations of NumPy API overrides via ``__array_function__`` that we are familiar with (namely, dask, CuPy and sparse, but not Pint) also include an explicit way to use their version of NumPy's API by importing a module directly (``dask.array``, ``cupy`` or ``sparse``, respectively). Local vs. non-local vs. global control ====================================== The final design axis is how users control the choice of API: - **Local control**, as exemplified by multiple dispatch and Python protocols for arithmetic, determines which implementation to use either by checking types or calling methods on the direct arguments of a function. - **Non-local control** such as `np.errstate <https://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html >`_ overrides behavior with global-state via function decorators or context-managers. Control is determined hierarchically, via the inner-most context. - **Global control** provides a mechanism for users to set default behavior, either via function calls or configuration files. For example, matplotlib allows setting a global choice of plotting backend. Local control is generally considered a best practice for API design, because control flow is entirely explicit, which makes it the easiest to understand. Non-local and global control are occasionally used, but generally either due to ignorance or a lack of better alternatives. In the case of duck typing for NumPy's public API, we think non-local or global control would be mistakes, mostly because they **don't compose well**. If one library sets/needs one set of overrides and then internally calls a routine that expects another set of overrides, the resulting behavior may be very surprising. Higher order functions are especially problematic, because the context in which functions are evaluated may not be the context in which they are defined. One class of override use cases where we think non-local and global control are appropriate is for choosing a backend system that is guaranteed to have an entirely consistent interface, such as a faster alternative implementation of ``numpy.fft`` on NumPy arrays. However, these are out of scope for the current proposal, which is focused on duck arrays. [View Less]

6 27

New DTypes: Are scalars a central concept in NumPy or not?
by Sebastian Berg April 9, 2020

April 9, 2020

Hi all, When we create new datatypes, we have the option to make new choices for the new datatypes [0] (not the existing ones). The question is: Should every NumPy datatype have a scalar associated and should operations like indexing return a scalar or a 0-D array? This is in my opinion a complex, almost philosophical, question, and we do not have to settle anything for a long time. But, if we do not decide a direction before we have many new datatypes the decision will make itself... So … [View More]happy about any ideas, even if its just a gut feeling :). There are various points. I would like to mostly ignore the technical ones, but I am listing them anyway here: * Scalars are faster (although that can be optimized likely) * Scalars have a lower memory footprint * The current implementation incurs a technical debt in NumPy. (I do not think that is a general issue, though. We could automatically create scalars for each new datatype probably.) Advantages of having no scalars: * No need to keep track of scalars to preserve them in ufuncs, or libraries using `np.asarray`, do they need `np.asarray_or_scalar`? (or decide they return always arrays, although ufuncs may not) * Seems simpler in many ways, you always know the output will be an array if it has to do with NumPy. Advantages of having scalars: * Scalars are immutable and we are used to them from Python. A 0-D array cannot be used as a dictionary key consistently [1]. I.e. without scalars as first class citizen `dict[arr1d[0]]` cannot work, `dict[arr1d[0].item()]` may (if `.item()` is defined, and e.g. `dict[arr1d[0].frozen()]` could make a copy to work. [2] * Object arrays as we have them now make sense, `arr1d[0]` can reasonably return a Python object. I.e. arrays feel more like container if you can take elements out easily. Could go both ways: * Scalar math `scalar = arr1d[0]; scalar += 1` modifies the array without scalars. With scalars `arr1d[0, ...]` clarifies the meaning. (In principle it is good to never use `arr2d[0]` to get a 1D slice, probably more-so if scalars exist.) Note: array-scalars (the current NumPy scalars) are not useful in my opinion [3]. A scalar should not be indexed or have a shape. I do not believe in scalars pretending to be arrays. I personally tend towards liking scalars. If Python was a language where the array (array-programming) concept was ingrained into the language itself, I would lean the other way. But users are used to scalars, and they "put" scalars into arrays. Array objects are in some ways strange in Python, and I feel not having scalars detaches them further. Having scalars, however also means we should preserve them. I feel in principle that is actually fairly straight forward. E.g. for ufuncs: * np.add(scalar, scalar) -> scalar * np.add.reduce(arr, axis=None) -> scalar * np.add.reduce(arr, axis=1) -> array (even if arr is 1d) * np.add.reduce(scalar, axis=()) -> array Of course libraries that do `np.asarray` would/could basically chose to not preserve scalars: Their signature is defined as taking strictly array input. Cheers, Sebastian [0] At best this can be a vision to decide which way they may evolve. [1] E.g. PyTorch uses `hash(tensor) == id(tensor)` which is arguably strange. E.g. Quantity defines hash correctly, but does not fully ensure immutability for 0-D Quantities. Ensuring immutability in a world where "views" are a central concept requires a write-only copy. [2] Arguably `.item()` would always return a scalar, but it would be a second class citizen. (Although if it returns a scalar, at least we already have a scalar implementation.) [3] They are necessary due to technical debt for NumPy datatypes though. [View Less]

8 14

Proposal to accept NEP #44: Restructuring the NumPy Documentation
by Melissa Mendonça March 2, 2020

March 2, 2020

Hi all, I am proposing the acceptance of NEP 44 - Restructuring the NumPy Documentation. https://numpy.org/neps/nep-0044-restructuring-numpy-docs.html There were some comments about reorganizing the text to make it clearer, and some rewording regarding Reference Guides, developer "under-the-hoods" documentation and the possible mechanism for community contributions in the form of new tutorials or how-tos. Overall I believe we have answered all comments and there were no other points of … [View More]

3 3

[ANN] pvlib participating in GSoC
by Dr. Mark Alexander Mikofski PhD Feb. 27, 2020

Feb. 27, 2020

Excited to announce that pvlib python <https://pvlib-python.readthedocs.io/en/latest/> is participating in its first ever Google Summer of Code GSoC under the NumFOCUS umbrella. If you are a student interested in modeling renewable solar energy please apply: https://summerofcode.withgoogle.com/organizations/4727917315096576/ For project ideas, take a look at the pvlib github wiki: https://summerofcode.withgoogle.com/organizations/4727917315096576/ and reach out on our Google … [View More]

1 1

Output type of round is inconsistent with python built-in
by Hameer Abbasi Feb. 27, 2020

Feb. 27, 2020

Hello, Currently, the built-in Python round (which is different from np.round) when called on a np.float64 returns a np.float64, due to its __round__ method. A congruous statement is true for np.float32. However, since Python 3, the default behavior of round is to return a Python int when it operates on a Python float. This is a mismatch according to the Liskov Substitution Principle<https://en.wikipedia.org/wiki/Liskov_substitution_principle>, as both these types subclass Python’s float.… [View More]

4 16

Re: [Numpy-discussion] Output type of round is inconsistent with python built-in
by Stefano Miccoli Feb. 27, 2020

Feb. 27, 2020

There several mixed issues here. 1. PEP 3141 <https://www.python.org/dev/peps/pep-3141/> compliance. Numpy scalars are `numbers.Real` instances, and have to respect the `__round__` semantics defined by PEP 3141: def __round__(self, ndigits:Integral=None): """Rounds self to ndigits decimal places, defaulting to 0. If ndigits is omitted or None, returns an Integral, otherwise returns a Real, preferably of the same type as self. Types may choose which … [View More]

1 0

help translating into Hindi
by Inessa Pawson Feb. 27, 2020

Feb. 27, 2020

Our collaboration with the students and faculty from the Master’s program in Survey Methodology at the University of Michigan and the University of Maryland at College Park is underway. We are looking for a volunteer to translate the survey questionnaire into Hindi. If you are available, or you know someone who would be interested to help, please leave a comment here: https://github.com/numpy/numpy-surveys/issues/1. -- Every good wish, *Inessa Pawson * Executive Director Albus Code

1 0

NumPy Development Meeting - Triage Focus
by Sebastian Berg Feb. 25, 2020

Feb. 25, 2020

Hi all, Our bi-weekly triage-focused NumPy development meeting is tomorrow (Wednesday, Februrary 26) at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized or simply discussed briefly. Just comment on it so we can label it, or add your PR/issue to this weeks topics for discussion. Best regards, Sebastian

1 0

Re: [Numpy-discussion] New DTypes: Are scalars a central concept in NumPy or not?
by Stefano Miccoli Feb. 25, 2020

Feb. 25, 2020

The fact that `isinstance(np.float64(1), float)` raises the problem that the current implementation of np.float64 scalars breaks the Liskov substitution principle: `sequence_or_array[round(x)]` works if `x` is a float, but breaks down if x is a np.float64. See https://github.com/numpy/numpy/issues/11810, where the issue is discussed in the broader setting of the semantics of `np.round` vs. python3 `round`. I do not have a strong opinion here, except that if np.float64’s are within the python … [View More]

1 0

Suggestion: prevent silent downcast in np.full_like
by A T Feb. 25, 2020

Feb. 25, 2020

Hello, This is my first activity on this mailing list, please let me know if I am doing anything improperly. I recently opened issue #15635 <https://github.com/numpy/numpy/issues/15635>and it was suggested to me that it could be worth discussing here. Here is a summarized code example of how unsafe downcasting in np.full_like() resulted in issues in our scientific toolbox: t0 = 20.5 # We're trying to make a constant-valued "ufunc" temperature = lambda x: np.full_like(x, t0) print(… [View More]

2 1

Scikit-learn in the news.
by Charles R Harris Feb. 21, 2020

Feb. 21, 2020

Hi All, Just thought I mention a new paper where scikit-learn was used: A Deep Learning Approach to Antibiotic Discovery <https://www.cell.com/cell/fulltext/S0092-8674(20)30102-1#secsectitle0020>. Congratulations to the scikit-learn team. Chuck

1 0

Tensor Developer Summit
by Stefan van der Walt Feb. 19, 2020

Feb. 19, 2020

Hi all, This has been mentioned on the community calls, but not on the mailing list, so a reminder about the Tensor Developer Summit happening at March in Berkeley: https://xd-con.org/tensor-2020/ We would love to have developers and advanced users of NumPy (or other array libraries with Python interfaces) attend. Registration closes 20 February. Best regards, Stéfan

2 2

numpy_financial functions
by Jordan Feb. 18, 2020

Feb. 18, 2020

I teach finance at IU Bloomington and use the numpy_financial module pretty heavily. I've written a couple of functions for bond math for my own use (duration, convexity, forward rates, etc.). Is there any appetite for expanding numpy_financial beyond the core Excel functions? Is there a point-person for numpy_financial with whom I could correspond about contributing? Thanks, Jordan

2 1

NumPy Community Meeting Wednesday, Feb. 19
by Sebastian Berg Feb. 18, 2020

Feb. 18, 2020

Hi all, There will be a NumPy Community meeting Wednesday February 19 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

recent changes in np.maximum.accumulate ?
by josef.pktd＠gmail.com Feb. 18, 2020

Feb. 18, 2020

I'm trying to track down test failures of statsmodels against recent master dev versions of numpy and scipy. The core computation is the following in one set of tests that fail pvals_corrected_raw = pvals * np.arange(ntests, 0, -1) pvals_corrected = np.maximum.accumulate(pvals_corrected_raw) this numpy version numpy-1.19.0.dev0%2B20200214184618_1f9ab28-cp38-cp38-manylinux2010_x86_64.whl is in the test run with failures (the first time statsmodel master failed) the previous … [View More]

2 2

Re: [Numpy-discussion] recent changes in np.maximum.accumulate ?
by Kevin Sheppard Feb. 18, 2020

Feb. 18, 2020

1 0

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 161, Issue 14
by Inessa Pawson Feb. 14, 2020

Feb. 14, 2020

Documentation is imperative for project sustainability, yet often overlooked. Millions of NumPy stakeholders will benefit from this initiative. Melissa, Mars and Ralf, thank you for taking a lead on this! On Thu, Feb 13, 2020 at 3:05 AM <numpy-discussion-request(a)python.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/… [View More]mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)python.org > > You can reach the person managing the list at > numpy-discussion-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > Today's Topics: > > 1. NEP 44 - Restructuring the NumPy Documentation (Melissa Mendon?a) > > > > ---------- Forwarded message ---------- > From: "Melissa Mendonça" <melissawm(a)gmail.com> > To: numpy-discussion(a)python.org > Cc: > Bcc: > Date: Wed, 12 Feb 2020 10:55:09 -0300 > Subject: [Numpy-discussion] NEP 44 - Restructuring the NumPy Documentation > Hi all, > > Please see the NEP below for a proposal to restructure the documentation > of NumPy. The main goal here is to make the documentation more visible and > organized, and also make contributions easier. > > Comments and feedback are welcome! > > > See https://github.com/numpy/numpy/pull/15554 for details. > > Best, > > Melissa > > ---- > > NEP 44 — Restructuring the NumPy Documentation > > Authors: Ralf Gommers, Melissa Mendonça, Mars Lee > Status: Draft > Type: Process > Created: 2020-02-11 > > Abstract > ====== > > This document proposes a restructuring of the NumPy Documentation, both in > form and content, with the goal of making it more organized and > discoverable for beginners and experienced users. > > Motivation and Scope > ================= > > See [here](numpy.org/devdocs) for the front page of the latest docs. The > organization is quite confusing and illogical (e.g. user and developer docs > are mixed). We propose the following: > > - Reorganizing the docs into the four categories mentioned in [1]; > - Creating dedicated sections for Tutorials and How-Tos, including > orientation on how to create new content; > - Adding an Explanations section for key concepts and techniques that > require deeper descriptions, some of which will be rearranged from the > Reference Guide. > > Usage and Impact > ============== > > The documentation is a fundamental part of any software project, > especially open source projects. In the case of NumPy, many beginners might > feel demotivated by the current structure of the documentation, since it is > difficult to discover what to learn (unless the user has a clear view of > what to look for in the Reference docs, which is not always the case). > > Looking at the results of a “NumPy Tutorial” search on any search engine > also gives an idea of the demand for this kind of content. Having official > high-level documentation written using up-to-date content and techniques > will certainly mean more users (and developers/contributors) are involved > in the NumPy community. > > Backward compatibility > ================== > > The restructuring will effectively demand a complete rewrite of links and > some of the current content. Input from the community will be useful for > identifying key links and pages that should not be broken. > > Detailed description > =============== > > As discussed in the article [1], there are four categories of doc content: > - Tutorials > - How-to guides > - Explanations > - Reference guide > > We propose to use those categories as the ones we use (for writing and > reviewing) whenever we add a new documentation section. > > The reasoning for this is that it is clearer both for > developers/documentation writers and to users where each information should > go, and the scope and tone of each document. For example, if explanations > are mixed with basic tutorials, beginners might be overwhelmed and > alienated. On the other hand, if the reference guide contains basic > how-tos, it might be difficult for experienced users to find the > information they need, quickly. > > Currently, there are many blogs and tutorials on the internet about NumPy > or using NumPy. One of the issues with this is that if users search for > this information and end up in an outdated (unofficial) tutorial before > they find the current official documentation, they end up creating content > that is confusing, especially for beginners. Having a better infrastructure > for the documentation also aims to solve this problem by giving users > high-level, up-to-date official documentation that can be easily updated. > > Status and ideas of each type of doc content > ------------------------------------------------------------ > > * Reference guide > > NumPy has a quite complete reference guide. All functions are documented, > most have examples, and most are cross-linked well with See Also sections. > Further improving the reference guide is incremental work that can be done > (and is being done) by many people. There are, however, many explanations > in the reference guide. These can be moved to a more dedicated Explanations > section on the docs. > > * How-to guides > > NumPy does not have many how-to’s. The subclassing and array ducktyping > section may be an example of a how-to. Others that could be added are: > - Parallelization (controlling BLAS multithreading with threadpoolctl, > using multiprocessing, random number generation, etc.) > - Storing and loading data (.npy/.npz format, text formats, Zarr, HDF5, > Bloscpack, etc.) > - Performance (memory layout, profiling, use with Numba, Cython, or > Pythran) > - Writing generic code that works with NumPy, Dask, CuPy, pydata/sparse, > etc. > > * Explanations > > There is a reasonable amount of content on fundamental NumPy concepts such > as indexing, vectorization, broadcasting, (g)ufuncs, and dtypes. This could > be organized better and clarified to ensure it’s really about explaining > the concepts and not mixed with tutorial or how-to like content. > > There are few explanations about anything other than those fundamental > NumPy concepts. > > Some examples of concepts that could be expanded: > - Copies vs. Views; > - BLAS and other linear algebra libraries; > - Fancy indexing. > > In addition, there are many explanations in the Reference Guide, which > should be moved to this new dedicated Explanations section. > > * Tutorials > > There’s a lot of scope for writing better tutorials. We have a new NumPy > for absolute beginners tutorial [3] (GSoD project of Anne Bonner). In > addition we need a number of tutorials addressing different levels of > experience with Python and NumPy. This could be done using engaging data > sets, ideas or stories. For example, curve fitting with polynomials and > functions in numpy.linalg could be done with the Keeling curve (decades > worth of CO2 concentration in air measurements) rather than with synthetic > random data. > > Ideas for tutorials (these capture the types of things that make sense, > they’re not necessarily the exact topics we propose to implement): > - Conway’s game of life with only NumPy (note: already in Nicolas > Rougier’s book) > - Using masked arrays to deal with missing data in time series measurements > - Using Fourier transforms to analyze the Keeling curve data, and > extrapolate it. > - Geospatial data (e.g. lat/lon/time to create maps for every year via a > stacked array, like gridMet data) > - Using text data and dtypes (e.g. use speeches from different people, > shape (n_speech, n_sentences, n_words)) > > The Preparing to Teach document [2] from the Software Carpentry Instructor > Training materials is a nice summary of how to write effective lesson plans > (and tutorials would be very similar). In addition to adding new tutorials, > we also propose a How to write a tutorial document, which would help users > contribute new high-quality content to the documentation. > > Data sets > ------------- > > Using interesting data in the NumPy docs requires giving all users access > to that data, either inside NumPy or in a separate package. The former is > not the best idea, since it’s hard to do without increasing the size of > NumPy significantly. Even for SciPy there has so far been no consensus on > this (see scipy PR 8707 on adding a new scipy.datasets subpackage). > > So we’ll aim for a new (pure Python) package, named numpy-datasets or > scipy-datasets or something similar. That package can take some lessons > from how, e.g., scikit-learn ships data sets. Small data sets can be > included in the repo, large data sets can be accessed via a downloader > class or function. > > Related Work > =========== > > Some examples of documentation organization in other projects: > - Documentation for Jupyter: https://jupyter.org/documentation > - Documentation for Python: https://docs.python.org/3/ > - Documentation for TensorFlow: https://www.tensorflow.org/learn > > These projects make the intended audience for each part of the > documentation more explicit, as well as previewing some of the content in > each section. > > Implementation > ============ > > Besides rewriting the current documentation to some extent, it would be > ideal to have a technical infrastructure that would allow more > contributions from the community. For example, if Jupyter Notebooks could > be submitted as-is as tutorials or How-Tos, this might create more > contributors and broaden the NumPy community. > > Similarly, if people could download some of the documentation in Notebook > format, this would certainly mean people would use less outdated material > for learning NumPy. > > It would also be interesting if the new structure for the documentation > makes translations easier. > > Currently, the documentation for NumPy can be confusing, especially for > beginners. Our proposal is to reorganize the docs in the following > structure: > > * For users: > - Absolute Beginners Tutorial > - main Tutorials section > - How To’s for common tasks with NumPy > - Reference Guide > - Explanations > - F2Py Guide > - Glossary > > * For developers/contributors: > - Contributor’s Guide > - Building and extending the documentation > - Benchmarking > - NumPy Enhancement Proposals > > * Meta information > - Reporting bugs > - Release Notes > - About NumPy > - License > > References and Footnotes > ==================== > > [1] What nobody tells you about documentation. > https://www.divio.com/blog/documentation/ > [2] Preparing to Teach (from the Software Carpentry Instructor Training > materials). > https://carpentries.github.io/instructor-training/15-lesson-study/index.html > [3] NumPy for absolute beginners Tutorial by Anne Bonner. > https://numpy.org/devdocs/user/absolute_beginners.html > > Copyright > ======== > > This document has been placed in the public domain. > > -- > Melissa Weber Mendonça > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Every good wish, *Inessa Pawson* Executive Director Albus Code inessa(a)albuscode.org [View Less]

1 0

Py-API: Deprecate `np.dtype(np.floating)` and similar dtype creation
by Sebastian Berg Feb. 14, 2020

Feb. 14, 2020

Hi all, In https://github.com/numpy/numpy/pull/15534 I would like to start deprecating creating dtypes from "abstract" scalar classes, such as: np.dtype(np.floating) is np.dtype(np.float64) While, at the same time, `isinstance(np.float32, np.floating)` is true. Right now `arr.astype(np.floating, copy=False)` and, more obviously, `arr.astype(np.dtype(np.floating), copy=False)` will cast a float32 array to float64. I think we should deprecate this, to consistently enable that in the future `… [View More]

2 3

manylinux2010.
by Charles R Harris Feb. 14, 2020

Feb. 14, 2020

Hi All, Just a note that I've moved the nightly NumPy wheels builds to manylinux2010. Downstream projects testing against those wheels should check that they are using pip >= 19.0 in order to fetch those wheels. If there are any problems, please note them here. Chuck

2 2

NEP 38 - Universal SIMD intrinsics
by Matti Picus Feb. 13, 2020

Feb. 13, 2020

Together with Sayed Adel (cc) and Ralf, I am pleased to put the draft version of NEP 38 [0] up for discussion. As per NEP 0, this is the next step in the community accepting the approach layed out in the NEP. The NEP PR [1] has already garnered a fair amount of discussion about the viability of Universal SIMD Intrinsics, so I will try to capture some of that here as well. Abstract While compilers are getting better at using hardware-specific routines to optimize code, they sometimes do … [View More]not produce optimal results. Also, we would like to be able to copy binary optimized C-extension modules from one machine to another with the same base architecture (x86, ARM, PowerPC) but with different capabilities without recompiling. We have a mechanism in the ufunc machinery to build alternative loops indexed by CPU feature name. At import (in InitOperators), the loop function that matches the run-time CPU info is chosen from the candidates.This NEP proposes a mechanism to build on that for many more features and architectures. The steps proposed are to: Establish a set of well-defined, architecture-agnostic, universal intrisics which capture features available across architectures. Capture these universal intrisics in a set of C macros and use the macros to build code paths for sets of features from the baseline up to the maximum set of features available on that architecture. Offer these as a limited number of compiled alternative code paths. At runtime, discover which CPU features are available, and choose from among the possible code paths accordingly. Motivation and Scope Traditionally NumPy has counted on the compilers to generate optimal code specifically for the target architecture. However few users today compile NumPy locally for their machines. Most use the binary packages which must provide run-time support for the lowest-common denominator CPU architecture. Thus NumPy cannot take advantage of more advanced features of their CPU processors, since they may not be available on all users’ systems. The ufunc machinery already has a loop-selection protocol based on dtypes, so it is easy to extend this to also select an optimal loop for specifically available CPU features at runtime. Traditionally, these features have been exposed through intrinsics which are compiler-specific instructions that map directly to assembly instructions. Recently there were discussions about the effectiveness of adding more intrinsics (e.g., `gh-11113`_ for AVX optimizations for floats). In the past, architecture-specific code was added to NumPy for fast avx512 routines in various ufuncs, using the mechanism described above to choose the best loop for the architecture. However the code is not generic and does not generalize to other architectures. Recently, OpenCV moved to using universal intrinsics in the Hardware Abstraction Layer (HAL) which provided a nice abstraction for common shared Single Instruction Multiple Data (SIMD) constructs. This NEP proposes a similar mechanism for NumPy. There are three stages to using the mechanism: - Infrastructure is provided in the code for abstract intrinsics. The ufunc machinery will be extended using sets of these abstract intrinsics, so that a single ufunc will be expressed as a set of loops, going from a minimal to a maximal set of possibly availabe intrinsics. - At compile time, compiler macros and CPU detection are used to turn the abstract intrinsics into concrete intrinsic calls. Any intrinsics not available on the platform, either because the CPU does not support them (and so cannot be tested) or because the abstract intrinsic does not have a parallel concrete intrinsic on the platform will not error, rather the corresponding loop will not be produced and added to the set of possibilities. - At runtime, the CPU detection code will further limit the set of loops available, and the optimal one will be chosen for the ufunc. The current NEP proposes only to use the runtime feature detection and optimal loop selection mechanism for ufuncs. Future NEPS may propose other uses for the proposed solution. Usage and Impact The end user will be able to get a list of intrinsics available for their platform and compiler. Optionally, the user may be able to specify which of the loops available at runtime will be used, perhaps via an environment variable to enable benchmarking the impact of the different loops. There should be no direct impact to naive end users, the results of all the loops should be identical to within a small number (1-3?) ULPs. On the other hand, users with more powerful machines should notice a significant performance boost. Binary releases - wheels on PyPI and conda packages The binaries released by this process will be larger since they include all possible loops for the architecture. Some packagers may prefer to limit the number of loops in order to limit the size of the binaries, we would hope they would still support a wide range of families of architectures. Note this problem already exists in the Intel MKL offering, where the binary package includes an extensive set of alternative shared objects (DLLs) for various CPU alternatives. Source builds See “Detailed Description” below. A source build where the packager knows details of the target machine could theoretically produce a smaller binary by choosing to compile only the loops needed by the target via command line arguments. How to run benchmarks to assess performance benefits Adding more code which use intrinsics will make the code harder to maintain. Therefore, such code should only be added if it yields a significant performance benefit. Assessing this performance benefit can be nontrivial. To aid with this, the implementation for this NEP will add a way to select which instruction sets can be used at runtime via environment variables. (name TBD). This ablility is critical for CI code verification. Diagnostics A new dictionary __cpu_features__ will be available to python. The keys are the available features, the value is a boolean whether the feature is available or not. Various new private C functions will be used internally to query available features. These might be exposed via specific c-extension modules for testing. Workflow for adding a new CPU architecture-specific optimization NumPy will always have a baseline C implementation for any code that may be a candidate for SIMD vectorization. If a contributor wants to add SIMD support for some architecture (typically the one of most interest to them), this is the proposed workflow: TODO (see https://github.com/numpy/numpy/pull/13516#issuecomment-558859638, needs to be worked out more) Reuse by other projects It would be nice if the universal intrinsics would be available to other libraries like SciPy or Astropy that also build ufuncs, but that is not an explicit goal of the first implementation of this NEP. ----------------------------------------------------------------------------------- My biased summary of select comments from the PR: (Raghuveer): A very similar SIMD library has been proposed for C++. Here is the link to the details: 1. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0214r8.pdf 2. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/n4808.pdf There is good discussion on the minimal/common set of instructions across architectures (which narrows down to loads, stores, arithmetic, compare, bitwise and shuffle instructions). Based on my developer experience so far, these instructions aren't by themselves enough to implement and optimize NumPy ufuncs. As i pointed out earlier, I think I would find it useful to learn the workflow of how to use instructions that don't fit in the Universal Intrinsic framework. (Raguveer) gave a well laid out table of currently proposed unversal intrinsics by use: load/store, reorder, operators, conversions, arithmatic and misc [2] which led to a long response from Sayed [3] with some sample code, demonstrating how more complex operations can be built up from the primitives. (catree) mentioned the Simd Library [4] and Halide [5] and asked about maintainability. (Ralf) responded [6] with concerns about competent developer bandwidth for code review. He also mentioned that our CI system currently supports all the architectures we are targeting (x86, aarch64, s390x, ppc64le) although some of these machines may not have the most advanced hardware to support the latest intrinsics. I apologize if my summary is not accurate, pleas correct any mistakes or misconceptions. ---------------------------------------------------------------------------------------- Barring complete rejection of the idea here, we will be pushing forward with PRs to implement this. Comments either on the mailing list or in those PRs are welcome. Matti [0] https://numpy.org/neps/nep-0038-SIMD-optimizations.html [1] https://github.com/numpy/numpy/pull/15228 [2] https://github.com/numpy/numpy/pull/15228#issuecomment-580479336 [3] https://github.com/numpy/numpy/pull/15228#issuecomment-580605718 [4] https://github.com/ermig1979/Simd [5] https://halide-lang.org [6] https://github.com/numpy/numpy/pull/15228#issuecomment-581029991 [View Less]

7 12

[ANN] 13ᵗʰ Advanced Scientific Programming in Python in Ghent, Belgium, 31 August—5 September, 2020
by Tiziano Zito Feb. 13, 2020

Feb. 13, 2020

13ᵗʰ Advanced Scientific Programming in Python ============================================== a Summer School by the ASPP faculty and the Ghent University https://aspp.school Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In … [View More]this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or R is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. Date & Location =============== 31 August–5 September, 2020. Ghent, Belgium. Application =========== You can apply online: https://aspp.school/wiki/applications Application deadline: 23:59 UTC, Sunday 24 May, 2020 There will be no deadline extension, so be sure to apply on time. Be sure to read the FAQ before applying: https://aspp.school/wiki/faq Participation is for free, i.e. no fee is charged! Accommodation in the student residence comes at no costs for participants. We are trying to arrange financial coverage for food expenses too, but this may not work. Participants however should take care of travel expenses by themselves. Program ======= • Version control with git and how to contribute to open source projects with GitHub • Best practices in data visualization • Testing and debugging scientific code • Advanced NumPy • Organizing, documenting, and distributing scientific code • Advanced scientific Python: context managers and generators • Writing parallel applications in Python • Profiling and speeding up scientific code with Cython and numba • Programming in teams Faculty ======= • Caterina Buizza, Personal Robotics Lab, Imperial College London UK • Lisa Schwetlick, Experimental and Biological Psychology, Universität Potsdam Germany • Nelle Varoquaux, CNRS, TIMC-IMAG, University Grenoble Alpes France • Nicolas P. Rougier, Inria Bordeaux Sud-Ouest, Institute of Neurodegenerative Disease, University of Bordeaux France • Pamela Hathway, Neural Reckoning, Imperial College London UK • Pietro Berkes, NAGRA Kudelski, Lausanne Switzerland • Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universität zu Berlin Germany • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany • Zbigniew Jędrzejewski-Szmek, Red Hat Inc., Warsaw Poland Organizers ========== Head of the organization for ASPP and responsible for the scientific program: • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany Local team: • Nina Turk, Photonics Research Group, INTEC, Ghent University – imec Belgium • Freya Acar, Office for Data and Information, City of Ghent Belgium • Joan Juvert Institutional organizers: • Wim Bogaerts, Photonics Research Group, INTEC, Ghent University –imec Belgium • Sven Degroeve, VIB-UGent Center for Medical Biotechnology, Ghent Belgium • Jeroen Famaey, Department of Mathematics and Computer Science, University of Antwerp – iMinds Belgium • Bernard Manderick, Artificial Intelligence Lab, Vrije Universiteit Brussel Belgium Website: https://aspp.school Contact: info(a)aspp.school [View Less]

1 0

[ANN] 13ᵗʰ Advanced Scientific Programming in Python in Ghent, Belgium, 31 August—5 September, 2020
by ASPP Feb. 13, 2020

Feb. 13, 2020

13ᵗʰ Advanced Scientific Programming in Python ============================================== a Summer School by the ASPP faculty and the Ghent University https://aspp.school Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In … [View More]this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or R is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. Date & Location =============== 31 August–5 September, 2020. Ghent, Belgium. Application =========== You can apply online: https://aspp.school/wiki/applications Application deadline: 23:59 UTC, Sunday 24 May, 2020 There will be no deadline extension, so be sure to apply on time. Be sure to read the FAQ before applying: https://aspp.school/wiki/faq Participation is for free, i.e. no fee is charged! Accommodation in the student residence comes at no costs for participants. We are trying to arrange financial coverage for food expenses too, but this may not work. Participants however should take care of travel expenses by themselves. Program ======= • Version control with git and how to contribute to open source projects with GitHub • Best practices in data visualization • Testing and debugging scientific code • Advanced NumPy • Organizing, documenting, and distributing scientific code • Advanced scientific Python: context managers and generators • Writing parallel applications in Python • Profiling and speeding up scientific code with Cython and numba • Programming in teams Faculty ======= • Caterina Buizza, Personal Robotics Lab, Imperial College London UK • Lisa Schwetlick, Experimental and Biological Psychology, Universität Potsdam Germany • Nelle Varoquaux, CNRS, TIMC-IMAG, University Grenoble Alpes France • Nicolas P. Rougier, Inria Bordeaux Sud-Ouest, Institute of Neurodegenerative Disease, University of Bordeaux France • Pamela Hathway, Neural Reckoning, Imperial College London UK • Pietro Berkes, NAGRA Kudelski, Lausanne Switzerland • Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universität zu Berlin Germany • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany • Zbigniew Jędrzejewski-Szmek, Red Hat Inc., Warsaw Poland Organizers ========== Head of the organization for ASPP and responsible for the scientific program: • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany Local team: • Nina Turk, Photonics Research Group, INTEC, Ghent University – imec Belgium • Freya Acar, Office for Data and Information, City of Ghent Belgium • Joan Juvert Institutional organizers: • Wim Bogaerts, Photonics Research Group, INTEC, Ghent University –imec Belgium • Sven Degroeve, VIB-UGent Center for Medical Biotechnology, Ghent Belgium • Jeroen Famaey, Department of Mathematics and Computer Science, University of Antwerp – iMinds Belgium • Bernard Manderick, Artificial Intelligence Lab, Vrije Universiteit Brussel Belgium Website: https://aspp.school Contact: info(a)aspp.school [View Less]

1 0

NEP 44 - Restructuring the NumPy Documentation
by Melissa Mendonça Feb. 12, 2020

Feb. 12, 2020

Hi all, Please see the NEP below for a proposal to restructure the documentation of NumPy. The main goal here is to make the documentation more visible and organized, and also make contributions easier. Comments and feedback are welcome! See https://github.com/numpy/numpy/pull/15554 for details. Best, Melissa ---- NEP 44 — Restructuring the NumPy Documentation Authors: Ralf Gommers, Melissa Mendonça, Mars Lee Status: Draft Type: Process Created: 2020-02-11 Abstract ====== This … [View More]document proposes a restructuring of the NumPy Documentation, both in form and content, with the goal of making it more organized and discoverable for beginners and experienced users. Motivation and Scope ================= See [here](numpy.org/devdocs) for the front page of the latest docs. The organization is quite confusing and illogical (e.g. user and developer docs are mixed). We propose the following: - Reorganizing the docs into the four categories mentioned in [1]; - Creating dedicated sections for Tutorials and How-Tos, including orientation on how to create new content; - Adding an Explanations section for key concepts and techniques that require deeper descriptions, some of which will be rearranged from the Reference Guide. Usage and Impact ============== The documentation is a fundamental part of any software project, especially open source projects. In the case of NumPy, many beginners might feel demotivated by the current structure of the documentation, since it is difficult to discover what to learn (unless the user has a clear view of what to look for in the Reference docs, which is not always the case). Looking at the results of a “NumPy Tutorial” search on any search engine also gives an idea of the demand for this kind of content. Having official high-level documentation written using up-to-date content and techniques will certainly mean more users (and developers/contributors) are involved in the NumPy community. Backward compatibility ================== The restructuring will effectively demand a complete rewrite of links and some of the current content. Input from the community will be useful for identifying key links and pages that should not be broken. Detailed description =============== As discussed in the article [1], there are four categories of doc content: - Tutorials - How-to guides - Explanations - Reference guide We propose to use those categories as the ones we use (for writing and reviewing) whenever we add a new documentation section. The reasoning for this is that it is clearer both for developers/documentation writers and to users where each information should go, and the scope and tone of each document. For example, if explanations are mixed with basic tutorials, beginners might be overwhelmed and alienated. On the other hand, if the reference guide contains basic how-tos, it might be difficult for experienced users to find the information they need, quickly. Currently, there are many blogs and tutorials on the internet about NumPy or using NumPy. One of the issues with this is that if users search for this information and end up in an outdated (unofficial) tutorial before they find the current official documentation, they end up creating content that is confusing, especially for beginners. Having a better infrastructure for the documentation also aims to solve this problem by giving users high-level, up-to-date official documentation that can be easily updated. Status and ideas of each type of doc content ------------------------------------------------------------ * Reference guide NumPy has a quite complete reference guide. All functions are documented, most have examples, and most are cross-linked well with See Also sections. Further improving the reference guide is incremental work that can be done (and is being done) by many people. There are, however, many explanations in the reference guide. These can be moved to a more dedicated Explanations section on the docs. * How-to guides NumPy does not have many how-to’s. The subclassing and array ducktyping section may be an example of a how-to. Others that could be added are: - Parallelization (controlling BLAS multithreading with threadpoolctl, using multiprocessing, random number generation, etc.) - Storing and loading data (.npy/.npz format, text formats, Zarr, HDF5, Bloscpack, etc.) - Performance (memory layout, profiling, use with Numba, Cython, or Pythran) - Writing generic code that works with NumPy, Dask, CuPy, pydata/sparse, etc. * Explanations There is a reasonable amount of content on fundamental NumPy concepts such as indexing, vectorization, broadcasting, (g)ufuncs, and dtypes. This could be organized better and clarified to ensure it’s really about explaining the concepts and not mixed with tutorial or how-to like content. There are few explanations about anything other than those fundamental NumPy concepts. Some examples of concepts that could be expanded: - Copies vs. Views; - BLAS and other linear algebra libraries; - Fancy indexing. In addition, there are many explanations in the Reference Guide, which should be moved to this new dedicated Explanations section. * Tutorials There’s a lot of scope for writing better tutorials. We have a new NumPy for absolute beginners tutorial [3] (GSoD project of Anne Bonner). In addition we need a number of tutorials addressing different levels of experience with Python and NumPy. This could be done using engaging data sets, ideas or stories. For example, curve fitting with polynomials and functions in numpy.linalg could be done with the Keeling curve (decades worth of CO2 concentration in air measurements) rather than with synthetic random data. Ideas for tutorials (these capture the types of things that make sense, they’re not necessarily the exact topics we propose to implement): - Conway’s game of life with only NumPy (note: already in Nicolas Rougier’s book) - Using masked arrays to deal with missing data in time series measurements - Using Fourier transforms to analyze the Keeling curve data, and extrapolate it. - Geospatial data (e.g. lat/lon/time to create maps for every year via a stacked array, like gridMet data) - Using text data and dtypes (e.g. use speeches from different people, shape (n_speech, n_sentences, n_words)) The Preparing to Teach document [2] from the Software Carpentry Instructor Training materials is a nice summary of how to write effective lesson plans (and tutorials would be very similar). In addition to adding new tutorials, we also propose a How to write a tutorial document, which would help users contribute new high-quality content to the documentation. Data sets ------------- Using interesting data in the NumPy docs requires giving all users access to that data, either inside NumPy or in a separate package. The former is not the best idea, since it’s hard to do without increasing the size of NumPy significantly. Even for SciPy there has so far been no consensus on this (see scipy PR 8707 on adding a new scipy.datasets subpackage). So we’ll aim for a new (pure Python) package, named numpy-datasets or scipy-datasets or something similar. That package can take some lessons from how, e.g., scikit-learn ships data sets. Small data sets can be included in the repo, large data sets can be accessed via a downloader class or function. Related Work =========== Some examples of documentation organization in other projects: - Documentation for Jupyter: https://jupyter.org/documentation - Documentation for Python: https://docs.python.org/3/ - Documentation for TensorFlow: https://www.tensorflow.org/learn These projects make the intended audience for each part of the documentation more explicit, as well as previewing some of the content in each section. Implementation ============ Besides rewriting the current documentation to some extent, it would be ideal to have a technical infrastructure that would allow more contributions from the community. For example, if Jupyter Notebooks could be submitted as-is as tutorials or How-Tos, this might create more contributors and broaden the NumPy community. Similarly, if people could download some of the documentation in Notebook format, this would certainly mean people would use less outdated material for learning NumPy. It would also be interesting if the new structure for the documentation makes translations easier. Currently, the documentation for NumPy can be confusing, especially for beginners. Our proposal is to reorganize the docs in the following structure: * For users: - Absolute Beginners Tutorial - main Tutorials section - How To’s for common tasks with NumPy - Reference Guide - Explanations - F2Py Guide - Glossary * For developers/contributors: - Contributor’s Guide - Building and extending the documentation - Benchmarking - NumPy Enhancement Proposals * Meta information - Reporting bugs - Release Notes - About NumPy - License References and Footnotes ==================== [1] What nobody tells you about documentation. https://www.divio.com/blog/documentation/ [2] Preparing to Teach (from the Software Carpentry Instructor Training materials). https://carpentries.github.io/instructor-training/15-lesson-study/index.html [3] NumPy for absolute beginners Tutorial by Anne Bonner. https://numpy.org/devdocs/user/absolute_beginners.html Copyright ======== This document has been placed in the public domain. -- Melissa Weber Mendonça [View Less]

1 0

Re: [Numpy-discussion] Proposal - extend histograms api to allow uneven bins
by Ross Barnowski Feb. 11, 2020

Feb. 11, 2020

Just a few thoughts re: the changes proposed in https://github.com/numpy/numpy/pull/14278 1. Though the PR is limited to the 'auto' kwarg, the issue of potential memory problems for the automated binning methods is a more general one (e.g. #15332 <https://github.com/numpy/numpy/issues/15332>). 2. The main concern that jumps out to me is downstream users who are relying on the implicit assumption of regular binning. This is of course bad practice and makes even less sense when using one … [View More]of the bin estimators, so I'm not sure how big of a concern it is. However, there is likely downstream user code that relies on the regular binning assumption, especially since, as far as I know, NumPy has never implemented binning techniques that return irregular bins. 3. The astropy project have at least one estimator that returns irregular bins <https://docs.astropy.org/en/stable/visualization/histogram.html#>. I checked for issues <https://github.com/astropy/astropy/issues?utf8=%E2%9C%93&q=is%3Aissue+histo…> related to irregular binning: though they have many of the same problems with the automatic bin estimators (i.e. memory problems for inputs with outliers), I didn't see anything specifically related to irregular binning I just wanted to add my two cents. The binning-data-with-outliers problem is very common in high-resolution spectroscopy, and I have seen practitioners rely on the assumption of regular binning (e.g. divide the `range` by the number of bins) to specify bin centers even though this is not the right way to do things. Thanks for taking the time to write up your work! On Mon, Feb 10, 2020 at 10:53 PM <numpy-discussion-request(a)python.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)python.org > > You can reach the person managing the list at > numpy-discussion-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Proposal - extend histograms api to allow uneven bins > (Alexander Reeves) > 2. Re: NEP 38 - Universal SIMD intrinsics (Ralf Gommers) > 3. Re: NEP 38 - Universal SIMD intrinsics (Matti Picus) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 10 Feb 2020 18:07:40 -0800 > From: Alexander Reeves <lxndr.rvs(a)gmail.com> > To: numpy-discussion(a)python.org > Subject: [Numpy-discussion] Proposal - extend histograms api to allow > uneven bins > Message-ID: > <CABeAeRyiVt4RJP2ew7= > C4c4itJO3WZptdD6yrhWV6NgUoqT_mQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Greetings, > > I have a PR that warrants discussion according to @seberg. See > https://github.com/numpy/numpy/pull/14278. > > It is an enhancement that fixes a bug. The original bug is that when using > the fd estimator on a dataset with small inter-quartile range and large > outliers, the current codebase produces more bins than memory allows. There > are several related bug reports (see #11879, #10297, #8203). > > In terms of scope, I restricted my changes to conditions where > np.histogram(bins='auto') defaults to the 'fd'. For the actual fix, I > actually enhanced the API. I used a suggestion from @eric-wieser to merge > empty histogram bins. In practice this solves the outsized bins issue. > > However @seberg is concerned that extending the API in this way may not be > the way to go. For example, if you use "auto" once, and then re-use the > bins, the uneven bins may not be what you want. > > Furthermore @eric-wieser is concerned that there may be a floating-point > devil in the details. He advocates using the hypothesis testing package to > increase our confidence that the current implementation adequately handles > corner cases. > > I would like to do my part in improving the code base. I don't have strong > opinions but I have to admit that I would like to eventually make a PR that > resolves these bugs. This has been a PR half a year in the making after > all. > > Thoughts? > > -areeves87 > [View Less]

1 0

NumPy Development Meeting - Triage Focus
by Sebastian Berg Feb. 11, 2020

Feb. 11, 2020

Hi all, Our bi-weekly triage-focused NumPy development meeting is tomorrow (Wednesday, Februrary 12) at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized or simply discussed briefly. Just comment on it so we can label it, or add your PR/issue to this weeks topics for discussion. Best regards, Sebastian

1 0

Proposal - extend histograms api to allow uneven bins
by Alexander Reeves Feb. 11, 2020

Feb. 11, 2020

Greetings, I have a PR that warrants discussion according to @seberg. See https://github.com/numpy/numpy/pull/14278. It is an enhancement that fixes a bug. The original bug is that when using the fd estimator on a dataset with small inter-quartile range and large outliers, the current codebase produces more bins than memory allows. There are several related bug reports (see #11879, #10297, #8203). In terms of scope, I restricted my changes to conditions where np.histogram(bins='auto') … [View More]

1 0

manylinux upgrade for numpy wheels
by Charles R Harris Feb. 6, 2020

Feb. 6, 2020

Hi All, Thought now would be a good time to decide on upgrading manylinux for the 1.19 release so that we can make sure that everything works as expected. The choices are manylinux1 <https://www.python.org/dev/peps/pep-0513/> -- CentOS 5, currently used, gcc 4.2 (in practice 4.5), only supports i686, x86_64. manylinux2010 <https://www.python.org/dev/peps/pep-0571/> -- CentOS 6, gcc 4.5, only supports i686, x86_64. manylinux2014 <https://www.python.org/dev/peps/pep-0599/> -- … [View More]

5 4

NumPy Community Meeting Wednesday, Feb. 05
by Sebastian Berg Feb. 4, 2020

Feb. 4, 2020

Hi all, There will be a NumPy Community meeting Wednesday February 5 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0