Mailman 3 June 2018 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

NumPy 1.15.0rc1 released
by Charles R Harris 09 Jul '18

09 Jul '18

Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.15.0rc1. This release has an unusual number of cleanups, many deprecations of old functions, and improvements to many existing functions. A total of 423 pull reguests were merged for this release, please look at the release notes <https://github.com/numpy/numpy/releases/tag/v1.15.0rc1>for details. Some highlights are: - NumPy has switched to pytest for testing. - A new `numpy.printoptions` context manager. - Many improvements to the histogram functions. - Support for unicode field names in python 2.7. - Improved support for PyPy. The Python versions supported by this release are 2.7, 3.4-3.6. The wheels are linked with OpenBLAS 3.0, which should fix some of the linalg problems reported for NumPy 1.14, and the source archives were created using Cython 0.28.2 and should work with the upcoming Python 3.7. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/#description>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.0rc1>. A total of 128 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Critchley + - Aarthi + - Aarthi Agurusa + - Alex Thomas + - Alexander Belopolsky - Allan Haldane - Anas Khan + - Andras Deak - Andrey Portnoy + - Anna Chiara - Aurelien Jarno + - Baurzhan Muftakhidinov - Berend Kapelle + - Bernhard M. Wiedemann - Bjoern Thiel + - Bob Eldering - Cenny Wenner + - Charles Harris - ChloeColeongco + - Chris Billington + - Christopher + - Chun-Wei Yuan + - Claudio Freire + - Daniel Smith - Darcy Meyer + - David Abdurachmanov + - David Freese - Deepak Kumar Gouda + - Dennis Weyland + - Derrick Williams + - Dmitriy Shalyga + - Eric Cousineau + - Eric Larson - Eric Wieser - Evgeni Burovski - Frederick Lefebvre + - Gaspar Karm + - Geoffrey Irving - Gerhard Hobler + - Gerrit Holl - Guo Ci + - Hameer Abbasi + - Han Shen - Hiroyuki V. Yamazaki + - Hong Xu - Ihor Melnyk + - Jaime Fernandez - Jake VanderPlas + - James Tocknell + - Jarrod Millman - Jeff VanOss + - John Kirkham - Jonas Rauber + - Jonathan March + - Joseph Fox-Rabinovitz - Julian Taylor - Junjie Bai + - Juris Bogusevs + - Jörg Döpfert - Kenichi Maehashi + - Kevin Sheppard - Kimikazu Kato + - Kirit Thadaka + - Kritika Jalan + - Lakshay Garg + - Lars G + - Licht Takeuchi - Louis Potok + - Luke Zoltan Kelley - MSeifert04 + - Mads R. B. Kristensen + - Malcolm Smith + - Mark Harfouche + - Marten H. van Kerkwijk + - Marten van Kerkwijk - Matheus Vieira Portela + - Mathieu Lamarre - Mathieu Sornay + - Matthew Brett - Matthew Rocklin + - Matthias Bussonnier - Matti Picus - Michael Droettboom - Miguel Sánchez de León Peque + - Mike Toews + - Milo + - Nathaniel J. Smith - Nelle Varoquaux - Nicholas Nadeau + - Nick Minkyu Lee + - Nikita + - Nikita Kartashov + - Nils Becker + - Oleg Zabluda - Orestis Floros + - Pat Gunn + - Paul van Mulbregt + - Pauli Virtanen - Pierre Chanial + - Ralf Gommers - Raunak Shah + - Robert Kern - Russell Keith-Magee + - Ryan Soklaski + - Samuel Jackson + - Sebastian Berg - Siavash Eliasi + - Simon Conseil - Simon Gibbons - Stefan Krah + - Stefan van der Walt - Stephan Hoyer - Subhendu + - Subhendu Ranjan Mishra + - Tai-Lin Wu + - Tobias Fischer + - Toshiki Kataoka + - Tyler Reddy + - Varun Nayyar - Victor Rodriguez + - Warren Weckesser - Zane Bradley + - fo40225 - lumbric + - luzpaz + - mamrehn + - tynn + - xoviat Cheers Chuck

4 3

Allowing broadcasting of code dimensions in generalized ufuncs
by Marten van Kerkwijk 03 Jul '18

03 Jul '18

Hi All, Following on a PR combining the ability to provide fixed and flexible dimensions [1] (useful for, e.g., 3-vector input with a signature like `(3),(3)->(3)`, and for `matmul`, resp.; based on earlier PRs by Jaime [2] and Matt (Picus) [3]), I've now made a PR with a further enhancement, which allows one can indicate that a core dimension can be broadcast [4]. A particular use case is `all_equal`, a new function suggested in a stalled PR by Matt (Harrigan) [5], which compares two arrays axis-by-axis, but short-circuits if a non-equality is found (unlike what is the case if one does `(a==b).all(axis)`). One thing that would be obviously useful for a routine like `all_equal` is to be able to provide an array as one argument and a constant as another, i.e., if the core dimensions can be broadcast if needed, just like they are in `(a==b).all(axis)`. This is currently not possible: with its signature of `(n),(n)->()`, the two arrays have to have the same trailing size. My PR provides the ability to indicate in the signature that a core dimension can be broadcast, by using a suffix of "|1". Thus, the signature of `all_equal` would become: ``` (n|1),(n|1)->() ``` Comments most welcome (yes, even on the notation - though I think it is fairly self-explanatory)! Marten p.s. There are some similarities to the new "flexible" dimensions implemented for `matmul` [1], but also differences. In particular, for a signature of `(n?),(n?)->()`, one could also pass in an array of trailing size n and a constant, but it would not be possible to pass in an array with trailing size 1: the dimensions with the same name have to be either present and the same or absent. In contrast, for broadcasting, dimensions with the same name can have trailing size n, size 1, or be absent (in which case they count as 1). For broadcasting, any output dimensions with the same name are never affected, while for flexible dimensions those are removed. [1] https://github.com/numpy/numpy/pull/11175 [2] https://github.com/numpy/numpy/pull/5015 [3] https://github.com/numpy/numpy/pull/11132 [4] https://github.com/numpy/numpy/pull/11179 [5] https://github.com/numpy/numpy/pull/8528

10 42

Polynomial evaluation inconsistencies
by Maxwell Aifer 02 Jul '18

02 Jul '18

Hi, I noticed some frustrating inconsistencies in the various ways to evaluate polynomials using numpy. Numpy has three ways of evaluating polynomials (that I know of) and each of them has a different syntax: - numpy.polynomial.polynomial.Polynomial <https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.polynomia…>: You define a polynomial by a list of coefficients *in order of increasing degree*, and then use the class’s call() function. - np.polyval <https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.polyval.h…>: Evaluates a polynomial at a point. *First* argument is the polynomial, or list of coefficients *in order of decreasing degree*, and the *second* argument is the point to evaluate at. - np.polynomial.polynomial.polyval <https://docs.scipy.org/doc/numpy-1.12.0/reference/generated/numpy.polynomia…>: Also evaluates a polynomial at a point, but has more support for vectorization. *First* argument is the point to evaluate at, and *second* argument the list of coefficients *in order of increasing degree*. Not only the order of arguments is changed between different methods, but the order of the coefficients is reversed as well, leading to puzzling bugs (in my experience). What could be the reason for this madness? As polyval is a shameless ripoff of Matlab’s function of the same name <https://www.mathworks.com/help/matlab/ref/polyval.html> anyway, why not just use matlab’s syntax (polyval([c0, c1, c2...], x)) across the board?

4 18

Revised NEP-18, __array_function__ protocol
by Stephan Hoyer 30 Jun '18

30 Jun '18

After much discussion (and the addition of three new co-authors!), I’m pleased to present a significantly revision of NumPy Enhancement Proposal 18: A dispatch mechanism for NumPy's high level array functions: http://www.numpy.org/neps/nep-0018-array-function-protocol.html The full text is also included below. Best, Stephan =========================================================== A dispatch mechanism for NumPy's high level array functions =========================================================== :Author: Stephan Hoyer <shoyer(a)google.com> :Author: Matthew Rocklin <mrocklin(a)gmail.com> :Author: Marten van Kerkwijk <mhvk(a)astro.utoronto.ca> :Author: Hameer Abbasi <hameerabbasi(a)yahoo.com> :Author: Eric Wieser <wieser.eric(a)gmail.com> :Status: Draft :Type: Standards Track :Created: 2018-05-29 Abstact ------- We propose the ``__array_function__`` protocol, to allow arguments of NumPy functions to define how that function operates on them. This will allow using NumPy as a high level API for efficient multi-dimensional array operations, even with array implementations that differ greatly from ``numpy.ndarray``. Detailed description -------------------- NumPy's high level ndarray API has been implemented several times outside of NumPy itself for different architectures, such as for GPU arrays (CuPy), Sparse arrays (scipy.sparse, pydata/sparse) and parallel arrays (Dask array) as well as various NumPy-like implementations in the deep learning frameworks, like TensorFlow and PyTorch. Similarly there are many projects that build on top of the NumPy API for labeled and indexed arrays (XArray), automatic differentiation (Autograd, Tangent), masked arrays (numpy.ma), physical units (astropy.units, pint, unyt), etc. that add additional functionality on top of the NumPy API. Most of these project also implement a close variation of NumPy's level high API. We would like to be able to use these libraries together, for example we would like to be able to place a CuPy array within XArray, or perform automatic differentiation on Dask array code. This would be easier to accomplish if code written for NumPy ndarrays could also be used by other NumPy-like projects. For example, we would like for the following code example to work equally well with any NumPy-like array object: .. code:: python def f(x): y = np.tensordot(x, x.T) return np.mean(np.exp(y)) Some of this is possible today with various protocol mechanisms within NumPy. - The ``np.exp`` function checks the ``__array_ufunc__`` protocol - The ``.T`` method works using Python's method dispatch - The ``np.mean`` function explicitly checks for a ``.mean`` method on the argument However other functions, like ``np.tensordot`` do not dispatch, and instead are likely to coerce to a NumPy array (using the ``__array__``) protocol, or err outright. To achieve enough coverage of the NumPy API to support downstream projects like XArray and autograd we want to support *almost all* functions within NumPy, which calls for a more reaching protocol than just ``__array_ufunc__``. We would like a protocol that allows arguments of a NumPy function to take control and divert execution to another function (for example a GPU or parallel implementation) in a way that is safe and consistent across projects. Implementation -------------- We propose adding support for a new protocol in NumPy, ``__array_function__``. This protocol is intended to be a catch-all for NumPy functionality that is not covered by the ``__array_ufunc__`` protocol for universal functions (like ``np.exp``). The semantics are very similar to ``__array_ufunc__``, except the operation is specified by an arbitrary callable object rather than a ufunc instance and method. A prototype implementation can be found in `this notebook < https://nbviewer.jupyter.org/gist/shoyer/1f0a308a06cd96df20879a1ddb8f0006 >`_. The interface ~~~~~~~~~~~~~ We propose the following signature for implementations of ``__array_function__``: .. code-block:: python def __array_function__(self, func, types, args, kwargs) - ``func`` is an arbitrary callable exposed by NumPy's public API, which was called in the form ``func(*args, **kwargs)``. - ``types`` is a ``frozenset`` of unique argument types from the original NumPy function call that implement ``__array_function__``. - The tuple ``args`` and dict ``kwargs`` are directly passed on from the original call. Unlike ``__array_ufunc__``, there are no high-level guarantees about the type of ``func``, or about which of ``args`` and ``kwargs`` may contain objects implementing the array API. As a convenience for ``__array_function__`` implementors, ``types`` provides all argument types with an ``'__array_function__'`` attribute. This allows downstream implementations to quickly determine if they are likely able to support the operation. A ``frozenset`` is used to ensure that ``__array_function__`` implementations cannot rely on the iteration order of ``types``, which would facilitate violating the well-defined "Type casting hierarchy" described in `NEP-13 <https://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_. Example for a project implementing the NumPy API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Most implementations of ``__array_function__`` will start with two checks: 1. Is the given function something that we know how to overload? 2. Are all arguments of a type that we know how to handle? If these conditions hold, ``__array_function__`` should return the result from calling its implementation for ``func(*args, **kwargs)``. Otherwise, it should return the sentinel value ``NotImplemented``, indicating that the function is not implemented by these types. This is preferable to raising ``TypeError`` directly, because it gives *other* arguments the opportunity to define the operations. There are no general requirements on the return value from ``__array_function__``, although most sensible implementations should probably return array(s) with the same type as one of the function's arguments. If/when Python gains `typing support for protocols <https://www.python.org/dev/peps/pep-0544/>`_ and NumPy adds static type annotations, the ``@overload`` implementation for ``SupportsArrayFunction`` will indicate a return type of ``Any``. It may also be convenient to define a custom decorators (``implements`` below) for registering ``__array_function__`` implementations. .. code:: python HANDLED_FUNCTIONS = {} class MyArray: def __array_function__(self, func, types, args, kwargs): if func not in HANDLED_FUNCTIONS: return NotImplemented # Note: this allows subclasses that don't override # __array_function__ to handle MyArray objects if not all(issubclass(t, MyArray) for t in types): return NotImplemented return HANDLED_FUNCTIONS[func](*args, **kwargs) def implements(numpy_function): """Register an __array_function__ implementation for MyArray objects.""" def decorator(func): HANDLED_FUNCTIONS[numpy_function] = func return func return decorator @implements(np.concatenate) def concatenate(arrays, axis=0, out=None): ... # implementation of concatenate for MyArray objects @implements(np.broadcast_to) def broadcast_to(array, shape): ... # implementation of broadcast_to for MyArray objects Note that it is not required for ``__array_function__`` implementations to include *all* of the corresponding NumPy function's optional arguments (e.g., ``broadcast_to`` above omits the irrelevant ``subok`` argument). Optional arguments are only passed in to ``__array_function__`` if they were explicitly used in the NumPy function call. Necessary changes within the NumPy codebase itself ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This will require two changes within the NumPy codebase: 1. A function to inspect available inputs, look for the ``__array_function__`` attribute on those inputs, and call those methods appropriately until one succeeds. This needs to be fast in the common all-NumPy case, and have acceptable performance (no worse than linear time) even if the number of overloaded inputs is large (e.g., as might be the case for `np.concatenate`). This is one additional function of moderate complexity. 2. Calling this function within all relevant NumPy functions. This affects many parts of the NumPy codebase, although with very low complexity. Finding and calling the right ``__array_function__`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Given a NumPy function, ``*args`` and ``**kwargs`` inputs, we need to search through ``*args`` and ``**kwargs`` for all appropriate inputs that might have the ``__array_function__`` attribute. Then we need to select among those possible methods and execute the right one. Negotiating between several possible implementations can be complex. Finding arguments ''''''''''''''''' Valid arguments may be directly in the ``*args`` and ``**kwargs``, such as in the case for ``np.tensordot(left, right, out=out)``, or they may be nested within lists or dictionaries, such as in the case of ``np.concatenate([x, y, z])``. This can be problematic for two reasons: 1. Some functions are given long lists of values, and traversing them might be prohibitively expensive. 2. Some functions may have arguments that we don't want to inspect, even if they have the ``__array_function__`` method. To resolve these issues, NumPy functions should explicitly indicate which of their arguments may be overloaded, and how these arguments should be checked. As a rule, this should include all arguments documented as either ``array_like`` or ``ndarray``. We propose to do so by writing "dispatcher" functions for each overloaded NumPy function: - These functions will be called with the exact same arguments that were passed into the NumPy function (i.e., ``dispatcher(*args, **kwargs)``), and should return an iterable of arguments to check for overrides. - Dispatcher functions are required to share the exact same positional, optional and keyword-only arguments as their corresponding NumPy functions. Otherwise, valid invocations of a NumPy function could result in an error when calling its dispatcher. - Because default *values* for keyword arguments do not have ``__array_function__`` attributes, by convention we set all default argument values to ``None``. This reduces the likelihood of signatures falling out of sync, and minimizes extraneous information in the dispatcher. The only exception should be cases where the argument value in some way effects dispatching, which should be rare. An example of the dispatcher for ``np.concatenate`` may be instructive: .. code:: python def _concatenate_dispatcher(arrays, axis=None, out=None): for array in arrays: yield array if out is not None: yield out The concatenate dispatcher is written as generator function, which allows it to potentially include the value of the optional ``out`` argument without needing to create a new sequence with the (potentially long) list of objects to be concatenated. Trying ``__array_function__`` methods until the right one works ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' Many arguments may implement the ``__array_function__`` protocol. Some of these may decide that, given the available inputs, they are unable to determine the correct result. How do we call the right one? If several are valid then which has precedence? For the most part, the rules for dispatch with ``__array_function__`` match those for ``__array_ufunc__`` (see `NEP-13 <https://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_). In particular: - NumPy will gather implementations of ``__array_function__`` from all specified inputs and call them in order: subclasses before superclasses, and otherwise left to right. Note that in some edge cases involving subclasses, this differs slightly from the `current behavior <https://bugs.python.org/issue30140>`_ of Python. - Implementations of ``__array_function__`` indicate that they can handle the operation by returning any value other than ``NotImplemented``. - If all ``__array_function__`` methods return ``NotImplemented``, NumPy will raise ``TypeError``. One deviation from the current behavior of ``__array_ufunc__`` is that NumPy will only call ``__array_function__`` on the *first* argument of each unique type. This matches Python's `rule for calling reflected methods < https://docs.python.org/3/reference/datamodel.html#object.__ror__>`_, and this ensures that checking overloads has acceptable performance even when there are a large number of overloaded arguments. To avoid long-term divergence between these two dispatch protocols, we should `also update <https://github.com/numpy/numpy/issues/11306>`_ ``__array_ufunc__`` to match this behavior. Special handling of ``numpy.ndarray`` ''''''''''''''''''''''''''''''''''''' The use cases for subclasses with ``__array_function__`` are the same as those with ``__array_ufunc__``, so ``numpy.ndarray`` should also define a ``__array_function__`` method mirroring ``ndarray.__array_ufunc__``: .. code:: python def __array_function__(self, func, types, args, kwargs): # Cannot handle items that have __array_function__ other than our own. for t in types: if (hasattr(t, '__array_function__') and t.__array_function__ is not ndarray.__array_function__): return NotImplemented # Arguments contain no overrides, so we can safely call the # overloaded function again. return func(*args, **kwargs) To avoid infinite recursion, the dispatch rules for ``__array_function__`` need also the same special case they have for ``__array_ufunc__``: any arguments with an ``__array_function__`` method that is identical to ``numpy.ndarray.__array_function__`` are not be called as ``__array_function__`` implementations. Changes within NumPy functions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Given a function defining the above behavior, for now call it ``try_array_function_override``, we now need to call that function from within every relevant NumPy function. This is a pervasive change, but of fairly simple and innocuous code that should complete quickly and without effect if no arguments implement the ``__array_function__`` protocol. In most cases, these functions should written using the ``array_function_dispatch`` decorator, which also associates dispatcher functions: .. code:: python def array_function_dispatch(dispatcher): """Wrap a function for dispatch with the __array_function__ protocol.""" def decorator(func): @functools.wraps(func) def new_func(*args, **kwargs): relevant_arguments = dispatcher(*args, **kwargs) success, value = try_array_function_override( new_func, relevant_arguments, args, kwargs) if success: return value return func(*args, **kwargs) return new_func return decorator # example usage def _broadcast_to_dispatcher(array, shape, subok=None, **ignored_kwargs): return (array,) @array_function_dispatch(_broadcast_to_dispatcher) def broadcast_to(array, shape, subok=False): ... # existing definition of np.broadcast_to Using a decorator is great! We don't need to change the definitions of existing NumPy functions, and only need to write a few additional lines for the dispatcher function. We could even reuse a single dispatcher for families of functions with the same signature (e.g., ``sum`` and ``prod``). For such functions, the largest change could be adding a few lines to the docstring to note which arguments are checked for overloads. It's particularly worth calling out the decorator's use of ``functools.wraps``: - This ensures that the wrapped function has the same name and docstring as the wrapped NumPy function. - On Python 3, it also ensures that the decorator function copies the original function signature, which is important for introspection based tools such as auto-complete. If we care about preserving function signatures on Python 2, for the `short while longer < http://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html>`_ that NumPy supports Python 2.7, we do could do so by adding a vendored dependency on the (single-file, BSD licensed) `decorator library <https://github.com/micheles/decorator>`_. - Finally, it ensures that the wrapped function `can be pickled < http://gael-varoquaux.info/programming/decoration-in-python-done-right-deco… >`_. In a few cases, it would not make sense to use the ``array_function_dispatch`` decorator directly, but override implementation in terms of ``try_array_function_override`` should still be straightforward. - Functions written entirely in C (e.g., ``np.concatenate``) can't use decorators, but they could still use a C equivalent of ``try_array_function_override``. If performance is not a concern, they could also be easily wrapped with a small Python wrapper. - The ``__call__`` method of ``np.vectorize`` can't be decorated with <p style="margin:0px;font-stretch:normal;font-size:17.4px;line-

6 20

rackspace ssl certificates
by Charles R Harris 30 Jun '18

30 Jun '18

Hi All, I've been trying to put out the NumPy 1.15.0rc1, but cannot get `numpy-wheels` to upload the wheels to rackspace on windows, there is a certification problem. I note that that requirement was supposedly disabled: on_success: # Upload the generated wheel package to Rackspace # On Windows, Apache Libcloud cannot find a standard CA cert bundle so we # disable the ssl checks. and nothing relevant seems to have changed in our `.appveyor.yml` since the last successful run 7 days ago, 6 if we count 1.14.5, so I'm thinking a policy has changed at either at rackspace or appveyor, but that is just a guess. I'm experimenting with various changes to the script and the `apache-libcloud` version to see if I can get success, but thought I'd ask if anyone knew anything that might be helpful. Chuck

4 19

Proposal to accept NEP 15: Merging multiarray and umath
by Nathaniel Smith 29 Jun '18

29 Jun '18

Hi all, I propose that we accept NEP 15: Merging multiarray and umath: http://www.numpy.org/neps/nep-0015-merge-multiarray-umath.html The core part of this proposal was uncontroversial. The main point of discussion was whether it was OK to deprecate set_numeric_ops, or whether it had some legitimate use cases. The conclusion was that in all the cases where set_numeric_ops is useful, PyUFunc_ReplaceLoopBySignature is a strictly better alternative, so there's no reason not to deprecate set_numeric_ops. So at this point I think the whole proposal is uncontroversial, and we can go ahead and accept it. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details: http://www.numpy.org/neps/nep-0000.html -n -- Nathaniel J. Smith -- https://vorpus.org

2 3

Github down on comcast
by Charles R Harris 29 Jun '18

29 Jun '18

Hi All, Just a note for those who may be having a problem reaching Github, it is currently down for comcast users. See http://downdetector.com/status/github/map/. Chuck

2 1