NumPy-Discussion October 2019

numpy-discussion@python.org

24 participants
28 discussions

Re: [Numpy-discussion] example reading binary Fortran file

by Neil Martinsen-Burrell

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

4 months, 2 weeks

argmax() indexes to value

by Daniele Nicolodi

Hello, this is a very basic question, but I cannot find a satisfying answer. Assume a is a 2D array and that I get the index of the maximum value along the second dimension: i = a.argmax(axis=1) Is there a better way to get the value of the maximum array entries along the second axis other than: v = a[np.arange(len(a)), i] ?? Thank you. Cheers, Daniele

2 years, 1 month

Transonic Vision: unifying Python-Numpy accelerators

by PIERRE AUGIER

Dear Python-Numpy community, Few years ago I started to use a lot Python and Numpy for science. I'd like to thanks all people who contribute to this fantastic community. I used a lot Cython, Pythran and Numba and for the FluidDyn project, we created Transonic, a pure Python package to easily accelerate modern Python-Numpy code with different accelerators. We wrote a long and serious text to explain why we think Transonic could have a positive impact on the scientific Python ecosystem. Here it is: http://tiny.cc/transonic-vision Feedback and discussions would be greatly appreciated! Pierre -- Pierre Augier - CR CNRS http://www.legi.grenoble-inp.fr LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels BP53, 38041 Grenoble Cedex, France tel:+33.4.56.52.86.16

2 years, 1 month

Numpy community meeting Wed Oct 30

by Matti Picus

Hi all, There will be a NumPy Community meeting Wednesday Oct 30 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Matti

2 years, 1 month

NEP 34 - deprecate automatic dtype=object on ragged arrays

by Matti Picus

After a few iterations by reviewers, I would like to submit NEP 34 to deprecate automatically using dtype=object for ragged arrays. https://github.com/numpy/numpy/pull/14674 and an associated PR for the implementation https://github.com/numpy/numpy/pull/14794 Commments? Matti Abstract -------- When users create arrays with sequences-of-sequences, they sometimes err in matching the lengths of the nested sequences_, commonly called "ragged arrays". Here we will refer to them as ragged nested sequences. Creating such arrays via ``np.array([<ragged_nested_sequence>])`` with no ``dtype`` keyword argument will today default to an ``object``-dtype array. Change the behaviour to raise a ``ValueError`` instead. Motivation and Scope -------------------- Users who specify lists-of-lists when creating a `numpy.ndarray` via ``np.array`` may mistakenly pass in lists of different lengths. Currently we accept this input and automatically create an array with ``dtype=object``. This can be confusing, since it is rarely what is desired. Changing the automatic dtype detection to never return ``object`` for ragged nested sequences (defined as a recursive sequence of sequences, where not all the sequences on the same level have the same length) will force users who actually wish to create ``object`` arrays to specify that explicitly. Note that ``lists``, ``tuples``, and ``nd.ndarrays`` are all sequences [0]_. See for instance `issue 5303`_. Usage and Impact ---------------- After this change, array creation with ragged nested sequences must explicitly define a dtype: >>> np.array([[1, 2], [1]]) ValueError: cannot guess the desired dtype from the input >>> np.array([[1, 2], [1]], dtype=object) # succeeds, with no change from current behaviour The deprecation will affect any call that internally calls ``np.asarray``. For instance, the ``assert_equal`` family of functions calls ``np.asarray``, so users will have to change code like:: np.assert_equal(a, [[1, 2], 3]) to:: np.assert_equal(a, np.array([[1, 2], 3], dtype=object)

2 years, 1 month

NumPy amuse-bouche

by Inessa Pawson

We are looking for code snippets that illustrate NumPy’s unique capabilities. To submit your ideas or for further info please refer to the following GitHub discussion: github.com/numpy/numpy.org/issues/40#issuecomment-534303380 -- Every good wish, Inessa Pawson NumPy Web Team

2 years, 1 month

New draft of NEP 31 — Context-local and global overrides of the NumPy API

by Hameer Abbasi

Hello everyone, I’ve improved upon the content of NEP 31 to make it simpler, and also according to the new NEP template, only part of the NEP is being sent out to the mailing list. For the full nep, please see PR 14793<https://github.com/numpy/numpy/pull/14793>. ============================================================ NEP 31 — Context-local and global overrides of the NumPy API ============================================================ :Author: Hameer Abbasi <habbasi(a)quansight.com> :Author: Ralf Gommers <rgommers(a)quansight.com> :Author: Peter Bell <pbell(a)quansight.com> :Status: Draft :Type: Standards Track :Created: 2019-08-22 Abstract -------- This NEP proposes to make all of NumPy's public API overridable via an extensible backend mechanism. Acceptance of this NEP means NumPy would provide global and context-local overrides in a separate namespace, as well as a dispatch mechanism similar to NEP-18 [2]_. First experiences with ``__array_function__`` show that it is necessary to be able to override NumPy functions that *do not take an array-like argument*, and hence aren't overridable via ``__array_function__``. The most pressing need is array creation and coercion functions, such as ``numpy.zeros`` or ``numpy.asarray``; see e.g. NEP-30 [9]_. This NEP proposes to allow, in an opt-in fashion, overriding any part of the NumPy API. It is intended as a comprehensive resolution to NEP-22 [3]_, and obviates the need to add an ever-growing list of new protocols for each new type of function or object that needs to become overridable. Motivation and Scope -------------------- The primary end-goal of this NEP is to make the following possible: .. code:: python # On the library side import numpy.overridable as unp def library_function(array): array = unp.asarray(array) # Code using unumpy as usual return array # On the user side: import numpy.overridable as unp import uarray as ua import dask.array as da ua.register_backend(da) # Can be done within Dask itself library_function(dask_array) # works and returns dask_array with unp.set_backend(da): library_function([1, 2, 3, 4]) # actually returns a Dask array. Here, ``backend`` can be any compatible object defined either by NumPy or an external library, such as Dask or CuPy. Ideally, it should be the module ``dask.array`` or ``cupy`` itself. These kinds of overrides are useful for both the end-user as well as library authors. End-users may have written or wish to write code that they then later speed up or move to a different implementation, say PyData/Sparse. They can do this simply by setting a backend. Library authors may also wish to write code that is portable across array implementations, for example ``sklearn`` may wish to write code for a machine learning algorithm that is portable across array implementations while also using array creation functions. This NEP takes a holistic approach: It assumes that there are parts of the API that need to be overridable, and that these will grow over time. It provides a general framework and a mechanism to avoid a design of a new protocol each time this is required. This was the goal of ``uarray``: to allow for overrides in an API without needing the design of a new protocol. This NEP proposes the following: That ``unumpy`` [8]_ becomes the recommended override mechanism for the parts of the NumPy API not yet covered by ``__array_function__`` or ``__array_ufunc__``, and that ``uarray`` is vendored into a new namespace within NumPy to give users and downstream dependencies access to these overrides. This vendoring mechanism is similar to what SciPy decided to do for making ``scipy.fft`` overridable (see [10]_). The motivation behind ``uarray`` is manyfold: First, there have been several attempts to allow dispatch of parts of the NumPy API, including (most prominently), the ``__array_ufunc__`` protocol in NEP-13 [4]_, and the ``__array_function__`` protocol in NEP-18 [2]_, but this has shown the need for further protocols to be developed, including a protocol for coercion (see [5]_, [9]_). The reasons these overrides are needed have been extensively discussed in the references, and this NEP will not attempt to go into the details of why these are needed; but in short: It is necessary for library authors to be able to coerce arbitrary objects into arrays of their own types, such as CuPy needing to coerce to a CuPy array, for example, instead of a NumPy array. In simpler words, one needs things like ``np.asarray(...)`` or an alternative to "just work" and return duck-arrays. Usage and Impact ---------------- This NEP allows for global and context-local overrides, as well as automatic overrides a-la ``__array_function__``. Here are some use-cases this NEP would enable, besides the first one stated in the motivation section: The first is allowing alternate dtypes to return their respective arrays. .. code:: python # Returns an XND array x = unp.ones((5, 5), dtype=xnd_dtype) # Or torch dtype The second is allowing overrides for parts of the API. This is to allow alternate and/or optimised implementations for ``np.linalg``, BLAS, and ``np.random``. .. code:: python import numpy as np import pyfftw # Or mkl_fft # Makes pyfftw the default for FFT np.set_global_backend(pyfftw) # Uses pyfftw without monkeypatching np.fft.fft(numpy_array) with np.set_backend(pyfftw) # Or mkl_fft, or numpy # Uses the backend you specified np.fft.fft(numpy_array) This will allow an official way for overrides to work with NumPy without monkeypatching or distributing a modified version of NumPy. Here are a few other use-cases, implied but not already stated: .. code:: python data = da.from_zarr('myfile.zarr') # result should still be dask, all things being equal result = library_function(data) result.to_zarr('output.zarr') This second one would work if ``magic_library`` was built on top of ``unumpy``. .. code:: python from dask import array as da from magic_library import pytorch_predict data = da.from_zarr('myfile.zarr') # normally here one would use e.g. data.map_overlap result = pytorch_predict(data) result.to_zarr('output.zarr') Backward compatibility ---------------------- There are no backward incompatible changes proposed in this NEP.

2 years, 1 month

proposed NEP template change

by Ralf Gommers

Hi all, In https://github.com/numpy/numpy/pull/14734 we're working on a NEP template update, with the goal of better separating content that's relevant for users and authors of packages that depend on NumPy from implementation details. Please have a look and comment on the PR if you're interested in this topic. Cheers, Ralf

2 years, 1 month

Numpy community meeting Wed Oct 23

by Matti Picus

Hi all, There will be a NumPy Community meeting Wednesday Oct 23 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Matti

2 years, 1 month

by Pankaj Jangid

Is there an officially recommended way to utilize AMD GPUs via OpenCL, ROCm? I came across ROCm website https://rocm.github.io/. This has Tensorflow and PyTorch versions for using AMD GPUs. Just wanted to know if there is a way to use my AMD GPUs for NumPy calculations. -- Regards, Pankaj Jangid

2 years, 1 month

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion October 2019