Mailman 3 November 2016 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

automatically avoiding temporary arrays
by Julian Taylor 27 Feb '17

27 Feb '17

hi, Temporary arrays generated in expressions are expensive as the imply extra memory bandwidth which is the bottleneck in most numpy operations. For example: r = a + b + c creates the b + c temporary and then adds a to it. This can be rewritten to be more efficient using inplace operations: r = b + c r += a This saves some memory bandwidth and can speedup the operation by 50% for very large arrays or even more if the inplace operation allows it to be completed completely in the cpu cache. The problem is that inplace operations are a lot less readable so they are often only used in well optimized code. But due to pythons refcounting semantics we can actually do some inplace conversions transparently. If an operand in python has a reference count of one it must be a temporary so we can use it as the destination array. CPython itself does this optimization for string concatenations. In numpy we have the issue that we can be called from the C-API directly where the reference count may be one for other reasons. To solve this we can check the backtrace until the python frame evaluation function. If there are only numpy and python functions in between that and our entry point we should be able to elide the temporary. This PR implements this: https://github.com/numpy/numpy/pull/7997 It currently only supports Linux with glibc (which has reliable backtraces via unwinding) and maybe MacOS depending on how good their backtrace is. On windows the backtrace APIs are different and I don't know them but in theory it could also be done there. A problem is that checking the backtrace is quite expensive, so should only be enabled when the involved arrays are large enough for it to be worthwhile. In my testing this seems to be around 180-300KiB sized arrays, basically where they start spilling out of the CPU L2 cache. I made a little crappy benchmark script to test this cutoff in this branch: https://github.com/juliantaylor/numpy/tree/elide-bench If you are interested you can run it with: python setup.py build_ext -j 4 --inplace ipython --profile=null check.ipy At the end it will plot the ratio between elided and non-elided runtime. It should get larger than one around 180KiB on most cpus. If no one points out some flaw in the approach, I'm hoping to get this into the next numpy version. cheers, Julian

11 22

__numpy_ufunc__
by Charles R Harris 22 Feb '17

22 Feb '17

Hi All, For those interested in continuing the __numpy_ufunc__ saga, there is a pull request enabling it <https://github.com/numpy/numpy/pull/8247>. Likely we will want to make some changes up front before merging that, so some discussion is in order. Chuck

5 8

Re: [Numpy-discussion] Integers to negative integer powers, time for a decision.
by Peter Creasey 03 Jan '17

03 Jan '17

> On Sun, Oct 9, 2016 at 12:59 PM, Stephan Hoyer <shoyer(a)gmail.com> wrote: > >> >> I agree with Sebastian and Nathaniel. I don't think we can deviating from >> the existing behavior (int ** int -> int) without breaking lots of existing >> code, and if we did, yes, we would need a new integer power function. >> >> I think it's better to preserve the existing behavior when it gives >> sensible results, and error when it doesn't. Adding another function >> float_power for the case that is currently broken seems like the right way >> to go. >> > I actually suspect that the amount of code broken by int**int->float may be relatively small (though extremely annoying for those that it happens to, and it would definitely be good to have statistics). I mean, Numpy silently transitioned to int32+uint64->float64 not so long ago which broke my code, but the world didn’t end. If the primary argument against int**int->float seems to be the difficulty of managing the transition, with int**int->Error being the seen as the required yet *very* painful intermediate step for the large fraction of the int**int users who didn’t care if it was int or float (e.g. the output is likely to be cast to float in the next step anyway), and fail loudly for those users who need int**int->int, then if you are prepared to risk a less conservative transition (i.e. we think that latter group is small enough) you could skip the error on users and just throw a warning for a couple of releases, along the lines of: WARNING int**int -> int is going to be deprecated in favour of int**int->float in Numpy 1.16. To avoid seeing this message, either use “from numpy import __future_float_power__” or explicitly set the type of one of your inputs to float, or use the new ipower(x,y) function for integer powers. Peter

5 7

Does numpy.bincount support numpy.float128 type weights?
by Wei, Huayi 30 Nov '16

30 Nov '16

Hi, There, Here is a sample code using `numpy.bincount` import numpy as np a = np.array([1.0, 2.0, 3.0], dtype=np.float128) b = np.array([1, 2, 0], dtype=np.int) c = np.bincount(b, weights=a) If run it, I get the following error report: ----> 1 c = np.bincount(b, weights=a) TypeError: Cannot cast array data from dtype('float128') to dtype('float64') according to the rule 'safe' Is it a bug of `np.bincount`? Does there exist any similar function which I can use to do the similar thing with numpy.float128 type weights? Best Huayi

4 3

Re: [Numpy-discussion] Wheel files for PPC64
by Leonardo Bianconi 29 Nov '16

29 Nov '16

Hi. Can anyone help me with this? > > From: Leonardo Bianconi > Sent: quarta-feira, 23 de novembro de 2016 12:47 > To: 'numpy-discussion(a)scipy.org' <numpy-discussion(a)scipy.org> > Subject: Wheel files for PPC64 > > Hi all! > > I realized that there is no "whl" files for PPC64 (https://pypi.python.org/pypi/numpy/1.12.0b1), > which makes numpy be compiled when installing it with pip. It is causing a low performance on > this architecture when the libopenblas is not previously installed, while for x86_64 it is installed > as a lib, which is in the "whl" file. > > What must to be done to add a build of numpy for PPC64, and upload the "whl" files to pypi > link?

2 1

Wheel files for PPC64
by Leonardo Bianconi 23 Nov '16

23 Nov '16

Hi all! I realized that there is no "whl" files for PPC64 (https://pypi.python.org/pypi/numpy/1.12.0b1), which makes numpy be compiled when installing it with pip. It is causing a low performance on this architecture when the libopenblas is not previously installed, while for x86_64 it is installed as a lib, which is in the "whl" file. What must to be done to add a build of numpy for PPC64, and upload the "whl" files to pypi link?

2 1

PyPy wheels (was: NumPy 1.12.0b1 released)
by Ralf Gommers 18 Nov '16

18 Nov '16

On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <njs(a)pobox.com> wrote: > Another thing to think about is that 1.12 on pypy won't pass its test > suite (though it's close), and we're not yet testing new PRs on pypy, so no > guarantees about 1.13 yet. I think on balance these probably aren't reasons > *not* to upload wheels, but it's a funny place where we're talking about > providing "official" builds even though it's not an "officially supported > platform". So we will at least want to be clear about that. And someone > will have to handle the bug reports about the test suite failing :-). > Those are good points. We could run PyPy on TravisCI; the PyPy install and numpy build aren't difficult anymore. Handling bug reports is mostly checking if it's PyPy specific, and if so refer to the PyPy tracker I'd think. It is some work, but given that PyPy has finally chosen a way to support Numpy that's not a dead end and has come quite a long way quite quickly, taking on that bit of extra work as Numpy maintainers is a good time investment imho. Many bug reports will go straight to PyPy though I expect, because often that is the obvious place to go. This is what I just got from downloading the OS X PyPy binary and pip installing numpy master: Python version 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] nose version 1.3.7 .................................S.......................................................E........................................................................................S.............................................................RPython traceback: File "pypy_interpreter.c", line 43348, in BuiltinCodePassThroughArguments1_funcrun_obj File "pypy_module_cpyext_4.c", line 16627, in generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_17 Fatal RPython error: AssertionError Abort trap: 6 So guess I'll go find out where that issue tracker is:) Ralf On Nov 18, 2016 01:14, "Ralf Gommers" <ralf.gommers(a)gmail.com> wrote: > > > > > > > > On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett <matthew.brett(a)gmail.com> > wrote: > >> > >> Hi, > >> > >> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus <matti.picus(a)gmail.com> > wrote: > >> > Congrats to all on the release.Two questions: > >> > > >> > Is there a guide to building standard wheels for NumPy? > >> > >> I don't think so - there is a repository that we use to build the > >> wheels, that has the Windows, OSX and manyllinux recipes for the > >> standard CPython build: > >> > >> https://github.com/MacPython/numpy-wheelso > >> > >> If you can work out a way to automate the PyPy builds and tests - > >> especially using the same repo - that would be very useful. > >> > >> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 > and > >> > OSX64, how can I get them blessed and uploaded to PyPI? > >> > >> If you can automate the build and tests, I'm guessing there will be no > >> objections - but it's not my call... > > > > > > I'm in favor, assuming that the wheel tags and PyPy backwards > compatibility situation is OK. Can't really find any examples. What I mean > is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels > should have tags "pp<something>". Are the PyPy cpyext layer and the > <something> defined such that a new PyPy release won't break older wheels? > >

2 2

Re: [Numpy-discussion] NumPy 1.12.0b1 released
by Matti Picus 18 Nov '16

18 Nov '16

Congrats to all on the release.Two questions: Is there a guide to building standard wheels for NumPy? Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and OSX64, how can I get them blessed and uploaded to PyPI? Matti On 17/11/16 07:47, numpy-discussion-request(a)scipy.org wrote: > Date: Wed, 16 Nov 2016 22:47:39 -0700 > From: Charles R Harris<charlesr.harris(a)gmail.com> > To: numpy-discussion<numpy-discussion(a)scipy.org>, SciPy Users List > <scipy-user(a)scipy.org>, SciPy Developers List<scipy-dev(a)scipy.org>, > python-announce-list(a)python.org > Subject: [Numpy-discussion] NumPy 1.12.0b1 released. > > Hi All, > > I'm pleased to annouce the release of NumPy 1.12.0b1. This release > supports Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests > submitted by 133 contributors. It is quite sizeable and rather than put the > release notes inline I've attached them as a file and they may also be > viewed at Github<https://github.com/numpy/numpy/releases/tag/v1.12.0b1>. > Zip files and tarballs may also be found the Github link. Wheels and source > archives may be downloaded from PyPI, which is the recommended method. > > This release is a large collection of fixes, enhancements, and improvements > and it is difficult to select just a few as highlights. However, the > following enhancements may be of particular interest > > - Order of operations in ``np.einsum`` now can be optimized for large > speed improvements. > - New ``signature`` argument to ``np.vectorize`` for vectorizing with > core dimensions. > - The ``keepdims`` argument was added to many functions. > - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this > is a milestone for PyPy's C-API compatibility layer. > > Thanks to all, > > Chuck

6 6

ANN: xtensor and xtensor-python
by Sylvain Corlay 18 Nov '16

18 Nov '16

Hi all, We are pleased to announce the release of the xtensor <https://github.com/QuantStack/xtensor> library and its python bindings xtensor-python <https://github.com/QuantStack/xtensor-python>, by Johan Mabille (@JohanMabille) and Sylvain Corlay (@SylvainCorlay). <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#1-xt…> 1. xtensor xtensor <https://github.com/QuantStack/xtensor> is a C++ template library for manipulating multi-dimensional array expressions. It provides - an API following the idioms of the *C++ standard library*. - an extensible expression system enabling lazy broadcasting and universal functions. - python bindings for operating inplace on numpy arrays in your C++ thanks to Python's buffer protocol and pybind. More details on lazy computing with xtensor are available below. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#2-xt…> 2. xtensor-cookiecutter Besides, xtensor <https://github.com/QuantStack/xtensor> and xtensor-python <https://github.com/QuantStack/xtensor-python>, we provide a cookiecutter template project to help extension authors get started. The xtensor-cookiecutter <https://github.com/QuantStack/xtensor-python> generates a simple project for a Python C++ extension with - a working setup.py compiling the extension module - a few example of functions making use of xtensor exposed to python, and the example of a vectorized function exposed to python. - all the boilerplate for the setup of the unit testing and generation of the HTML documentation for these examples. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#3-yo…>3. You can try it now! You can try the xtensor live on the project website <http://quantstack.net/xtensor>. The Try it Now button is powered by - The Cling C++ interpreter. - The Jupyter notebook. - The Binder project. ------------------------------ ------------------------------ <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#gett…>Getting started xtensor requires a modern C++ compiler supporting C++14. The following C+ compilers are supported: - On Windows platforms, Visual C++ 2015 Update 2, or more recent - On Unix platforms, gcc 4.9 or a recent version of Clang <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#inst…> Installation xtensor and xtensor-python are header-only libraries. We provide packages for the conda package manager. conda install -c conda-forge xtensor # installs xtensor conda install -c conda-forge xtensor-python # installs xtensor and xtensor-python <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#usage> Usage <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#basi…>Basic Usage Initialize a 2-D array and compute the sum of one of its rows and a 1-D array. #include <iostream> #include "xtensor/xarray.hpp" #include "xtensor/xio.hpp" xt::xarray<double> arr1 {{1.0, 2.0, 3.0}, {2.0, 5.0, 7.0}, {2.0, 5.0, 7.0}}; xt::xarray<double> arr2 {5.0, 6.0, 7.0}; xt::xarray<double> res = xt::make_xview(arr1, 1) + arr2; std::cout << res; Outputs: {7, 11, 14} Initialize a 1-D array and reshape it inplace. #include <iostream> #include "xtensor/xarray.hpp" #include "xtensor/xio.hpp" xt::xarray<int> arr {1, 2, 3, 4, 5, 6, 7, 8, 9}; arr.reshape({3, 3}); std::cout << arr; Outputs: {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}} <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#lazy…>Lazy Broadcasting with xtensor We can operate on arrays of different shapes of dimensions in an elementwise fashion. Broadcasting rules of xtensor are similar to those of numpy <http://www.numpy.org/> and libdynd <http://libdynd.org/>. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#broa…>Broadcasting rules In an operation involving two arrays of different dimensions, the array with the lesser dimensions is broadcast across the leading dimensions of the other. For example, if A has shape (2, 3), and B has shape (4, 2, 3), the result of a broadcasted operation with A and B has shape (4, 2, 3). (2, 3) # A (4, 2, 3) # B --------- (4, 2, 3) # Result The same rule holds for scalars, which are handled as 0-D expressions. If matched up dimensions of two input arrays are different, and one of them has size 1, it is broadcast to match the size of the other. Let's say B has the shape (4, 2, 1) in the previous example, so the broadcasting happens as follows: (2, 3) # A (4, 2, 1) # B --------- (4, 2, 3) # Result <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#univ…>Universal functions, Laziness and Vectorization With xtensor, if x, y and z are arrays of *broadcastable shapes*, the return type of an expression such as x + y * sin(z) is not an array. It is an xexpression object offering the same interface as an N-dimensional array, which does not hold the result. Values are only computed upon access or when the expression is assigned to an xarray object. This allows to operate symbolically on very large arrays and only compute the result for the indices of interest. We provide utilities to vectorize any scalar function (taking multiple scalar arguments) into a function that will perform on xexpressions, applying the lazy broadcasting rules which we just described. These functions are called *xfunction*s. They are xtensor's counterpart to numpy's universal functions. In xtensor, arithmetic operations (+, -, *, /) and all special functions are *xfunction*s. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#iter…>Iterating over xexpressions and Broadcasting Iterators All xexpressions offer two sets of functions to retrieve iterator pairs (and their const counterpart). - begin() and end() provide instances of xiterators which can be used to iterate over all the elements of the expression. The order in which elements are listed is row-major in that the index of last dimension is incremented first. - xbegin(shape) and xend(shape) are similar but take a *broadcasting shape* as an argument. Elements are iterated upon in a row-major way, but certain dimensions are repeated to match the provided shape as per the rules described above. For an expression e, e.xbegin(e.shape()) and e.begin() are equivalent. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#fixe…> Fixed-dimension *and* Dynamic dimension Two container classes implementing multi-dimensional arrays are provided: xarray and xtensor. - xarray can be reshaped dynamically to any number of dimensions. It is the container that is the most similar to numpy arrays. - xtensor has a dimension set at compilation time, which enables many optimizations. For example, shapes and strides of xtensor instances are allocated on the stack instead of the heap. xarray and xtensor container are both xexpressions and can be involved and mixed in universal functions, assigned to each other etc... Besides, two access operators are provided: - The variadic template operator() which can take multiple integral arguments or none. - And the operator[] which takes a single multi-index argument, which can be of size determined at runtime. operator[] also supports access with braced initializers. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#pyth…>Python bindings The python bindings are built upon the pybind11 <https://github.com/pybind/pybind11> library, a lightweight header-only for creating bindings between the Python and C++ programming languages. <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#exam…>Example 1: Use an algorithm of the C++ library on a numpy array inplace. C++ code #include <numeric> // std::accumulate #include "pybind11/pybind11.h" // pybind11 #include "xtensor/xmath.hpp" // C++ universal functions #include "xtensor-python/pyarray.hpp" // numpy bindings double sum_of_sines(xt::pyarray<double> &m) { auto sines = xt::sin(m); // sines does not hold any value return std::accumulate(sines.begin(), sines.end(), 0.0); } PYBIND11_PLUGIN(xtensor_python_test) { pybind11::module m("xtensor_python_test", "Test module for xtensor python bindings"); m.def("sum_of_sines", sum_of_sines, "Return the sum of the sines"); return m.ptr(); } Python Code import numpy as npimport xtensor_python_test as xt a = np.arange(15).reshape(3, 5) s = xt.sum_of_sines(v) s Outputs 1.2853996391883833 <https://gist.github.com/SylvainCorlay/acbb515a5f421897b9254bcb2a65b2b8#exam…>Example 2: Create a universal function from a C++ scalar function C++ code #include "pybind11/pybind11.h" #include "xtensor-python/pyvectorize.hpp" #include <numeric> #include <cmath> namespace py = pybind11; double scalar_func(double i, double j) { return std::sin(i) - std::cos(j); } PYBIND11_PLUGIN(xtensor_python_test) { py::module m("xtensor_python_test", "Test module for xtensor python bindings"); m.def("vectorized_func", xt::pyvectorize(scalar_func), ""); return m.ptr(); } Python Code import numpy as npimport xtensor_python_test as xt x = np.arange(15).reshape(3, 5) y = [1, 2, 3] z = xt.vectorized_func(x, y) z Outputs [[-1. , 0.30116, 1.32544, 1.13111, -0.10315], [-1.95892, -0.81971, 1.07313, 1.97935, 1.06576], [-1.54402, -1.54029, -0.12042, 1.41016, 1.64425]]

1 0

Status of NAs for integer arrays
by Dmitrii Izgurskii 18 Nov '16

18 Nov '16

Hi, I would like to know where I can follow the progress of what's being done on implementing missing values in integer arrays. Some threads I found are from 2012. Is really nothing being done on this issue?

2 1

The --compilers flag is ignored for build_ext
by Andres Guzman-Ballen 17 Nov '16

17 Nov '16

Hello Numpy community! First time poster here. I was recommended by Charles Harris to present a question I have to you guys regarding using --compiler=intelem being ignored when I use build_ext. I am trying to compile Numpy v1.11.2 that I am getting from https://pypi.python.org/packages/16/f5/b432f028134dd30cfbf6f21b8264a9938e5e…, using Python 2.7.12 on my Ubuntu 10.04 LTS Machine. I specifically want to compile this with the Intel compiler. I have sourced the compilervars.sh so the Intel compiler is in the path. This is the command that is failing: 'python setup.py build_ext --inplace --debug --force --compiler=intelem --fcompiler=intelem'. The reason it fails is because I have added compiler flags to umath's configuration extension code in numpy/core/setup.py as extra_compiler_arg and they are specifically meant for Intel. Since the compiler setup is ignored (unlike in setup.py build, where config_cc is correctly called and takes care of this), it defaults to GCC. This is a snippet of what you get for 'build_ext': running build_clib running build_src build_src building py_modules sources building library "npymath" sources creating build creating build/src.linux-x86_64-2.7 customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config This is what you get for using 'build': running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build/src.linux-x86_64-2.7 creating build/src.linux-x86_64-2.7/numpy creating build/src.linux-x86_64-2.7/numpy/distutils building library "npymath" sources Found executable /localdisk/psxe_16/compilers_and_libraries_2016.3.210/linux/bin/intel64/icc Could not locate executable ecc customize IntelEM64TFCompiler Found executable /localdisk/psxe_16/compilers_and_libraries_2016.3.210/linux/bin/intel64/ifort customize IntelEM64TFCompiler using config I do not experience this problem with 'python setup.py build --debug --force --compiler=intelem --fcompiler=intelem'. because the configuration function is correctly called here: https://github.intel.com/SAT/numpy/blob/develop/numpy/distutils/command/bui… (that is where I got the output from above). However, I do not see anything like this in build_ext: https://github.intel.com/SAT/numpy/blob/develop/numpy/distutils/command/bui…. This of course could be because build_ext is doing something else below in the run command on line 109. This is most likely not the right answer, but what harm does this do? diff --git a/numpy/distutils/command/build_ext.py b/numpy/distutils/command/build_ext.py index 0fa52a2..ffa88ae 100644 --- a/numpy/distutils/command/build_ext.py +++ b/numpy/distutils/command/build_ext.py @@ -79,6 +79,8 @@ class build_ext (old_build_ext): return # Make sure that extension sources are complete. + self.run_command('config_cc') + self.run_command('config_fc') self.run_command('build_src') if self.distribution.has_c_libraries(): I am able to get build_ext to behave properly by adding this snippet, but I'm pretty sure this is the wrong approach because there seems to be code below line 109 in numpy/distutils/command/build_ext.py that sets compiler-related things up. This is the ticket I created under Numpy's issues page: https://github.com/numpy/numpy/issues/8283. Any feedback is very well appreciated and I'd be more than happy to help contribute and get this fixed for the community. I don't want to just throw a problem at you guys and expect someone else to fix it for me. Thanks! Andres

1 0

NumPy 1.12.0b1 released.
by Charles R Harris 17 Nov '16

17 Nov '16

Hi All, I'm pleased to annouce the release of NumPy 1.12.0b1. This release supports Python 2.7 and 3.4 - 3.6 and is the result of 388 pull requests submitted by 133 contributors. It is quite sizeable and rather than put the release notes inline I've attached them as a file and they may also be viewed at Github <https://github.com/numpy/numpy/releases/tag/v1.12.0b1>. Zip files and tarballs may also be found the Github link. Wheels and source archives may be downloaded from PyPI, which is the recommended method. This release is a large collection of fixes, enhancements, and improvements and it is difficult to select just a few as highlights. However, the following enhancements may be of particular interest - Order of operations in ``np.einsum`` now can be optimized for large speed improvements. - New ``signature`` argument to ``np.vectorize`` for vectorizing with core dimensions. - The ``keepdims`` argument was added to many functions. - Support for PyPy 2.7 v5.6.0 has been added. While not complete, this is a milestone for PyPy's C-API compatibility layer. Thanks to all, Chuck

1 0

ufunc for sum of squared difference
by Matthew Harrigan 16 Nov '16

16 Nov '16

I was reading this <http://notes-on-cython.readthedocs.io/en/latest/std_dev.html> and got thinking about if a ufunc could compute the sum of squared differences in a single pass without a temporary array. The python code below demonstrates a possible approach. import numpy as np x = np.arange(10) c = 1.0 def add_square_diff(x1, x2): return x1 + (x2-c)**2 ufunc = np.frompyfunc(add_square_diff, 2, 1) print(ufunc.reduce(x) - x[0] + (x[0]-c)**2) print(np.sum(np.square(x-c))) I have (at least) 4 questions: 1. Is it possible to pass run time constants to a ufunc written in C for use in its inner loop, and if so how? 2. Is it possible to pass an initial value to reduce to avoid the clean up required for the first element? 3. Does that ufunc work, or are there special cases which cause it to fall apart? 4. Would a very specialized ufunc such as this be considered for incorporating in numpy since it would help reduce time and memory of functions already in numpy? Thank you, Matt

6 12

subclassing ndarray and keeping same ufunc behavior
by Stuart Reynolds 15 Nov '16

15 Nov '16

I'm trying to subclass an ndarray so that I can add some additional fields. When I do this however, I get new odd behavior when my object is passed to a variety of numpy functions. For example nanmin returns now return an object of the type of my new array class, whereas previously I'd get a float64. Why? Is this a bug with nanmin or my class? import numpy as np class NDArrayWithColumns(np.ndarray): def __new__(cls, obj, columns=None): obj = obj.view(cls) obj.columns = tuple(columns) return obj def __array_finalize__(self, obj): if obj is None: return self.columns = getattr(obj, 'columns', None) NAN = float("nan") r = np.array([1.,0.,1.,0.,1.,0.,1.,0.,NAN, 1., 1.])print "MIN", np.nanmin(r), type(np.nanmin(r)) gives: MIN 0.0 <type 'numpy.float64'> but >>> r = NDArrayWithColumns(r, ["a"])>>> print "MIN", np.nanmin(r), type(np.nanmin(r)) MIN 0.0 <class '__main__.NDArrayWithColumns'>>>> print r.shape # ?!(11,) Note the change in type, and also that str(np.nanmin(r)) shows 1 field, not 11 as indicated by its shape. This seems wrong. Is there a way to get my subclass to behave more like an ndarray? I realize from the docs that I can override __array_wrap__, but its not clear me how how to use it to solve this issue. Or whether its the right tool. In case you're interested, I'm subclassing because I'd like to track column names in matrices of a single type. This is pretty common wish in scikit pipelines. Structured arrays and record type arrays allow for varying type. Pandas provides this functionality, but dealing with numpy arrays is easier (and more efficient) when writing cython extensions. Also, I think the structured arrays and record types are unlikely to play nice with cython because they're more freely typed -- I want to deal exclusively with arrays of doubles. Any thoughts of how to subclass ndarray and keep original behavior in ufuncs?

3 3

Ensuring one can operate on array-like argument in place
by Pavlyk, Oleksandr 14 Nov '16

14 Nov '16

Hi, In my Cython code a function processes it argument x as follows: x_arr = PyArray_CheckFromAny( x, NULL, 0, 0, cnp.NPY_ELEMENTSTRIDES | cnp.NPY_ENSUREARRAY | cnp.NPY_NOTSWAPPED, NULL) if x_arr is not x: in_place = 1 # a copy was made, so we can work in place. The logic is of the last line turns out to be incorrect, because the input x can be a class with an array interface: class FakeArray(object): def __init__(self, data): self._data = data self.__array_interface__ = data.__array_interface__ Feeding my function FakeArray(xx), x_arr will point into the content of xx, resulting in unwarranted content overwrite of xx. I am trying to replace that condition with if x_arr is not x and cnp.PyArray_Check(x): # a copy was made, so we can work in place. in_place = 1 if cnp.PyArray_CHKFLAGS(x_arr, cnp.NPY_WRITEABLE) else 0 I am wondering if I perhaps overlooked some case. Thank you, Sasha

3 2

Re: [Numpy-discussion] Numpy 1.12.x branched
by Matti Picus 10 Nov '16

10 Nov '16

On 07/11/16 10:19, numpy-discussion-request(a)scipy.org wrote: > Date: Sun, 06 Nov 2016 17:56:12 +0100 > From: Sebastian Berg<sebastian(a)sipsolutions.net> > To:numpy-discussion@scipy.org > Subject: Re: [Numpy-discussion] Numpy 1.12.x branched > Message-ID:<1478451372.3875.5.camel(a)sipsolutions.net> > Content-Type: text/plain; charset="utf-8" > > On Sa, 2016-11-05 at 17:04 -0600, Charles R Harris wrote: >> >Hi All, >> > >> >Numpy 1.12.x has been branched and the 1.13 development branch is >> >open. It would be helpful if folks could review the release notes as >> >it is likely I've missed something.? I'd like to make the first beta >> >release in a couple of days. >> > > Very cool, thanks for all the hard work! > > - Sebastian > > >> >Chuck Thanks for managing this. I don't know where, but it would be nice if the release notes could mention the PyPy support - we are down to only a few failures on the test suite, the only real oustanding issue is nditer using UPDATEIFCOPY which depends on refcounting semantics to trigger the copy. Other than that PyPy + NumPy 1.12 is a working thing, we (PyPy devs) will soon try to make it work faster :). Matti

3 3

Numpy 1.12.x branched
by Charles R Harris 06 Nov '16

06 Nov '16

Hi All, Numpy 1.12.x has been branched and the 1.13 development branch is open. It would be helpful if folks could review the release notes as it is likely I've missed something. I'd like to make the first beta release in a couple of days. Chuck

2 1

Re: [Numpy-discussion] array comprehension
by Nathaniel Smith 04 Nov '16

04 Nov '16

Are you sure fromiter doesn't make an intermediate list or equivalent? It has to collect all the values before it can know the shape or dtype of the array to put them in. On Nov 4, 2016 5:26 AM, "Francesc Alted" <faltet(a)gmail.com> wrote: 2016-11-04 13:06 GMT+01:00 Neal Becker <ndbecker2(a)gmail.com>: > I find I often write: > np.array ([some list comprehension]) > > mainly because list comprehensions are just so sweet. > > But I imagine this isn't particularly efficient. > Right. Using a generator and np.fromiter() will avoid the creation of the intermediate list. Something like: np.fromiter((i for i in range(x))) # use xrange for Python 2 > > I wonder if numpy has a "better" way, and if not, maybe it would be a nice > addition? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Francesc Alted _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

3 3

ANN: pandas v0.19.1 released!
by Joris Van den Bossche 04 Nov '16

04 Nov '16

Hi all, I'm pleased to announce the release of pandas 0.19.1. This is a bug-fix release from 0.19.0 and includes some small regression fixes, bug fixes and performance improvements. We recommend that all users upgrade to this version. See the v0.19.1 Whatsnew page <http://pandas.pydata.org/pandas-docs/version/0.19.1/whatsnew.html> for an overview of all bugs that have been fixed in 0.19.1. Thanks to all contributors! Joris --- *How to get it:* Source tarballs and windows/mac/linux wheels are available on PyPI (thanks to Christoph Gohlke for the windows wheels, and to Matthew Brett for setting up the mac/linux wheels). Conda packages are already available via the conda-forge channel (conda install pandas -c conda-forge). It will be available on the main channel shortly. *Issues:* Please report any issues on our issue tracker: https://github.com/pydata/ pandas/issues *Thanks to all the contributors of the 0.19.1 release:* - Adam Chainz - Anthonios Partheniou - Arash Rouhani - Ben Kandel - Brandon M. Burroughs - Chris - chris-b1 - Chris Warth - David Krych - dubourg - gfyoung - Iván Vallés Pérez - Jeff Reback - Joe Jevnik - Jon M. Mease - Joris Van den Bossche - Josh Owen - Keshav Ramaswamy - Larry Ren - mattrijk - Michael Felt - paul-mannino - Piotr Chromiec - Robert Bradshaw - Sinhrks - Thiago Serafim - Tom Bird

1 0

array comprehension
by Neal Becker 04 Nov '16

04 Nov '16

I find I often write: np.array ([some list comprehension]) mainly because list comprehensions are just so sweet. But I imagine this isn't particularly efficient. I wonder if numpy has a "better" way, and if not, maybe it would be a nice addition?

6 8

Branching NumPy 1.12.x
by Charles R Harris 04 Nov '16

04 Nov '16

Hi All, I'm thinking that it is time to branch NumPy 1.12.x. I haven't got everything in it that I would have liked, in particular __numpy_ufunc__, but I think there is plenty of material and not branching is holding up some of the more risky stuff. My current thoughts on __numpy_ufunc__ is that it would be best to work it out over the 1.13.0 release cycle, starting with enabling it again right after the branch. Julian's work on avoiding temporary copies and Pauli's overlap handling PR are two other changes I've been putting off but don't want to delay further. There are some other smaller things that I had scheduled for 1.12.0, but would like to spend more time looking at. If there are some things that you think just have to be in 1.12.0, please mention them, but I'd rather aim at getting 1.13.0 out in a timely manner. Thoughts? Chuck

2 1

Useless but tricky exercise
by Nicolas P. Rougier 03 Nov '16

03 Nov '16

Hi all, Given an array V that is a view of a base B, I was wondering if it is possible to find a (string) index such that `V = B[index]` For example: ``` B = np.arange(8*8).reshape(8,8) V = B[::2,::2] index = find_view(B,V) print(index) "::2,::2" print(np.allclose(V, eval("B[%s]" % index))) True ``` I wrote a solution at https://gist.github.com/rougier/b8c2256434b3a4a4271260cd4cc6cbc7 (not very thoroughly tested) but maybe there are better ways to do that (like a magic numpy call ?) (no use-case at all, only for teaching) Nicolas

1 0

Using numpy.rint() with scalars
by Yuri Sukhov 03 Nov '16

03 Nov '16

Hi all, According to the documentation for numpy.rint() (https://docs.scipy.org/doc/ numpy/reference/generated/numpy.rint.html), it's a ufunc that accepts an array-like object as an input. But it also works with scalar inputs. Could anyone clarify if such use case is considered to be common and acceptable? It it just the documentation that does not cover one of the possible scenarios, or is it a side effect and such use case should be avoided? I want to use rint() with scalars as I have not found an alternative with the same behavior in cPython, but if it's not how it should be used, I don't want to rewrite the app when that behavior will change. Thanks! Yuri.

2 1

missing from contributor list?
by Sturla Molden 03 Nov '16

03 Nov '16

Why am I missing from the contributor hist here? https://github.com/numpy/numpy/blob/master/numpy/_build_utils/src/apple_sge… Sturla

6 6