Mailman 3 October 2016 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

automatically avoiding temporary arrays
by Julian Taylor 27 Feb '17

27 Feb '17

hi, Temporary arrays generated in expressions are expensive as the imply extra memory bandwidth which is the bottleneck in most numpy operations. For example: r = a + b + c creates the b + c temporary and then adds a to it. This can be rewritten to be more efficient using inplace operations: r = b + c r += a This saves some memory bandwidth and can speedup the operation by 50% for very large arrays or even more if the inplace operation allows it to be completed completely in the cpu cache. The problem is that inplace operations are a lot less readable so they are often only used in well optimized code. But due to pythons refcounting semantics we can actually do some inplace conversions transparently. If an operand in python has a reference count of one it must be a temporary so we can use it as the destination array. CPython itself does this optimization for string concatenations. In numpy we have the issue that we can be called from the C-API directly where the reference count may be one for other reasons. To solve this we can check the backtrace until the python frame evaluation function. If there are only numpy and python functions in between that and our entry point we should be able to elide the temporary. This PR implements this: https://github.com/numpy/numpy/pull/7997 It currently only supports Linux with glibc (which has reliable backtraces via unwinding) and maybe MacOS depending on how good their backtrace is. On windows the backtrace APIs are different and I don't know them but in theory it could also be done there. A problem is that checking the backtrace is quite expensive, so should only be enabled when the involved arrays are large enough for it to be worthwhile. In my testing this seems to be around 180-300KiB sized arrays, basically where they start spilling out of the CPU L2 cache. I made a little crappy benchmark script to test this cutoff in this branch: https://github.com/juliantaylor/numpy/tree/elide-bench If you are interested you can run it with: python setup.py build_ext -j 4 --inplace ipython --profile=null check.ipy At the end it will plot the ratio between elided and non-elided runtime. It should get larger than one around 180KiB on most cpus. If no one points out some flaw in the approach, I'm hoping to get this into the next numpy version. cheers, Julian

11 22

Re: [Numpy-discussion] Integers to negative integer powers, time for a decision.
by Peter Creasey 04 Jan '17

04 Jan '17

> On Sun, Oct 9, 2016 at 12:59 PM, Stephan Hoyer <shoyer(a)gmail.com> wrote: > >> >> I agree with Sebastian and Nathaniel. I don't think we can deviating from >> the existing behavior (int ** int -> int) without breaking lots of existing >> code, and if we did, yes, we would need a new integer power function. >> >> I think it's better to preserve the existing behavior when it gives >> sensible results, and error when it doesn't. Adding another function >> float_power for the case that is currently broken seems like the right way >> to go. >> > I actually suspect that the amount of code broken by int**int->float may be relatively small (though extremely annoying for those that it happens to, and it would definitely be good to have statistics). I mean, Numpy silently transitioned to int32+uint64->float64 not so long ago which broke my code, but the world didn’t end. If the primary argument against int**int->float seems to be the difficulty of managing the transition, with int**int->Error being the seen as the required yet *very* painful intermediate step for the large fraction of the int**int users who didn’t care if it was int or float (e.g. the output is likely to be cast to float in the next step anyway), and fail loudly for those users who need int**int->int, then if you are prepared to risk a less conservative transition (i.e. we think that latter group is small enough) you could skip the error on users and just throw a warning for a couple of releases, along the lines of: WARNING int**int -> int is going to be deprecated in favour of int**int->float in Numpy 1.16. To avoid seeing this message, either use “from numpy import __future_float_power__” or explicitly set the type of one of your inputs to float, or use the new ipower(x,y) function for integer powers. Peter

5 7

__numpy_ufunc__
by Charles R Harris 31 Oct '16

31 Oct '16

Hi All, Does anyone remember discussion of numpy scalars apropos __numpy_ufunc__? Chuck

3 8

Problem with compiling openacc with f2py
by Vikram Singh 30 Oct '16

30 Oct '16

I am a newbie to f2py so I have been creating simple test cases. Eventually I want to be able to use openacc subroutine from python. So here's the test case module test use iso_c_binding, only: sp => C_FLOAT, dp => C_DOUBLE, i8 => C_INT use omp_lib use openacc implicit none contains subroutine add_acc (a, b, n, c) integer(kind=i8), intent(in) :: n real(kind=dp), intent(in) :: a(n) real(kind=dp), intent(in) :: b(n) real(kind=dp), intent(out) :: c(n) integer(kind=i8) :: i !$acc kernels do i = 1, n c(i) = a(i) + b(i) end do !$acc end kernels end subroutine add_acc subroutine add_omp (a, b, n, c) integer(kind=i8), intent(in) :: n real(kind=dp), intent(in) :: a(n) real(kind=dp), intent(in) :: b(n) real(kind=dp), intent(out) :: c(n) integer(kind=i8) :: i, j !$omp parallel do do i = 1, n c(i) = a(i) + b(i) end do !$omp end parallel do end subroutine add_omp subroutine nt (c) integer(kind=i8), intent(out) :: c c = omp_get_max_threads() end subroutine nt subroutine mult (a, b, c) real(kind=dp), intent(in) :: a real(kind=dp), intent(in) :: b real(kind=dp), intent(out) :: c c = a * b end subroutine mult end module test I compile using f2py -c -m --f90flags='-fopenacc -foffload=nvptx-none -foffload=-O3 -O3 -fPIC' hello hello.f90 -L/usr/local/cuda/lib64 -lcublas -lcudart -lgomp Now, until I add the acc directives everything works fine. But as soon as I add the acc directives I get this error. gfortran:f90: /tmp/tmpld6ssow3/src.linux-x86_64-3.5/hello-f2pywrappers2.f90 /home//Experiments/Nvidia/OpenACC/OLCFHack15/gcc6/install/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpld6ssow3/tmp/tmpld6ssow3/src.linux-x86_64-3.5/hellomodule.o /tmp/tmpld6ssow3/tmp/tmpld6ssow3/src.linux-x86_64-3.5/fortranobject.o /tmp/tmpld6ssow3/hello.o /tmp/tmpld6ssow3/tmp/tmpld6ssow3/src.linux-x86_64-3.5/hello-f2pywrappers2.o -L/usr/local/cuda/lib64 -L/home//usr/local/miniconda/lib -lcublas -lcudart -lgomp -lpython3.5m -lgfortran -o ./hello.cpython-35m-x86_64-linux-gnu.so /usr/bin/ld: /tmp/cc2yQ89d.target.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC /tmp/cc2yQ89d.target.o: error adding symbols: Bad value collect2: error: ld returned 1 exit status /usr/bin/ld: /tmp/cc2yQ89d.target.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC /tmp/cc2yQ89d.target.o: error adding symbols: Bad value collect2: error: ld returned 1 exit status error: Command "/home//Experiments/Nvidia/OpenACC/OLCFHack15/gcc6/install/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpld6ssow3/tmp/tmpld6ssow3/src.linux-x86_64-3.5/hellomodule.o /tmp/tmpld6ssow3/tmp/tmpld6ssow3/src.linux-x86_64-3.5/fortranobject.o /tmp/tmpld6ssow3/hello.o /tmp/tmpld6ssow3/tmp/tmpld6ssow3/src.linux-x86_64-3.5/hello-f2pywrappers2.o -L/usr/local/cuda/lib64 -L/home//usr/local/miniconda/lib -lcublas -lcudart -lgomp -lpython3.5m -lgfortran -o ./hello.cpython-35m-x86_64-linux-gnu.so" failed with exit status 1 I don't get why just putting acc directives should create errors, when omp does not. Vikram

1 1

Numpy scalar integers to negative scalar integer powers.
by Charles R Harris 28 Oct '16

28 Oct '16

Hi All, I've put up a PR to deal with the numpy scalar integer powers at https://github.com/numpy/numpy/pull/8221. Note that for now everything goes through the np.power function. Chuck

1 0

How to use user input as equation directly
by djxvillain 28 Oct '16

28 Oct '16

Hello all, I am an electrical engineer and new to numpy. I need the ability to take in user input, and use that input as a variable. For example: t = input('enter t: ') x = input('enter x: ') I need the user to be able to enter something like x = 2*np.sin(2*np.pi*44100*t+np.pi/2) and it be the same as if they just typed it in the .py file. There's no clean way to cast or evaluate it that I've found. I could make a function to parse this string character by character, but I figured this is probably a common problem and someone else has probably figured it out and created an object for it. I can't find a library that does it though. If I can provide any more information please let me know. Thank you in advance for your help. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/How-to-use-user-input-as-equati… Sent from the Numpy-discussion mailing list archive at Nabble.com.

6 8

Re: [Numpy-discussion] padding options for diff
by Peter Creasey 28 Oct '16

28 Oct '16

> Date: Wed, 26 Oct 2016 16:18:05 -0400 > From: Matthew Harrigan <harrigan.matthew(a)gmail.com> > > Would it be preferable to have to_begin='first' as an option under the > existing kwarg to avoid overlapping? > >> if keep_left: >> if to_begin is None: >> to_begin = np.take(a, [0], axis=axis) >> else: >> raise ValueError(?np.diff(a, keep_left=False, to_begin=None) >> can be used with either keep_left or to_begin, but not both.?) >> >> Generally I try to avoid optional keyword argument overlap, but in >> this case it is probably justified. >> It works for me. I can't *think* of a case where you could have a np.diff on a string array and 'first' could be confused with an element, since you're not allowed diff on strings in the present numpy anyway (unless wiser heads than me know something!). Feel free to move the conversation to github btw. Peter

2 1