Mailman 3 January 2015 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer March 31, 2016

March 31, 2016

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

Generalize hstack/vstack --> stack; Block matrices like in matlab
by Stefan Otte Jan. 9, 2016

Jan. 9, 2016

Hey, quite often I work with block matrices. Matlab offers the convenient notation [ a b; c d ] to stack matrices. The numpy equivalent is kinda clumsy: vstack([hstack([a,b]), hstack([c,d])]) I wrote the little function `stack` that does exactly that: stack([[a, b], [c, d]]) In my case `stack` replaced `hstack` and `vstack` almost completely. If you're interested in including it in numpy I created a pull request [1]. I'm looking forward to getting some feedback! Best, Stefan [1] https://github.com/numpy/numpy/pull/5057

12 24

PR added: frozen dimensions in gufunc signatures
by Jaime Fernández del Río June 23, 2015

June 23, 2015

Hi, I have just sent a PR (https://github.com/numpy/numpy/pull/5015), adding the possibility of having frozen dimensions in gufunc signatures. As a proof of concept, I have added a `cross1d` gufunc to `numpy.core.umath_tests`: In [1]: import numpy as np In [2]: from numpy.core.umath_tests import cross1d In [3]: cross1d.signature Out[3]: '(3),(3)->(3)' In [4]: a = np.random.rand(1000, 3) In [5]: b = np.random.rand(1000, 3) In [6]: np.allclose(np.cross(a, b), cross1d(a, b)) Out[6]: True In [7]: %timeit np.cross(a, b) 10000 loops, best of 3: 76.1 us per loop In [8]: %timeit cross1d(a, b) 100000 loops, best of 3: 13.1 us per loop In [9]: c = np.random.rand(1000, 2) In [10]: d = np.random.rand(1000, 2) In [11]: cross1d(c, d) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-11-72c66212e40c> in <module>() ----> 1 cross1d(c, d) ValueError: cross1d: Operand 0 has a mismatch in its core dimension 0, with gufunc signature (3),(3)->(3) (size 2 is different from 3) The speed up over `np.cross` is nice, and while `np.cross` is not the best of examples, as it needs to handle more sizes, in many cases this will allow producing gufuncs that work without a Python wrapper redoing checks that are best left to the iterator, such as dimension sizes. It still needs tests, but before embarking on fully developing those, I wanted to make sure that there is an interest on this. I would also like to further enhance gufuncs providing computed dimensions, e.g. making it possible to e.g. define `pairwise_cross` with signature '(n, 3)->($m, 3)', where the $ indicates that m is a computed dimension, that would have to be calculated by a function passed to the gufunc constructor and stored in the gufunc object, based on the other core dimensions. In this case it would make $m be n*(n-1), so that all pairwise cross products between 3D vectors could be computed. The syntax with '$' is kind of crappy, so any suggestions on how to better express this in the signature are more than welcome, as well as any feedback on the merits (or lack of them) of implementing this. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 4

Problem with _dotblas.pyd when using Matplotlib for 3d plot
by Geoffrey Mégardon March 19, 2015

March 19, 2015

Hi, I am on Windows 8.1, 64-bit, using the usual version of Anaconda-64-bit for windows. I just come from the mailing list of matplotlib: my PC is not able to do 3D plot with matplotlib. Python crashes completely if I try to plot something in 3D and to show it: "Python.exe has stopped working..." The main problem in resolving my issue is that the python traceback and the faulthandler package do not succeed in catching the errors. Only the Windows crash report allows us to see what happened before the crash. The windows report tells that the file _dotblas.pyd from numpy is likely responsible for the crash. So I tried a simple: import numpy; numpy.test() numpy.test() does not work! I dont get any traceback from python. It just crashes as before: "Python.exe has stopped working..." And again, the Windows report blames _dotblas.pyd :) Here the new Windows report for the numpy.test(), I highlighted the lines which blame _dotblas.pyd: Version=1 EventType=APPCRASH EventTime=130608138227608275 ReportType=2 Consent=1 UploadTime=130608138229728384 ReportIdentifier=216b16d2-6f5c-11e4-bec3-48d22435da2b IntegratorReportIdentifier=216b16d1-6f5c-11e4-bec3-48d22435da2b NsAppName=python.exe Response.BucketId=398b7eee350a0fd8a7a96f705d3488f6 Response.BucketTable=4 Response.LegacyBucketId=85979911964 Response.type=4 Sig[0].Name=Application Name Sig[0].Value=python.exe Sig[1].Name=Application Version Sig[1].Value=0.0.0.0 Sig[2].Name=Application Timestamp Sig[2].Value=53b4679e *Sig[3].Name=Fault Module NameSig[3].Value=_dotblas.pyd* Sig[4].Name=Fault Module Version Sig[4].Value=0.0.0.0 Sig[5].Name=Fault Module Timestamp Sig[5].Value=545678cb Sig[6].Name=Exception Code Sig[6].Value=c000001d Sig[7].Name=Exception Offset Sig[7].Value=0000000000324022 DynamicSig[1].Name=OS Version DynamicSig[1].Value=6.3.9600.2.0.0.768.101 DynamicSig[2].Name=Locale ID DynamicSig[2].Value=2057 DynamicSig[22].Name=Additional Information 1 DynamicSig[22].Value=f32f DynamicSig[23].Name=Additional Information 2 DynamicSig[23].Value=f32feb95f950f918532aa47b4372840e DynamicSig[24].Name=Additional Information 3 DynamicSig[24].Value=9882 DynamicSig[25].Name=Additional Information 4 DynamicSig[25].Value=98823b6c7f579b24e92112ab827fe4a1 UI[2]=C:\Anaconda\python.exe UI[3]=python.exe has stopped working UI[4]=Windows can check online for a solution to the problem. UI[5]=Check online for a solution and close the program UI[6]=Check online for a solution later and close the program UI[7]=Close the program LoadedModule[0]=C:\Anaconda\python.exe LoadedModule[1]=C:\WINDOWS\SYSTEM32\ntdll.dll LoadedModule[2]=C:\WINDOWS\system32\KERNEL32.DLL LoadedModule[3]=C:\WINDOWS\system32\KERNELBASE.dll LoadedModule[4]=C:\Anaconda\python27.dll LoadedModule[5]=C:\WINDOWS\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\MSVCR90.dll LoadedModule[6]=C:\WINDOWS\system32\USER32.dll LoadedModule[7]=C:\WINDOWS\system32\ADVAPI32.dll LoadedModule[8]=C:\WINDOWS\system32\SHELL32.dll LoadedModule[9]=C:\WINDOWS\system32\GDI32.dll LoadedModule[10]=C:\WINDOWS\system32\msvcrt.dll LoadedModule[11]=C:\WINDOWS\SYSTEM32\sechost.dll LoadedModule[12]=C:\WINDOWS\system32\RPCRT4.dll LoadedModule[13]=C:\WINDOWS\SYSTEM32\combase.dll LoadedModule[14]=C:\WINDOWS\system32\SHLWAPI.dll LoadedModule[15]=C:\WINDOWS\system32\IMM32.DLL LoadedModule[16]=C:\WINDOWS\system32\MSCTF.dll LoadedModule[17]=C:\Anaconda\DLLs\_socket.pyd LoadedModule[18]=C:\WINDOWS\system32\WS2_32.dll LoadedModule[19]=C:\WINDOWS\system32\NSI.dll LoadedModule[20]=C:\Anaconda\DLLs\_ssl.pyd LoadedModule[21]=C:\Anaconda\DLLs\_ctypes.pyd LoadedModule[22]=C:\WINDOWS\system32\ole32.dll LoadedModule[23]=C:\WINDOWS\system32\OLEAUT32.dll LoadedModule[24]=C:\Anaconda\DLLs\_hashlib.pyd LoadedModule[25]=C:\WINDOWS\SYSTEM32\CRYPTSP.dll LoadedModule[26]=C:\WINDOWS\system32\rsaenh.dll LoadedModule[27]=C:\WINDOWS\SYSTEM32\bcrypt.dll LoadedModule[28]=C:\WINDOWS\SYSTEM32\CRYPTBASE.dll LoadedModule[29]=C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll LoadedModule[30]=C:\Users\User\AppData\Roaming\Python-Eggs\faulthandler-2.4-py2.7-win-amd64.egg-tmp\faulthandler.pyd LoadedModule[31]=C:\Anaconda\lib\site-packages\numpy\core\multiarray.pyd LoadedModule[32]=C:\Anaconda\lib\site-packages\numpy\core\umath.pyd LoadedModule[33]=C:\Anaconda\lib\site-packages\numpy\core\_dotblas.pyd LoadedModule[34]=C:\Anaconda\lib\site-packages\numpy\core\libiomp5md.dll LoadedModule[35]=C:\Anaconda\lib\site-packages\numpy\core\scalarmath.pyd LoadedModule[36]=C:\Anaconda\lib\site-packages\numpy\lib\_compiled_base.pyd LoadedModule[37]=C:\Anaconda\lib\site-packages\numpy\linalg\lapack_lite.pyd LoadedModule[38]=C:\Anaconda\lib\site-packages\numpy\linalg\_umath_linalg.pyd LoadedModule[39]=C:\Anaconda\lib\site-packages\numpy\fft\fftpack_lite.pyd LoadedModule[40]=C:\Anaconda\lib\site-packages\numpy\random\mtrand.pyd LoadedModule[41]=C:\Anaconda\DLLs\_multiprocessing.pyd State[0].Key=Transport.DoneStage1 State[0].Value=1 FriendlyEventName=Stopped working ConsentKey=APPCRASH AppName=python.exe AppPath=C:\Anaconda\python.exe NsPartner=windows NsGroup=windows8 ApplicationIdentity=5B036AF1EC2E20F320DBF28D119DE93D Any idea, of what I can do? I try to reinstall anaconda, I also try conda remove numpy and conda install numpy. That does not seem to change anything. -- -- MEGARDON Geoffrey

5 4

3D array and the right hand rule
by Dieter Van Eessen Feb. 11, 2015

Feb. 11, 2015

Hello, I'm a novice with respect to scientific computing using python. I've read that numpy.array isn't arranged according to the 'right-hand-rule' (right-hand-rule => thumb = +x; index finger = +y, bend middle finder = +z). This is also confirmed by an old message I dug up from the mailing list archives. (see message below) I guess this has consequences for certain algorithms (been a while, should actually revise some algebra textbooks). At least it does when I try to mentally visualize a 3D array when I'm not sure which dimensions to use... What are the consequences in using 3D arrays in for example transformation algorithms or other algorithms which expect certain shape? Should I always 'reshape' before using them or do most algorithms take this into account in the underlying algebra? Does the 'Fortran contiguous' shape always respect the 'right-hand-rule' (for 3D arrays)? And just to be sure: Is the information telling if an array is C-contiguous or Fortran-contiguous always stored within the array? kind regards, Dieter / Old message From: Anne Archibald <peridot.faceted <at> gmail.com> Subject: Re: dimension aligment <http://news.gmane.org/find-root.php?message_id=ce557a360805201104r3fd2e192y…> Newsgroups: gmane.comp.python.numeric.general <http://news.gmane.org/gmane.comp.python.numeric.general> Date: 2008-05-20 18:04:46 GMT (6 years, 35 weeks, 5 days, 4 hours and 34 minutes ago) 2008/5/20 Thomas Hrabe <thrabe <at> burnham.org>: > given a *3d* *array* > a = > numpy.*array*([[[1,2,3],[4,5,6]],[[7,8,9],[10,11,12]],[[13,14,15],[16,17,18]],[[19,20,21],[22,23,24]]]) > a.shape > returns (4,2,3) > > so I assume the first digit is the 3rd dimension, second is 2nd dim and > third is the first. > > how is the data aligned in memory now? > according to the strides it should be > 1,2,3,4,5,6,7,8,9,10,... > *right*? > > if I had an *array* of more dimensions, the first digit returned by shape > should always be the highest dim. You are basically *right*, but this is a surprisingly subtle issue for numpy. A numpy *array* is basically a block of memory and some description. One piece of that description is the type of data it contains (i.e., how to interpret each chunk of memory) for example int32, float64, etc. Another is the sizes of all the various dimensions. A third piece, which makes many of the things numpy does possible, is the "strides". The way numpy works is that basically it translates A[i,j,k] into a lookup of the item in the memory block at position i*strides[0]+j*strides[1]+k*strides[2] This means, if you have an *array* A and you want every second element (A[::2]), all numpy needs to do is *hand* you back a new *array* pointing to the same data block, but with strides[0] doubled. Similarly if you want to transpose a two-dimensional *array*, all it needs to do is exchange strides[0] and strides[1]; no data need be moved. This means, though, that if you are *hand*ed a numpy *array*, the elements can be arranged in memory in quite a complicated fashion. Sometimes this is no problem - you can always use the strides to find it all. But sometimes you need the data arranged in a particular way. numpy defines two particular ways: "C contiguous" and "FORTRAN contiguous". "C contiguous" *array*s are what you describe, and they're what numpy produces by default; they are arranged so that the *right*most index has the smallest stride. "FORTRAN contiguous" *array*s are arranged the other way around; the leftmost index has the smallest stride. (This is how FORTRAN *array*s are arranged in memory.) There is also a special case: the reshape() function changes the shape of the *array*. It has an "order" argument that describes not how the elements are arranged in memory but how you want to think of the elements as arranged in memory for the reshape operation. Anne /Old message

2 2

new mingw-w64 based numpy and scipy wheel (still experimental)
by Carl Kleffner Feb. 9, 2015

Feb. 9, 2015

I took time to create mingw-w64 based wheels of numpy-1.9.1 and scipy-0.15.1 source distributions and put them on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads as well as on binstar.org. The test matrix is python-2.7 and 3.4 for both 32bit and 64bit. Feedback is welcome. The wheels can be pip installed with: pip install -i https://pypi.binstar.org/carlkl/simple numpy pip install -i https://pypi.binstar.org/carlkl/simple scipy Some technical details: the binaries are build upon OpenBLAS as accelerated BLAS/Lapack. OpenBLAS itself is build with dynamic kernels (similar to MKL) and automatic runtime selection depending on the CPU. The minimal requested feature supplied by the CPU is SSE2. SSE1 and non-SSE CPUs are not supported with this builds. This is the default for 64bit binaries anyway. OpenBLAS is deployed as part of the numpy wheel. That said, the scipy wheels mentioned above are dependant on the installation of the OpenBLAS based numpy and won't work i.e. with an installed numpy-MKL. For the numpy 32bit builds there are 3 failures for special FP value tests, due to a bug in mingw-w64 that is still present. All scipy versions show up 7 failures with some numerical noise, that could be ignored (or corrected with relaxed asserts in the test code). PR's for numpy and scipy are in preparation. The mingw-w64 compiler used for building can be found at https://bitbucket.org/carlkl/mingw-w64-for-python/downloads.

7 31

Characteristic of a Matrix.
by Colin J. Williams Feb. 4, 2015

Feb. 4, 2015

One of the essential characteristics of a matrix is that it be rectangular. This is neither spelt out or checked currently. The Doc description refers to a class: - *class *numpy.matrix[source] <http://github.com/numpy/numpy/blob/v1.9.1/numpy/matrixlib/defmatrix.py#L206> Returns a matrix from an array-like object, or from a string of data. A matrix is a specialized 2-D array that retains its 2-D nature through operations. It has certain special operators, such as * (matrix multiplication) and ** (matrix power). This illustrates a failure, which is reported later in the calculation: A2= np.matrix([[1, 2, -2], [-3, -1, 4], [4, 2 -6]]) Here 2 - 6 is treated as an expression. Wikipedia offers: In mathematics <http://en.wikipedia.org/wiki/Mathematics>, a *matrix* (plural *matrices*) is a rectangular <http://en.wikipedia.org/wiki/Rectangle> *array <http://en.wiktionary.org/wiki/array>*[1] <http://en.wikipedia.org/wiki/Matrix_%28mathematics%29#cite_note-1> of numbers <http://en.wikipedia.org/wiki/Number>, symbols <http://en.wikipedia.org/wiki/Symbol_%28formal%29>, or expressions <http://en.wikipedia.org/wiki/Expression_%28mathematics%29>, arranged in *rows <http://en.wiktionary.org/wiki/row>* and *columns <http://en.wiktionary.org/wiki/column>*.[2] <http://en.wikipedia.org/wiki/Matrix_%28mathematics%29#cite_note-2>[3] <http://en.wikipedia.org/wiki/Matrix_%28mathematics%29#cite_note-3> The individual items in a matrix are called its *elements* or *entries*. An example of a matrix with 2 rows and 3 columns is [image: \begin{bmatrix}1 & 9 & -13 \\20 & 5 & -6 \end{bmatrix}.]In the Numpy context, the symbols or expressions need to be evaluable. Colin W.

11 20

Views of a different dtype
by Jaime Fernández del Río Feb. 4, 2015

Feb. 4, 2015

HI all, There has been some recent discussion going on on the limitations that numpy imposes to taking views of an array with a different dtype. As of right now, you can basically only take a view of an array if it has no Python objects and neither the old nor the new dtype are structured. Furthermore, the array has to be either C or Fortran contiguous. This seem to be way too strict, but the potential for disaster getting a loosening of the restrictions wrong is big, so it should be handled with care. Allan Haldane and myself have been looking into this separately and discussing some of the details over at github, and we both think that the only true limitation that has to be imposed is that the offsets of Python objects within the new and old dtypes remain compatible. I have expanded Allan's work from here: https://github.com/ahaldane/numpy/commit/e9ca367 to make it as flexible as I have been able. An implementation of the algorithm in Python, with a few tests, can be found here: https://gist.github.com/jaimefrio/b4dae59fa09fccd9638c#file-dtype_compat-py I would appreciate getting some eyes on it for correctness, and to make sure that it won't break with some weird dtype. I am also trying to figure out what the ground rules for stride and shape conversions when taking a view with a different dtype should be. I submitted a PR (gh-5508) a couple for days ago working on that, but I am not so sure that the logic is completely sound. Again, to get more eyes on it, I am going to reproduce my thoughts here on the hope of getting some feedback. The objective would be to always allow a view of a different dtype (given that the dtypes be compatible as described above) to be taken if: - The itemsize of the dtype doesn't change. - The itemsize changes, but the array being viewed is the result of slicing and transposing axes of a contiguous array, and it is still contiguous, defined as stride == dtype.itemsize, along its smallest-strided dimension, and the itemsize of the newtype exactly divides the size of that dimension. - Ideally taking a view should be a reversible process, i.e. if oldtype = arr.dtype, then doing arr.view(newtype).view(oldtype) should give you back a view of arr with the same original shape, strides and dtype. This last point can get tricky if the minimal stride dimension has size 1, as there could be several of those, e.g.: >>> a = np.ones((3, 4, 1), dtype=float)[:, :2, :].transpose(0, 2, 1) >>> a.flags.contiguous False >>> a.shape (3, 1, 2) >>> a.strides # the stride of the size 1 dimension could be anything, ignore it! (32, 8, 8) b = a.view(complex) # this fails right now, but should work >>> b.flags.contiguous False >>> b.shape (3, 1, 1) >>> b.strides # the stride of the size 1 dimensions could be anything, ignore them! (32, 16, 16) c = b.view(float) # which of the two size 1 dimensions should we expand? "In the face of ambiguity refuse the temptation to guess" dictates that last view should raise an error, unless we agree and document some default. Any thoughts? Then there is the endless complication one could get into with arrays created with as_strided. I'm not smart enough to figure when and when not those could work, but am willing to retake the discussion if someone wiser si interested. With all these in mind, my proposal for the new behavior is that taking a view of an array with a different dtype would require: 1. That the newtype and oldtype be compatible, as defined by the algorithm checking object offsets linked above. 2. If newtype.itemsize == oldtype.itemsize no more checks are needed, make it happen! 3. If the array is C/Fortran contiguous, check that the size in bytes of the last/first dimension is evenly divided by newtype.itemsize. If it does, go for it. 4. For non-contiguous arrays: 1. Ignoring dimensions of size 1, check that no stride is smaller than either oldtype.itemsize or newtype.itemsize. If any is found this is an as_strided product, sorry, can't do it! 2. Ignoring dimensions of size 1, find a contiguous dimension, i.e. stride == oldtype.itemsize 1. If found, check that it is the only one with that stride, that it is the minimal stride, and that the size in bytes of that dimension is evenly divided by newitem,itemsize. 2. If none is found, check if there is a size 1 dimension that is also unique (unless we agree on a default, as mentioned above) and that newtype.itemsize evenly divides oldtype.itemsize. Apologies for the long, dense content, but any thought or comments are very welcome. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

6 13

missing FloatingPointError for numpy on cygwin64
by Sebastien Gouezel Feb. 2, 2015

Feb. 2, 2015

Dear all, I tried to use numpy (version 1.9.1, installed by `pip install numpy`) on cygwin64. I encountered the following weird bug: >>> import numpy >>> with numpy.errstate(all='raise'): ... print 1/float64(0.0) inf I was expecting a FloatingPointError, but it didn't show up. Curiously, with different numerical types (all intxx, or float128), I indeed get the FloatingPointError. Same thing with the most recent git version, or with 1.7.1 provided as a precompiled package by cygwin. This behavior does not happen on cygwin32 (I always get the FloatingPointError there). I wonder if there is something weird with my config, or if this is a genuine reproducible bug. If so, where should I start looking if I want to fix it? (I don't know anything about numpy's code) Sebastien

2 3

Numpy 'None' comparison FutureWarning
by Demitri Muna Jan. 31, 2015

Jan. 31, 2015

Hi, I just encountered the following in my code: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future. I'm very concerned about this. This is a very common programming pattern (lazy loading): class A(object): def __init__(self): self._some_array = None @property def some_array(self): if self._some_array == None: # perform some expensive setup of array return self._some_array It seems to me that the new behavior will break this pattern. I think that redefining the "==" operator is a little too aggressive here. It strikes me as very nonstandard and not at all obvious to someone reading the code that the comparison is a very special case for numpy objects. Unless there's some aspect I'm missing here, I think an element-wise comparator should be more explicit. Cheers, Demitri _________________________________________ Demitri Muna Department of Astronomy Le Ohio State University http://trillianverse.org http://scicoder.org

5 6

question np.partition
by Neal Becker Jan. 30, 2015

Jan. 30, 2015

It sounds like np.partition could be used to answer the question: give me the highest K elements in a vector. Is this a correct interpretation? Something like partial sort, but returned elements are unsorted. I could really make some use of this, but in my case it is a list of objects I need to sort on a particular key. Is this algorithm available in general python code (not specific to numpy arrays)? -- -- Those who don't understand recursion are doomed to repeat it

3 2

Re: [Numpy-discussion] Testing of scipy
by Carl Kleffner Jan. 30, 2015

Jan. 30, 2015

Hi Colin. this is an interesting test with different hardware. As a summary: - Python-2.7 amd64 - numpy-0.1.9.openblas: OK - scipy-0.15.1.openblas: 2 errors 11 failures - CPU: AMD A8-5600K APU (piledriver) scipy errors and failures due to piledriver: (1) ERROR: test_improvement (test_quadpack.TestCtypesQuad) WindowsError: [Error 193] %1 is not a valid Win32 application (2) ERROR: test_typical (test_quadpack.TestCtypesQuad) WindowsError: [Error 193] %1 is not a valid Win32 application (3) FAIL: test_interpolate.TestInterp1D.test_circular_refs ReferenceError: Remaining reference(s) to object (4) FAIL: test__gcutils.test_assert_deallocated ReferenceError: Remaining reference(s) to object other failures are known failures due to mingw-w64 / openblas build. (1) and (2) is a problem with ctypes.util.find_msvcrt(). This methods seems to be buggy. Not a scipy problem. Maybe the test could be enhanced. (3) and (4) is a problem due to garbage collection. No idea why. Maybe you can file a bug for (1) ... (4) Carl 2015-01-29 19:36 GMT+01:00 cjw <cjw(a)ncf.ca>: > Carl, > > I have already sent the test result for numpy. Here is the test result > for scipy. > > I hope that it helps. > > Colin W. > > ------------------------------ > *** Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit > (AMD64)] on win32. *** > >>> > [Dbg]>>> > C:\Python27\lib\site-packages\numpy\core\__init__.py:6: Warning: Numpy > 64bit experimental build with Mingw-w64 and OpenBlas. > from . import multiarray > Running unit tests for scipy > NumPy version 1.9.1 > NumPy is installed in C:\Python27\lib\site-packages\numpy > SciPy version 0.15.1 > SciPy is installed in C:\Python27\lib\site-packages\scipy > Python version 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit > (AMD64)] > nose version 1.3.4 > C:\Python27\lib\site-packages\numpy\lib\utils.py:95: DeprecationWarning: > `scipy.lib.blas` is deprecated, use `scipy.linalg.blas` instead! > warnings.warn(depdoc, DeprecationWarning) > C:\Python27\lib\site-packages\numpy\lib\utils.py:95: DeprecationWarning: > `scipy.lib.lapack` is deprecated, use `scipy.linalg.lapack` instead! > warnings.warn(depdoc, DeprecationWarning) > C:\Python27\lib\site-packages\numpy\lib\utils.py:95: DeprecationWarning: > `scipy.weave` is deprecated, use `weave` instead! > warnings.warn(depdoc, DeprecationWarning) > ...............................................................................................................................................................................................................................................................K......................................................................................................................................................................................................................................................................................EE.........................................................................................................................................F.....K..K............................................................................................................................................................................................................................................................................................................... > ....... > ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................SSSSSS......SSSSSS......SSSS......F....................... > ........ > ............................................................................F............S..........K..........................................................................................................................................................................................................................................F....................................................K.............................................................................................................................................................................................K...............................................................................................SSSSSSSS.K.................S................................................................................................................................................................................................................................................................................................ > ........ > .....................................S........................................................................................................................................................................................C:\Python27\lib\site-packages\scipy\optimize\minpack.py:604: > OptimizeWarning: Covariance of the parameters could not be estimated > category=OptimizeWarning) > .....................................................................................................................................................F.................K.F.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... > ....... > ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.....................................................................................................................................................................................................................................................SS..........S....S...SSSSS.SSSSSSSSSS.....SSS..S.....S....K...........S..S........SSSSK...SSSS..S.........S........SS..........S...S...SSSSS.SSSSSSSSSS.....SSS........S................S..S...... > ..SSSS.. > .SSSS..S............S..............................S.................................................S........................S..........S................................S...................................................S........................S..........S..........SS.......S...S...SSSSS.SSSSSSSSSS.....SSS..S.....S............SS..S..S........SSSS....SSSS..S.........S......K.....................S.....K...................S..................SS...S.........................S..........S.........................S.................................................SS..S...........................S.........S........SS..........S....S...SSSSS.SSSSSSSSSS.....SSS..S.....S....K...........S..S........SSSSK...SSSS..S.........S........SS..........S...S...SSSSS.SSSSSSSSSS.....SSS........S................S..S........SSSS...SSSS..S............S..............................S.................................................S........................S..........S................................S... > ........ > ........................................S........................S..........S..........SS.......S...S...SSSSS.SSSSSSSSSS.....SSS..S.....S............SS..S..S........SSSS....SSSS..S.........S......K.....................S.....K...................S..................SS...S.........................S..........S.........................S.................................................SS..S...........................S.........S........SS..........S....S...SSSSS.SSSSSSSSSS.....SSS..S.....S....K...........S..S........SSSSK...SSSS..S.........S........SS..........S...S...SSSSS.SSSSSSSSSS.....SSS........S................S..S........SSSS...SSSS..S............S..............................S.................................................S........................S..........S................................S...................................................S........................S..........S..........SS.......S...S...SSSSS.SSSSSSSSSS.....SSS..S.....S............SS..S..S........SSSS....S > SSS..S.. > .......S......K.....................S.....K...................S..................SS...S.........................S..........S.........................S.................................................SS..S...........................S.........S........SS..........S....S...SSSSS.SSSSSSSSSS.....SSS..S.....S....K...........S..S........SSSSK...SSSS..S.........S........SS..........S...S...SSSSS.SSSSSSSSSS.....SSS........S................S..S........SSSS...SSSS..S............S..............................S.................................................S........................S..........S................................S...................................................S........................S..........S..........SS.......S...S...SSSSS.SSSSSSSSSS.....SSS..S.....S............SS..S..S........SSSS....SSSS..S.........S......K.....................S.....K...................S..................SS...S.........................S..........S.........................S........................ > ........ > .................SS..S...........................S.........S........SS..........S...S...SSSSS.SSSSSSSSSS.....SSS..S.....S....K...........S..S........SSSSK...SSSS..S.........S........SS..........S...S...SSSSS.SSSSSSSSSS.....SSS........S................S..S........SSSS...SSSS..S............S..............................S.................................................S........................S..........S................................S...................................................S........................S..........S..........SS.......S...S...SSSSS.SSSSSSSSSS.....SSS..S.....S............SS..S..S........SSSS....SSSS..S.........S......K.....................S.....K...................S..................SS...S.........................S..........S.........................S.................................................SS..S...........................S.........S..............SSSSSSSSSSSSSSSSSSSSSSSSSSSSS...........S....S......SSSS.SSSSSSSSSS.......SSS..S..........S....K..... > ........ > .........................................S....S..............SSSSK...SSSS...................................S.........S.........K............SSK.K......KS....SK.KKSSSS.SSSSSSSSSS....KSSS..K.K......S....KKK...K...K.....SK.S..............SSSSK...SSSS...K...............................SK.............SSSSSSSSSSSSSSSSSSSSSSSSSSSSS...........S...S..SSS.SSSS.SSSSSSSSSS....SSSSSS....SSS......S.....SSSSSS................................................SSSS.S..............SSSS...SSSS......................................S.........S.........K............SSK.K.......S...SK.SSS.SSSS.SSSSSSSSSS....SSSSSS..K.SSS......S.....SSSSSS..........................................K.....SSSS.S..............SSSS....SSSS...K..................................SK..................SSSSSSSSSSSSSSSSSSSSSSSSSSS............................................................................................................................................................................S.............................. > ........ > ....................................................................................................................................................................................................................................................................................................................................S..........S..........K.................K.K...........K..K...................................K...........................K...............K.................................................................K......S......K.........................................................................................................................................................................................................................................................................................................................K.K.......................................S.K...................SSSSSSSSSSSSSSSSSSSSSSSSSSS............................................ > ........ > ..........................................................................................................................S..........................................................................................................................................................................................................................................................................................................................................................................S..........S..........K.................K.K...........K..K...................................K...........................K.................K.................................................................K......S.................................................................................................................................................................................................................................................................................................... > ........ > ....................K.K.......................................S.K..............SSSSSSSSSSSSSSSSSSSSSSSSSSSSS........S...S..SSS.SSSS.SSSSSSSSSS....SSSSSS..S.SSS......S.....SSSSSS..........................................SS....SSSS.S..............SSSS....SSSS...................................S.........S....................K....SSSSSSSSSSSSSSSSSSSSSSSSSSS.SSS.............SSS....................K.............SSS...................S.SSS............SSSSSS..........................................SS.....SSSS.............................................................................................................................................................................................................................................................................................................................................................SSSSSSSSSSSSSSSSSSSS...............SSSSSSSSSSSSSSSSSSSSSSSSSSS...............SSS...................................SSS................ > .....SSS > .............SSS...SSS..........................................SS....SSSS..............................................................................................................................................................................................................................................................................................................................................................SSSSSSSSSSSSSSSSSSSS.................................................................S...S............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ > ........ > .........................................................F..F............................................................................................................................................................................................................................................................................................................................................................................................F...........................................F......................................K.K.............................K.....K..K...KK........................SSSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................... > ........ > .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...........K.....................................................................................................................................................................................................................................................................................................................................K......................................K.................................................................................S........................................................... > ........ > .......................................................................................................................................................................................................................................................S..S..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.SSS....SSSSSS................................................................................................. > ====================================================================== > ERROR: test_improvement (test_quadpack.TestCtypesQuad) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\numpy\testing\decorators.py", line > 146, in skipper_func > return f(*args, **kwargs) > File > "C:\Python27\lib\site-packages\scipy\integrate\tests\test_quadpack.py", > line 42, in setUp > self.lib = ctypes.CDLL(file) > File "C:\Python27\Lib\ctypes\__init__.py", line 365, in __init__ > self._handle = _dlopen(self._name, mode) > WindowsError: [Error 193] %1 is not a valid Win32 application > > ====================================================================== > ERROR: test_typical (test_quadpack.TestCtypesQuad) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\numpy\testing\decorators.py", line > 146, in skipper_func > return f(*args, **kwargs) > File > "C:\Python27\lib\site-packages\scipy\integrate\tests\test_quadpack.py", > line 42, in setUp > self.lib = ctypes.CDLL(file) > File "C:\Python27\Lib\ctypes\__init__.py", line 365, in __init__ > self._handle = _dlopen(self._name, mode) > WindowsError: [Error 193] %1 is not a valid Win32 application > > ====================================================================== > FAIL: test_interpolate.TestInterp1D.test_circular_refs > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\scipy\interpolate\tests\test_interpolate.py", > line 353, in test_circular_refs > del interp > File "C:\Python27\Lib\contextlib.py", line 24, in __exit__ > self.gen.next() > File "C:\Python27\lib\site-packages\scipy\lib\_gcutils.py", line 96, in > assert_deallocated > raise ReferenceError("Remaining reference(s) to object") > ReferenceError: Remaining reference(s) to object > > ====================================================================== > FAIL: test__gcutils.test_assert_deallocated > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\scipy\lib\tests\test__gcutils.py", > line 57, in test_assert_deallocated > del c > File "C:\Python27\Lib\contextlib.py", line 24, in __exit__ > self.gen.next() > File "C:\Python27\lib\site-packages\scipy\lib\_gcutils.py", line 96, in > assert_deallocated > raise ReferenceError("Remaining reference(s) to object") > ReferenceError: Remaining reference(s) to object > > ====================================================================== > FAIL: test_syr_her (test_blas.TestFBLAS2Simple) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\scipy\linalg\tests\test_blas.py", > line 316, in test_syr_her > resz_reverse, rtol=rtol) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 1297, > in assert_allclose > verbose=verbose, header=header) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 665, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-07, atol=0 > > (mismatch 100.0%) > x: array([[ 0.000000e+00+0.j, 0.000000e+00+0.j, 0.000000e+00+0.j, > 0.000000e+00+0.j], > [ 0.000000e+00+0.j, 0.000000e+00+0.j, 0.000000e+00+0.j,... > y: array([[-15.+112.j, -13. +82.j, -11. +52.j, -9. +22.j], > [ 0. +0.j, -11. +60.j, -9. +38.j, -7. +16.j], > [ 0. +0.j, 0. +0.j, -7. +24.j, -5. +10.j], > [ 0. +0.j, 0. +0.j, 0. +0.j, -3. +4.j]]) > > ====================================================================== > FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 4)) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\scipy\linalg\tests\test_decomp.py", > line 648, in eigenhproblem_general > assert_array_almost_equal(diag2_, ones(diag2_.shape[0]), DIGITS[dtype]) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 842, > in assert_array_almost_equal > precision=decimal) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 665, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 4 decimals > > (mismatch 100.0%) > x: array([ 0., 0., 0.], dtype=float32) > y: array([ 1., 1., 1.]) > > ====================================================================== > FAIL: Tests for the minimize wrapper. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\scipy\optimize\tests\test_optimize.py", line > 512, in test_minimize > self.test_powell(True) > File > "C:\Python27\lib\site-packages\scipy\optimize\tests\test_optimize.py", line > 241, in test_powell > atol=1e-14, rtol=1e-7) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 1297, > in assert_allclose > verbose=verbose, header=header) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 665, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-07, atol=1e-14 > > (mismatch 100.0%) > x: array([[ 0.750776, -0.441569, 0.47101 ], > [ 0.750776, -0.441569, 0.480525], > [ 1.501553, -0.883139, 0.951535],... > y: array([[ 0.72949 , -0.441569, 0.47101 ], > [ 0.72949 , -0.441569, 0.480525], > [ 1.45898 , -0.883139, 0.951535],... > > ====================================================================== > FAIL: Powell (direction set) optimization routine > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\scipy\optimize\tests\test_optimize.py", line > 241, in test_powell > atol=1e-14, rtol=1e-7) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 1297, > in assert_allclose > verbose=verbose, header=header) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 665, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-07, atol=1e-14 > > (mismatch 100.0%) > x: array([[ 0.750776, -0.441569, 0.47101 ], > [ 0.750776, -0.441569, 0.480525], > [ 1.501553, -0.883139, 0.951535],... > y: array([[ 0.72949 , -0.441569, 0.47101 ], > [ 0.72949 , -0.441569, 0.480525], > [ 1.45898 , -0.883139, 0.951535],... > > ====================================================================== > FAIL: test_arpack.test_linearoperator_deallocation > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File > "C:\Python27\lib\site-packages\scipy\sparse\linalg\eigen\arpack\tests\test_arpack.py", > line 798, in test_linearoperator_deallocation > pass > File "C:\Python27\Lib\contextlib.py", line 24, in __exit__ > self.gen.next() > File "C:\Python27\lib\site-packages\scipy\lib\_gcutils.py", line 96, in > assert_deallocated > raise ReferenceError("Remaining reference(s) to object") > ReferenceError: Remaining reference(s) to object > > ====================================================================== > FAIL: test_beta (test_basic.TestCephes) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\scipy\special\tests\test_basic.py", > line 144, in test_beta > assert_allclose(cephes.beta(0.0342, 171), 24.070498359873497, > rtol=1e-14, atol=0) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 1297, > in assert_allclose > verbose=verbose, header=header) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 665, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-14, atol=0 > > (mismatch 100.0%) > x: array(24.07049835987154) > y: array(24.070498359873497) > > ====================================================================== > FAIL: test_betaincinv (test_basic.TestCephes) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\scipy\special\tests\test_basic.py", > line 157, in test_betaincinv > assert_allclose(cephes.betaincinv(0.0342, 171, 0.25), > 8.4231316935498957e-21, rtol=1e-12, atol=0) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 1297, > in assert_allclose > verbose=verbose, header=header) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 665, > in assert_array_compare > raise AssertionError(msg) > AssertionError: > Not equal to tolerance rtol=1e-12, atol=0 > > (mismatch 100.0%) > x: array(8.423131693529815e-21) > y: array(8.423131693549896e-21) > > ====================================================================== > FAIL: test_data.test_boost(<Data for binom: > binomial_data_ipp-binomial_data>,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\scipy\special\tests\test_data.py", > line 481, in _test_factory > test.check(dtype=dtype) > File "C:\Python27\lib\site-packages\scipy\special\_testutils.py", line > 292, in check > assert_(False, "\n".join(msg)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 53, in > assert_ > raise AssertionError(smsg) > AssertionError: > Max |adiff|: 9.70919e+24 > Max |rdiff|: 7.10315e-14 > Bad results (70 out of 159) for the following points (in output 0): > 42.0 21.0 > => 538257874440.0024 != 538257874440.0 > (rdiff 4.422361858108481e-15) > 46.0 21.0 > => 6943526580276.032 != 6943526580276.0 > (rdiff 4.6412384438109286e-15) > 82.0 21.0 => > 1.8330655594514242e+19 != 1.8330655594514647e+19 (rdiff > 2.21216310518291e-14) > 82.0 29.0 => > 1.2576607627509685e+22 != 1.2576607627510093e+22 (rdiff > 3.2516291524073137e-14) > 82.0 33.0 => > 8.999865905487466e+22 != 8.999865905487887e+22 (rdiff > 4.679048843863541e-14) > 82.0 42.0 => > 4.146706622571353e+23 != 4.1467066225715386e+23 (rdiff > 4.482872076557015e-14) > 82.0 46.0 => > 2.322318339533674e+23 != 2.3223183395337372e+23 (rdiff > 2.71635163388777e-14) > 95.0 21.0 => > 6.112305389981363e+20 != 6.112305389981495e+20 (rdiff > 2.165839426429588e-14) > 95.0 29.0 => > 2.146280142106093e+24 != 2.1462801421060996e+24 (rdiff > 3.0016822210721096e-15) > 95.0 33.0 => > 3.7802224438384017e+25 != 3.780222443838485e+25 (rdiff > 2.204165674911803e-14) > 95.0 42.0 => > 1.7198752232706476e+27 != 1.7198752232706825e+27 (rdiff > 2.0297690035618223e-14) > 95.0 46.0 => > 3.086205608690921e+27 != 3.08620560869098e+27 (rdiff > 1.923839025264666e-14) > 122.0 21.0 > => 2.05054879430629e+23 != 2.050548794306221e+23 > (rdiff 3.370908808019204e-14) > 122.0 29.0 => > 9.654999814547143e+27 != 9.65499981454662e+27 (rdiff > 5.4093012244985185e-14) > 122.0 33.0 => > 6.889061799493289e+29 != 6.88906179949298e+29 (rdiff > 4.473977857490729e-14) > 122.0 42.0 => > 9.820488930268602e+32 != 9.820488930268152e+32 (rdiff > 4.5785845286258346e-14) > 122.0 46.0 => > 9.517963588769975e+33 != 9.517963588769497e+33 (rdiff > 5.0148279981909684e-14) > 122.0 82.0 => > 2.546052685625168e+32 != 2.5460526856250765e+32 (rdiff > 3.594314640260528e-14) > 122.0 95.0 > => 8.77923835320522e+26 != 8.779238353204766e+26 > (rdiff 5.1661491374367e-14) > 125.0 21.0 => > 3.577965774451988e+23 != 3.577965774451971e+23 (rdiff > 4.689037586607358e-15) > 125.0 29.0 => > 2.1471697865847835e+28 != 2.1471697865846783e+28 (rdiff > 4.895435483124065e-14) > 125.0 33.0 => > 1.7431114722001235e+30 != 1.7431114722001072e+30 (rdiff > 9.365751364490987e-15) > 125.0 42.0 => > 3.3961976443365834e+33 != 3.396197644336376e+33 (rdiff > 6.110535739146696e-14) > 125.0 46.0 => > 3.824445086978443e+34 != 3.8244450869782213e+34 (rdiff > 5.800111975502932e-14) > 125.0 82.0 => > 6.555451266975118e+33 != 6.555451266974865e+33 (rdiff > 3.851600739996281e-14) > 125.0 95.0 => > 6.870943317071316e+28 != 6.870943317070972e+28 (rdiff > 5.018333445043497e-14) > 135.0 29.0 => > 2.6545984839545678e+29 != 2.654598483954587e+29 (rdiff > 7.289767083732218e-15) > 135.0 33.0 => > 3.2226838569785926e+31 != 3.222683856978583e+31 (rdiff > 2.9346841443966352e-15) > 135.0 42.0 > => 1.65539796634103e+35 != 1.6553979663410087e+35 > (rdiff 1.2814897756371891e-14) > 135.0 46.0 => > 2.9618674395653716e+36 != 2.9618674395653185e+36 (rdiff > 1.7936867201619373e-14) > 135.0 82.0 => > 1.3239687524906834e+38 != 1.3239687524906744e+38 (rdiff > 6.8483063743406465e-15) > 135.0 95.0 => > 3.1921559888457253e+34 != 3.1921559888457077e+34 (rdiff > 5.4898341219098594e-15) > 137.0 21.0 => > 2.891654383984857e+24 != 2.8916543839848387e+24 (rdiff > 6.312514769778831e-15) > 137.0 29.0 => > 4.280069137507773e+29 != 4.28006913750795e+29 (rdiff > 4.126698476389006e-14) > 137.0 33.0 => > 5.605400076850502e+31 != 5.605400076850724e+31 (rdiff > 3.9689909469779893e-14) > 137.0 42.0 => > 3.453905364934577e+35 != 3.453905364934566e+35 (rdiff > 3.2045019404969756e-15) > 137.0 46.0 => > 6.738158013916879e+36 != 6.738158013917095e+36 (rdiff > 3.206340162178697e-14) > 137.0 95.0 => > 3.453905364934577e+35 != 3.453905364934566e+35 (rdiff > 3.2045019404969756e-15) > 143.0 21.0 => > 7.639598299464504e+24 != 7.639598299464654e+24 (rdiff > 1.967693188404081e-14) > 143.0 29.0 => > 1.7138496567724698e+30 != 1.713849656772535e+30 (rdiff > 3.810263889765402e-14) > 143.0 33.0 => > 2.7947962468926867e+32 != 2.7947962468928852e+32 (rdiff > 7.10315364832748e-14) > 143.0 42.0 > => 2.91038363316998e+36 != 2.910383633170114e+36 > (rdiff 4.60410605063329e-14) > 143.0 46.0 => > 7.281844590777859e+37 != 7.281844590778005e+37 (rdiff > 2.010388427601942e-14) > 143.0 95.0 => > 3.0056975544912714e+38 != 3.005697554491347e+38 (rdiff > 2.5138212463529437e-14) > 143.0 122.0 => > 7.639598299464504e+24 != 7.639598299464654e+24 (rdiff > 1.967693188404081e-14) > 144.0 21.0 => > 8.943919960348817e+24 != 8.943919960348863e+24 (rdiff > 5.162266504697016e-15) > 144.0 29.0 => > 2.1460378310892105e+30 != 2.1460378310890872e+30 (rdiff > 5.744821363969209e-14) > 144.0 33.0 => > 3.6256816175907336e+32 != 3.62568161759077e+32 (rdiff > 1.0135852188743159e-14) > 144.0 42.0 => > 4.1087768938873574e+36 != 4.10877689388722e+36 (rdiff > 3.347442009280187e-14) > 144.0 46.0 => > 1.0699853276245263e+38 != 1.0699853276245233e+38 (rdiff > 2.8246317692471727e-15) > 144.0 122.0 > => 5.00046434146803e+25 != 5.000464341467773e+25 > (rdiff 5.136303886238907e-14) > 145.0 21.0 => > 1.045861608266637e+25 != 1.045861608266601e+25 (rdiff > 3.44956971374012e-14) > 145.0 29.0 > => 2.68254728886152e+30 != 2.6825472888613593e+30 > (rdiff 5.980909913173422e-14) > 145.0 33.0 => > 4.693962808488381e+32 != 4.69396280848805e+32 (rdiff > 7.046164831898694e-14) > 145.0 42.0 => > 5.784200481685929e+36 != 5.784200481685892e+36 (rdiff > 6.327294560089766e-15) > 145.0 46.0 => > 1.5671502273289442e+38 != 1.5671502273288471e+38 (rdiff > 6.195440181461709e-14) > 145.0 122.0 => > 3.152466650055975e+26 != 3.1524666500557704e+26 (rdiff > 6.495993880527083e-14) > 145.0 125.0 => > 1.7570475018879216e+24 != 1.7570475018878894e+24 (rdiff > 1.8333172373193666e-14) > 148.0 21.0 > => 1.66081725375304e+25 != 1.6608172537529971e+25 > (rdiff 2.586056525059899e-14) > 148.0 29.0 => > 5.186381531354783e+30 != 5.18638153135483e+30 (rdiff > 8.900597825568916e-15) > 148.0 33.0 => > 1.0064458536532161e+33 != 1.0064458536531621e+33 (rdiff > 5.369707206034169e-14) > 148.0 42.0 => > 1.5872551307291334e+37 != 1.5872551307291024e+37 (rdiff > 1.9487415642240105e-14) > 148.0 122.0 => > 6.418858594896177e+28 != 6.418858594895863e+28 (rdiff > 4.892155143322956e-14) > 148.0 125.0 > => 5.25225250880544e+26 != 5.252252508805427e+26 > (rdiff 2.3550858972873946e-15) > 149.0 21.0 => > 1.9332950844468164e+25 != 1.9332950844468484e+25 (rdiff > 1.655076176038335e-14) > 149.0 29.0 => > 6.439757068098981e+30 != 6.439757068098914e+30 (rdiff > 1.0490146397789524e-14) > 149.0 33.0 => > 1.2927623465028022e+33 != 1.2927623465027686e+33 (rdiff > 2.5974487045134947e-14) > 149.0 42.0 => > 2.2102898549405485e+37 != 2.210289854940526e+37 (rdiff > 1.0255378527439437e-14) > 149.0 122.0 => > 3.542259002368573e+29 != 3.542259002368458e+29 (rdiff > 3.2380764064090125e-14) > 149.0 125.0 => > 3.2607734325499903e+27 != 3.260773432550036e+27 (rdiff > 1.3993530521689755e-14) > > ====================================================================== > FAIL: test_data.test_boost(<Data for gamma: > test_gamma_data_ipp-factorials>,) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Python27\lib\site-packages\nose\case.py", line 197, in runTest > self.test(*self.arg) > File "C:\Python27\lib\site-packages\scipy\special\tests\test_data.py", > line 481, in _test_factory > test.check(dtype=dtype) > File "C:\Python27\lib\site-packages\scipy\special\_testutils.py", line > 292, in check > assert_(False, "\n".join(msg)) > File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 53, in > assert_ > raise AssertionError(smsg) > AssertionError: > Max |adiff|: 3.36416e+140 > Max |rdiff|: 3.56868e-14 > Bad results (35 out of 198) for the following points (in output 0): > 47.0 => 5.502622159812033e+57 > != 5.502622159812089e+57 (rdiff 1.0258520328008188e-14) > 50.0 => 6.082818640342751e+62 > != 6.082818640342675e+62 (rdiff 1.2463859588664233e-14) > 51.0 => 3.0414093201712955e+64 > != 3.0414093201713376e+64 (rdiff 1.3839389152907178e-14) > 52.0 => 1.5511187532873519e+66 > != 1.5511187532873822e+66 (rdiff 1.9537961157045428e-14) > 53.0 => 8.065817517094281e+67 > != 8.065817517094388e+67 (rdiff 1.3210853128030717e-14) > 55.0 => 2.308436973392385e+71 > != 2.308436973392414e+71 (rdiff 1.2533812334944586e-14) > 56.0 => 1.2696403353658055e+73 > != 1.2696403353658276e+73 (rdiff 1.742759977049922e-14) > 57.0 => 7.10998587804854e+74 > != 7.109985878048635e+74 (rdiff 1.3278171253713691e-14) > 59.0 => 2.350561331282903e+78 > != 2.3505613312828785e+78 (rdiff 1.050071233255035e-14) > 60.0 => 1.386831185456872e+80 > != 1.3868311854568984e+80 (rdiff 1.8984338680317015e-14) > 62.0 => 5.075802138772193e+83 > != 5.075802138772248e+83 (rdiff 1.083538910645766e-14) > 67.0 => 5.443449390774354e+92 > != 5.443449390774431e+92 (rdiff 1.403940285109478e-14) > 68.0 => 3.6471110918189365e+94 > != 3.647111091818868e+94 (rdiff 1.869380461041295e-14) > 71.0 => 1.1978571669969762e+100 != > 1.1978571669969892e+100 (rdiff 1.0865971283156417e-14) > 72.0 => 8.504785885678504e+101 > != 8.504785885678623e+101 (rdiff 1.4034165979338454e-14) > 73.0 => 6.123445837688725e+103 != > 6.1234458376886085e+103 (rdiff 1.8972113268364947e-14) > 74.0 => 4.470115461512762e+105 != > 4.4701154615126844e+105 (rdiff 1.7316637068366878e-14) > 79.0 => 1.1324281178206174e+115 != > 1.1324281178206297e+115 (rdiff 1.0816230951910165e-14) > 80.0 => 8.946182130782832e+116 > != 8.946182130782976e+116 (rdiff 1.611677089398549e-14) > 81.0 => 7.156945704626265e+118 > != 7.156945704626381e+118 (rdiff 1.614806559475051e-14) > 83.0 => 4.7536433370127884e+122 > != 4.753643337012842e+122 (rdiff 1.127054006184548e-14) > 84.0 => 3.9455239697205287e+124 > != 3.945523969720659e+124 (rdiff 3.297642089724322e-14) > 85.0 => 3.314240134565268e+126 > != 3.314240134565353e+126 (rdiff 2.5759817183612377e-14) > 87.0 => 2.422709538367351e+130 != > 2.4227095383672734e+130 (rdiff 3.21206322943374e-14) > 89.0 => 1.8548264225739368e+134 != > 1.8548264225739844e+134 (rdiff 2.5668295182660588e-14) > 90.0 => 1.6507955160908254e+136 > != 1.650795516090846e+136 (rdiff 1.2513966038394803e-14) > 91.0 => 1.485715964481777e+138 != > 1.4857159644817615e+138 (rdiff 1.0456113845414324e-14) > 92.0 => 1.3520015276784442e+140 > != 1.352001527678403e+140 (rdiff 3.05102410078959e-14) > 93.0 => 1.243841405464157e+142 != > 1.2438414054641308e+142 (rdiff 2.1115671814606213e-14) > 94.0 => 1.1567725070816173e+144 != > 1.1567725070816416e+144 (rdiff 2.0972887646477295e-14) > 95.0 => 1.0873661566567573e+146 > != 1.087366156656743e+146 (rdiff 1.3055463191484954e-14) > 96.0 => 1.0329978488238868e+148 > != 1.032997848823906e+148 (rdiff 1.8552500324741778e-14) > 97.0 => 9.916779348709327e+149 > != 9.916779348709496e+149 (rdiff 1.7040815113096152e-14) > 98.0 => 9.61927596824809e+151 > != 9.619275968248212e+151 (rdiff 1.2694188843809571e-14) > 99.0 => 9.426890448883584e+153 > != 9.426890448883248e+153 (rdiff 3.568683137742694e-14) > > ---------------------------------------------------------------------- > Ran 17009 tests in 2055.336s > > FAILED (KNOWNFAIL=95, SKIP=1211, errors=2, failures=11) > > >>> >

1 0

Datetime again
by Charles R Harris Jan. 29, 2015

Jan. 29, 2015

Hi All, I'm playing with the idea of building a simplified datetime class on top of the current numpy implementation. I believe Pandas does something like this, and blaze will (does?) have a simplified version. The reason for the new class would be to have an easier, and hopefully more portable, API that can be implemented in Python, and maybe pushed down into C when things settle. Thoughts? Chuck

6 11

Float view of complex array
by Jens Jørgen Mortensen Jan. 28, 2015

Jan. 28, 2015

Hi! I have a view of a 2-d complex array that I would like to view as a 2-d float array. This works OK: >>> np.ones((2, 4), complex).view(float) array([[ 1., 0., 1., 0., 1., 0., 1., 0.], [ 1., 0., 1., 0., 1., 0., 1., 0.]]) but this doesn't: >>> np.ones((2, 4), complex)[:, :2].view(float) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: new type not compatible with array. >>> np.__version__ '1.9.0' and I don't understand why. When looking at the memory layout, I think it should be possible. Jens Jørgen

4 7

F2PY cannot see module-scope variables
by Yuxiang Wang Jan. 27, 2015

Jan. 27, 2015

Dear all, Sorry about being new to both Fortran 90 and f2py. I have a module in fortran, written as follows, with a module-scope variable dp: ======================================== ! testf2py.f90 module testf2py implicit none private public dp, i1 integer, parameter :: dp=kind(0.d0) contains real(dp) function i1(m) real(dp), intent(in) :: m(3, 3) i1 = m(1, 1) + m(2, 2) + m(3, 3) return end function i1 end module testf2py ======================================== Then, if I run f2py -c testf2py.f90 -m testf2py It would report an error, stating that dp was not declared. If I copy the module-scope to the function-scope, it would work. ======================================== ! testf2py.f90 module testf2py implicit none private public i1 integer, parameter :: dp=kind(0.d0) contains real(dp) function i1(m) integer, parameter :: dp=kind(0.d0) real(dp), intent(in) :: m(3, 3) i1 = m(1, 1) + m(2, 2) + m(3, 3) return end function i1 end module testf2py ======================================== However, this does not look like the best coding practice though, as it is pretty "wet". Any ideas? Thanks, Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj(a)virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/

2 2

New function `count_unique` to generate contingency tables.
by Warren Weckesser Jan. 25, 2015

Jan. 25, 2015

I created a pull request (https://github.com/numpy/numpy/pull/4958) that defines the function `count_unique`. `count_unique` generates a contingency table from a collection of sequences. For example, In [7]: x = [1, 1, 1, 1, 2, 2, 2, 2, 2] In [8]: y = [3, 4, 3, 3, 3, 4, 5, 5, 5] In [9]: (xvals, yvals), counts = count_unique(x, y) In [10]: xvals Out[10]: array([1, 2]) In [11]: yvals Out[11]: array([3, 4, 5]) In [12]: counts Out[12]: array([[3, 1, 0], [1, 1, 3]]) It can be interpreted as a multi-argument generalization of `np.unique(x, return_counts=True)`. It overlaps with Pandas' `crosstab`, but I think this is a pretty fundamental counting operation that fits in numpy. Matlab's `crosstab` (http://www.mathworks.com/help/stats/crosstab.html) and R's `table` perform the same calculation (with a few more bells and whistles). For comparison, here's Pandas' `crosstab` (same `x` and `y` as above): In [28]: import pandas as pd In [29]: xs = pd.Series(x) In [30]: ys = pd.Series(y) In [31]: pd.crosstab(xs, ys) Out[31]: col_0 3 4 5 row_0 1 3 1 0 2 1 1 3 And here is R's `table`: > x <- c(1,1,1,1,2,2,2,2,2) > y <- c(3,4,3,3,3,4,5,5,5) > table(x, y) y x 3 4 5 1 3 1 0 2 1 1 3 Is there any interest in adding this (or some variation of it) to numpy? Warren

5 11

What should recfromcsv defaults be?
by Charles R Harris Jan. 24, 2015

Jan. 24, 2015

Hi All, This question comes apropos this bugfix #5495 <https://github.com/numpy/numpy/pull/5495> that ensures that the default options get passed down the call chain. The current defaults are kwargs.setdefault("case_sensitive", "lower") kwargs.setdefault("names", True) kwargs.setdefault("delimiter", ",") kwargs.setdefault("dtype", None) The ones in question are for "names" and "case_sensitive", that, due to the bug, were defaulting to 'True' and None respectively. I think those defaults should be kept rather than the values currently specified in the recfromcsv definition. However, I don't use these tools, so would like some feedback from those who do. Chuck

1 0

Aligned array allocations
by Antoine Pitrou Jan. 22, 2015

Jan. 22, 2015

Hello, In https://github.com/numpy/numpy/issues/5312 there's a request for an aligned allocator in Numpy (more than the default alignment of the platform's memory allocator). The reason is that on modern vectorization instruction sets, a certain alignment is required for optimal performance (even though unaligned data still works: it's just that performance is degraded... by how much will depend on the CPU micro-architecture). For example Intel recommends a 32-byte alignment for AVX loads and stores. In https://github.com/numpy/numpy/pull/5457 I have proposed a patch to wrap the system allocator in an aligned allocator. The proposed scheme makes the alignment configurable at runtime (through a Python API), because different platforms may have different desirable alignments, and it is not reasonable for Numpy to know about them all, nor for users to recompile Numpy each time they have a different CPU. By always using an aligned allocator there is some overhead: - all arrays occupy a bit more memory by a small average amount (probably 16 bytes average on a 64-bit machine, for a 16 byte guaranteed alignment) - array resizes can be more expensive in CPU time, when the physical start changes and its alignment changes too There is also a limitation: while the physical start of an array will always be aligned, this can be defeated when taking a view starting at a non-zero index. (note that to take advantage of certain instruction set features such as AVX, Numpy may need to be compiled with specific compiler flags... but Numpy's allocations also affect other packages such as Numba which is able to generate code at runtime) I would like to know if people are interested in this feature, and if the proposed approach is acceptable. Regards Antoine.

2 1

EDF+ specification
by Eric Firing Jan. 20, 2015

Jan. 20, 2015

http://www.edfplus.info/specs/edfplus.html#additionalspecs Io, Is this the file format you have? Eric

2 2

structured arrays, recarrays, and record arrays
by Allan Haldane Jan. 19, 2015

Jan. 19, 2015

Hello all, Documentation of recarrays is poor and I'd like to improve it. In order to do this I've been looking at core/records.py, and I would appreciate some feedback on my plan. Let me start by describing what I see. In the docs there is some confusion about 'structured arrays' vs 'record arrays' vs 'recarrays' - the docs use them often interchangeably. They also refer to structured dtypes alternately as 'struct data types', 'record data types' or simply 'records' (eg, see the reference/arrays.dtypes and reference/arrays.indexing doc pages). But by my reading of the code there are really three (or four) distinct types of arrays with structure. Here's a possible nomenclature: * "Structured arrays" are simply ndarrays with structured dtypes. That is, the data type is subdivided into fields of different type. * "recarrays" are a subclass of ndarrays that allow access to the fields by attribute. * "Record arrays" are recarrays where the elements have additionally been converted to 'numpy.core.records.record' type such that each data element is an object with field attributes. * (it is also possible to create arrays with dtype.dtype of numpy.core.records.record, but which are not recarrays. However I have never seen this done.) Here's code demonstrating the creation of the different types of array (in order: structured array, recarray, ???, record array). >>> arr = np.array([(1,'a'), (2,'b')], dtype=[('foo', int), ('bar', 'S1')]) >>> recarr = arr.view(type=np.recarray) >>> noname = arr.view(dtype=dtype(np.record, arr.dtype)) >>> recordarr = arr.view(dtype=dtype((np.record, arr.dtype)), type=np.recarray) >>> type(arr), arr.dtype.type (numpy.ndarray, numpy.void) >>> type(recarr), recarr.dtype.type (numpy.core.records.recarray, numpy.void) >>> type(recordarr), recordarr.dtype.type (numpy.core.records.recarray, numpy.core.records.record) Note that the functions numpy.rec.array, numpy.rec.fromrecords, numpy.rec.fromarrays, and np.recarray.__new__ create record arrays. However, in the docs you can see examples of the creation of recarrays, eg in the recarray and ndarray.view doctrings and in http://www.scipy.org/Cookbook/Recarray. The files numpy/lib/recfunctions.py and numpy/lib/npyio.py (and possibly masked arrays, but I haven't looked yet) make extensive use of recarrays (but not record arrays). The main functional difference between recarrays and record arrays is field access on individual elements: >>> recordarr[0].foo 1 >>> recarr[0].foo Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'numpy.void' object has no attribute 'foo' Also, note that recarrays have a small performance penalty relative to structured arrays, and record arrays have another one relative to recarrays because of the additional python logic. So my first goal in updating the docs is to use the right terms in the right place. In almost all cases, references to 'records' (eg 'record types') should be replaced with 'structured' (eg 'structured types'), with the exception of docs that deal specifically with record arrays. It's my guess that in the distant past structured datatypes were intended to always be of type numpy.core.records.record (thus the description in reference/arrays.dtypes) but that numpy.core.records.record became generally obsolete without updates to the docs. doc/records.rst.txt seems to document the transition. I've made a preliminary pass of the docs, which you can see here https://github.com/ahaldane/numpy/commit/d87633b228dabee2ddfe75d1ee9e41ba70… Mostly I renamed 'record type' to 'structured type', and added a very rough draft to numpy/doc/structured_arrays.py. I would love to hear from those more knowledgeable than myself on whether this works! Cheers, Allan

1 1

ANN: Scipy 0.15.1
by Pauli Virtanen Jan. 18, 2015

Jan. 18, 2015

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, We are pleased to announce the Scipy 0.15.1 release. Scipy 0.15.1 contains only bugfixes. The module ``scipy.linalg.calc_lwork`` removed in Scipy 0.15.0 is restored. This module is not a part of Scipy's public API, and although it is available again in Scipy 0.15.1, using it is deprecated and it may be removed again in a future Scipy release. Source tarballs, binaries, and full release notes are available at https://sourceforge.net/projects/scipy/files/scipy/0.15.1/ Best regards, Pauli Virtanen ========================== SciPy 0.15.1 Release Notes ========================== SciPy 0.15.1 is a bug-fix release with no new features compared to 0.15.0. Issues fixed - ------------ * `#4413 <https://github.com/scipy/scipy/pull/4413>`__: BUG: Tests too strict, f2py doesn't have to overwrite this array * `#4417 <https://github.com/scipy/scipy/pull/4417>`__: BLD: avoid using NPY_API_VERSION to check not using deprecated... * `#4418 <https://github.com/scipy/scipy/pull/4418>`__: Restore and deprecate scipy.linalg.calc_work -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlS8CA4ACgkQ6BQxb7O0pWCmOQCgzg9AXDaqRaK5/QBWopIrv2OA WkEAn0ltDfDHFpw0zMzB9mUscAAb2xnE =JrGj -----END PGP SIGNATURE-----

1 0

using f2py with a module containing a derived type?
by Greg Fischer Jan. 17, 2015

Jan. 17, 2015

Hello, I would like to use f2py to wrap a Fortran module that contains a derived data type. I don't necessarily need to access the data that is inside the derived type from Python, but I would really like to be able to call the subroutines that are contained inside the module. When I attempt to use f2py on this module, it appears to choke when it gets to the derived data type. Is there any way around this? Thanks, Greg

2 1

Re: [Numpy-discussion] Sorting refactor
by Lars Buitinck Jan. 17, 2015

Jan. 17, 2015

2015-01-16 11:55 GMT+01:00 <numpy-discussion-request(a)scipy.org>: > Message: 2 > Date: Thu, 15 Jan 2015 21:24:00 -0800 > From: Jaime Fern?ndez del R?o <jaime.frio(a)gmail.com> > Subject: [Numpy-discussion] Sorting refactor > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAPOWHWkF6RnWcrGmcwsmq_LO3hShjgBVLsrN19z-MDPe25E2Aw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > This changes will make it easier for me to add a Timsort generic type > function to numpy's arsenal of sorting routines. And I think they have > value by cleaning the source code on their own. Yes, they do. I've been looking at the sorting functions as well and I've found the following: * The code is generally hard to read because it prefers pointer over indices. I'm wondering if it would get slower using indices. The closer these algorithms are to the textbook, the easier to insert fancy optimizations. * The heap sort exploits undefined behavior by using a pointer that points before the start of the array. However, rewriting it to always point within the array made it slower. I haven't tried rewriting it using indices. * Quicksort has a quadratic time worst case. I think it should be turned into an introsort [1] for O(n log n) worst case; we have the heapsort needed to do that. * Quicksort is robust to repeated elements, but doesn't exploit them. It can be made to run in linear time if the input array has only O(1) distinct elements [2]. This may come at the expense of some performance on arrays with no repeated elements. * Using optimal sorting networks instead of insertion sort as the base case can speed up quicksort on float arrays by 5-10%, but only if NaNs are moved out of the way first so that comparisons become cheaper [3]. This has consequences for the selection algorithms that I haven't fully worked out yet. * Using Cilk Plus to parallelize merge sort can make it significantly faster than quicksort at the expense of only a few lines of code, but I haven't checked whether Cilk Plus plays nicely with multiprocessing and other parallelism options (remember the trouble with OpenMP-ified OpenBLAS?). This isn't really an answer to your questions, more like a brain dump from someone who's stared at the same code for a while and did some experiments. I'm not saying we should implement all of this, but keep in mind that there are some interesting options besides implementing timsort. [1] https://en.wikipedia.org/wiki/Introsort [2] http://www.sorting-algorithms.com/static/QuicksortIsOptimal.pdf [3] https://github.com/larsmans/numpy/tree/sorting-nets

6 6

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 100, Issue 28
by Lars Buitinck Jan. 16, 2015

Jan. 16, 2015

2015-01-16 13:29 GMT+01:00 <numpy-discussion-request(a)scipy.org>: > Date: Fri, 16 Jan 2015 12:43:43 +0100 > From: Julian Taylor <jtaylor.debian(a)googlemail.com> > Subject: Re: [Numpy-discussion] Sorting refactor > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: <54B8F96F.7090903(a)googlemail.com> > Content-Type: text/plain; charset=windows-1252 > > On 16.01.2015 12:33, Lars Buitinck wrote: >> * Quicksort has a quadratic time worst case. I think it should be >> turned into an introsort [1] for O(n log n) worst case; we have the >> heapsort needed to do that. > > This probably rarely happens in numeric data, and we do have guaranteed > nlog runtime algorithms available. It's no more likely or unlikely than in any other type of data (AFAIK), but it's possible for an adversary to DOS any (web server) code that uses np.sort. > I was also thinking about this, an advantage of a sorting network is > that it can be vectorized to be significantly faster than an insertion > sort. Handling NaN's should also be directly possible. Tried that, and it didn't give any speedup, at least not without explicit vector instructions. Just moving the NaNs aside didn't cost anything in my preliminary benchmarks (without sorting nets), the cost of the operation was almost exactly compensated by simpler comparisons. > The issue is that its probably too much complicated code for only a very > small gain. Maybe. The thing is that the selection algorithms are optimized for NaNs and seem to depend on the current comparison code. We'd need distinct <TYPE>_LT and <TYPE>_LT_NONAN for each <TYPE>. The sorting nets themselves aren't complicated, just lengthy. My branch has the length-optimal (not optimally parallel) ones for n <= 16. > But I assume you will get the same trouble as openmp but that needs > testing, also cilk+ in gcc is not really production ready yet, I got > lots of crashes when I last tried it (it will be in 5.0 though). The data parallel constructs tend to crash the compiler, but task spawning seems to be stable in 4.9.2. I've still to see how it handles multiprocessing/fork. What do you mean by will be in 5.0, did they do a big push? > Date: Fri, 16 Jan 2015 13:28:56 +0100 > From: Da?id <davidmenhur(a)gmail.com> > Subject: Re: [Numpy-discussion] Sorting refactor > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAJhcF=1O5Own_5ydzu+To8HHbm3e66k=iuNQrEiaSDY23DNW6w(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On 16 January 2015 at 13:15, Eelco Hoogendoorn > <hoogendoorn.eelco(a)gmail.com> wrote: >> Perhaps an algorithm can be made faster, but often these multicore >> algorithms are also less efficient, and a less data-dependent way of putting >> my cores to good use would have been preferable. Potentially, other code >> could slow down due to cache trashing if too many parallel tasks run in >> parallel. Id rather be in charge of such matters myself; but I imagine >> adding a keyword arg for these matters would not be much trouble? > > As I understand it, that is where the strength of Cilk+ lies. It does > not force parallelisation, just suggests it. The decision to actually > spawn parallel is decided at runtime depending on the load of the > other cores. cilk+ guarantees that the amount of space used by a pool of P threads is at most P times the stack space used by the sequential version (+ a constant). The idea is that you can say for (i = 0; i < 1000000; i++) { cilk_spawn f(a[i]); } and it will never create more than P work items in memory, rather than 1e6, even if each f() spawns a bunch itself. Of course, it won't guarantee that OpenMP will not also spawn P threads and/or check that you're one of P processes cooperating on a task using multiprocessing. Personally I'd rather have an np.setnumthreads() to turn this on or off for a process and have the runtime distribute work for me instead of having to do it myself.

3 2

Re: [Numpy-discussion] Sorting refactor
by Lars Buitinck Jan. 16, 2015

Jan. 16, 2015

2015-01-16 15:14 GMT+01:00 <numpy-discussion-request(a)scipy.org>: > Date: Fri, 16 Jan 2015 06:11:29 -0800 > From: Jaime Fern?ndez del R?o <jaime.frio(a)gmail.com> > Subject: Re: [Numpy-discussion] Sorting refactor > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > > Most of my proposed original changes do not affect the core sorting > functionality, just the infrastructure around it. But if we agree that > sorting has potential for being an actively developed part of the code > base, then cleaning up its surroundings for clarity makes sense, so I'm > taking your brain dump as an aye for my proposal. ;-) It is!

1 0

Sorting refactor
by Jaime Fernández del Río Jan. 16, 2015

Jan. 16, 2015

Hi all, I have been taking a deep look at the sorting functionality in numpy, and I think it could use a face lift in the form of a big code refactor, to get rid of some of the ugliness in the code and make it easier to maintain. What I have in mind basically amounts to: 1. Refactor _new_argsortlike to get rid of code duplication (there are two branches, one with buffering, one without, virtually identical, that could be merged into a single one). 2. Modify _new_argsortlike so that it can properly handle byte-swapped inputs of any dtype, see gh-5441. Add proper handling of types with references, in preparation for the rest of changes. 3. Add three functions to the npy_sort library: npy_aquicksort, npy_aheapsort, npy_amergesort, with a signature compatible with PyArray_ArgSortFunc , i.e. (char* start, npy_intp* result, npy_intp length, PyArrayObject *arr). These turn out to be almost identical to the string and unicode sort functions, but using the dtype's compare function to handle comparisons. 4. Modify PyArray_ArgSort (and PyArray_ArgPartition) to always call _new_argsortlike, even when there is no type specific argsort function, by using the newly added npy_axxx functions. This simplifies PyArray_ArgSort a lot, and gets rid of some of the global variable ugliness in the current code. And makes argsorting over non-contiguous axis more memory efficient. 5. Refactor _new_sortlike similarly to _new_argsortlike 6. Modify the npy_quicksort, npy_mergesort and npy_heapsort functions in npy_sort to have a signature compatible with PyArray_SortFunc, i.e. (char* start, npy_intp length, PyArrayObject *arr). npy_quicksort will no longer rely on libc's qsort, but be very similar to the string or unicode quicksort functions 7. Modify PyArray_Sort (and PyArray_Partition) to always call _new_sortlike, similarly to what was done with PyArray_ArgSort. This allows completing the removal of the remaining global variable ugliness, as well as similar benefits as for argsort before. This changes will make it easier for me to add a Timsort generic type function to numpy's arsenal of sorting routines. And I think they have value by cleaning the source code on their own. So my questions, mostly to the poor souls that will have to code review changes to several hundred lines of code: 1. Does this make sense, or is it better left alone? A subset of 1, 2 and 5 are a must to address the issues in gh-5441, the rest could arguably be left as is. 2. Would you rather see it submitted as one ginormous PR? Or split into 4 or 5 incremental ones? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 2

Optimizing numpy's einsum expression (again)
by Daniel Smith Jan. 16, 2015

Jan. 16, 2015

Hello everyone, I originally brought an optimized einsum routine forward a few weeks back that attempts to contract numpy arrays together in an optimal way. This can greatly reduce the scaling and overall cost of the einsum expression for the cost of a few intermediate arrays. The current version (and more details) can be found here: https://github.com/dgasmith/opt_einsum <https://github.com/dgasmith/opt_einsum> I think this routine is close to a finalized version, but there are two problems which I would like the community to weigh in on: Memory usage- Currently the routine only considers the maximum size of intermediates created rather than cumulative memory usage. If we only use np.einsum it is straightforward to calculate cumulative memory usage as einsum does not require any copies of inputs to be made; however, if we attempt to use a vendor BLAS the memory usage can becomes quite complex. For example, the np.tensordot routine always forces a copy for ndarrays because it uses ndarray.transpose(…).reshape(…). A more memory-conscious way to do this is to first try and do ndarray.reshape(…).T which does not copy the data and numpy can just pass a transpose flag to the vendor BLAS. The caveat here is that the summed indices must be in the correct order- if not a copy is required. Maximum array size is usually close to the total overhead of the opt_einsum function, but can occasionally be 2-5 times this size. I see the following ways forward: Ignore cumulative memory and stick with maximum array size as the limiting memory factor. Implement logic to figure out if the input arrays needs to be copied to use BLAS, compute the extra memory required, and add an extra dimension to the optimization algorithms (use BLAS or do not use BLAS at each step). Some of this is already done, but may get fairly complex. Build an in-place nd-transpose algorithm. Cut out BLAS entirely. Keeping in mind that vendor BLAS can be orders of magnitude faster than a pure einsum implementation, especially if the BLAS threading is used. Path algorithms- There are two algorithms “optimal” (a brute force algorithm, scales like N!) and “opportunistic” (a greedy algorithm, scales like N^3). The optimal path can take seconds to minutes to calculate for a 7-10 term expression while the opportunistic path takes microseconds even for 20+ term expressions. The opportunistic algorithm works surprisingly well and appears to obtain the correct scaling in all test cases that I can think of. Both algorithms use the maximum array size as a sieve- this is very beneficial from several aspects. The problem occurs when a much needed intermediate cannot be created due to this limit- on occasions not making this intermediate can have slowdowns of orders of magnitude even for small systems. This leads to certain (and sometimes unexpected) performance walls. Possible solutions: Warn the user if the ratio between an unlimited memory solution and a limited memory solution becomes large. Do not worry about it. Thank you for your time, -Daniel Smith

3 2

The future of ndarray.diagonal()
by Alexander Belopolsky Jan. 15, 2015

Jan. 15, 2015

A discussion [1] is currently underway at GitHub which will benefit from a larger forum. In version 1.9, the diagonal() method was changed to return a read-only (non-contiguous) view into the original array instead of a plain copy. Also, it has been announced [2] that in 1.10 the view will become read/write. A concern has now been raised [3] that this change breaks backward compatibility too much. Consider the following code: x = numy.eye(2) d = x.diagonal() d[0] = 2 In 1.8, this code runs without errors and results in [2, 1] stored in array d. In 1.9, this is an error. With the current plan, in 1.10 this will become valid again, but the result will be different: x[0,0] will be 2 while it is 1 in 1.8. Two alternatives are suggested for discussion: 1. Add copy=True flag to diagonal() method. 2. Roll back 1.9 change to diagonal() and introduce an additional diagonal_view() method to return a view. [1] https://github.com/numpy/numpy/pull/5409 [2] http://docs.scipy.org/doc/numpy/reference/generated/numpy.diagonal.html [3] http://khinsen.wordpress.com/2014/09/12/the-state-of-numpy/

11 19

Memory allocation of arrays and tracemalloc
by Antoine Pitrou Jan. 15, 2015

Jan. 15, 2015

Hello, I see that the PyDataMem_* APIs call malloc()/free()/etc. directly, instead of going through PyMem_Malloc, etc. This means the memory allocated by those APIs won't be seen by tracemalloc. Is it deliberate? Am I missing something? Regards Antoine.

2 1

proposed change to recarray access
by Allan Haldane Jan. 14, 2015

Jan. 14, 2015

Hello all, I've submitted a pull request on github which changes how string values in recarrays are returned, which may break old code. https://github.com/numpy/numpy/pull/5454 See also: https://github.com/numpy/numpy/issues/3993 Previously, recarray fields of type 'S' or 'U' (ie, strings) would be returned as chararrays when accessed by attribute, but ndarrays when accessed by indexing: >>> arr = np.array([('abc ', 1), ('abc', 2)], dtype=[('str', 'S4'), ('id', int)]) >>> arr = arr.view(np.recarray) >>> type(arr.str) numpy.core.defchararray.chararray >>> type(arr['str']) numpy.core.records.recarray Chararray is deprecated, and furthermore this led to bugs in my code since chararrays trim trailing whitespace but but ndarrays do not (and I was not aware of conversion to chararray). For example: >>> arr.str[0] == arr.str[1] True >>> arr['str'][0] == arr['str'][1] False In the pull request I have changed recarray attribute access so ndarrays are always returned. I think this is a sensible thing to do but it may break code which depends on chararray features (including the trimmed whitespace). Does this sound reasonable? Best, Allan

1 0

linspace handling of extra return
by Jaime Fernández del Río Jan. 14, 2015

Jan. 14, 2015

While working on something else, I realized that linspace is not handling requests for returning the sampling spacing consistently: >>> np.linspace(0, 1, 3, retstep=True) (array([ 0. , 0.5, 1. ]), 0.5) >>> np.linspace(0, 1, 1, retstep=True) array([ 0.]) >>> np.linspace(0, 1, 0, retstep=True) array([], dtype=float64) Basically, retstep is ignored if the number of samples is 0 or 1. One could argue that it makes sense, because those sequences do not have a spacing defined. But at the very least it should be documented as doing so, and the following inconsistency removed: >>> np.linspace(0, 1, 1, endpoint=True, retstep=True) array([ 0.]) >>> np.linspace(0, 1, 1, endpoint=False, retstep=True) (array([ 0.]), 1.0) I am personally inclined to think that if a step is requested, then a step should be returned, and if it cannot be calculated in a reasonable manner, then a placeholder such as None, nan, 0 or stop - start should be returned. What does the collective wisdom think is the best approach for this? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 6

Equality of dtypes does not imply equality of type kinds
by Alexander Belopolsky Jan. 13, 2015

Jan. 13, 2015

Consider this (on a 64-bit platform): >>> numpy.dtype('q') == numpy.dtype('l') True but >>> numpy.dtype('q').char == numpy.dtype('l').char False Is that intended? Shouldn't dtype constructor "normalize" 'l' to 'q' (or 'i')?

5 5

ANN: Scipy 0.15.0 release
by Pauli Virtanen Jan. 12, 2015

Jan. 12, 2015

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear all, We are pleased to announce the Scipy 0.15.0 release. The 0.15.0 release contains bugfixes and new features, most important of which are mentioned in the excerpt from the release notes below. Source tarballs, binaries, and full release notes are available at https://sourceforge.net/projects/scipy/files/scipy/0.15.0/ Best regards, Pauli Virtanen ========================== SciPy 0.15.0 Release Notes ========================== SciPy 0.15.0 is the culmination of 6 months of hard work. It contains several new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.16.x branch, and on adding new features on the master branch. This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.5.1 or greater. New features ============ Linear Programming Interface - ---------------------------- The new function `scipy.optimize.linprog` provides a generic linear programming similar to the way `scipy.optimize.minimize` provides a generic interface to nonlinear programming optimizers. Currently the only method supported is *simplex* which provides a two-phase, dense-matrix-based simplex algorithm. Callbacks functions are supported, allowing the user to monitor the progress of the algorithm. Differential evolution, a global optimizer - ------------------------------------------ A new `scipy.optimize.differential_evolution` function has been added to the ``optimize`` module. Differential Evolution is an algorithm used for finding the global minimum of multivariate functions. It is stochastic in nature (does not use gradient methods), and can search large areas of candidate space, but often requires larger numbers of function evaluations than conventional gradient based techniques. ``scipy.signal`` improvements - ----------------------------- The function `scipy.signal.max_len_seq` was added, which computes a Maximum Length Sequence (MLS) signal. ``scipy.integrate`` improvements - -------------------------------- It is now possible to use `scipy.integrate` routines to integrate multivariate ctypes functions, thus avoiding callbacks to Python and providing better performance. ``scipy.linalg`` improvements - ----------------------------- The function `scipy.linalg.orthogonal_procrustes` for solving the procrustes linear algebra problem was added. BLAS level 2 functions ``her``, ``syr``, ``her2`` and ``syr2`` are now wrapped in ``scipy.linalg``. ``scipy.sparse`` improvements - ----------------------------- `scipy.sparse.linalg.svds` can now take a ``LinearOperator`` as its main input. ``scipy.special`` improvements - ------------------------------ Values of ellipsoidal harmonic (i.e. Lame) functions and associated normalization constants can be now computed using ``ellip_harm``, ``ellip_harm_2``, and ``ellip_normal``. New convenience functions ``entr``, ``rel_entr`` ``kl_div``, ``huber``, and ``pseudo_huber`` were added. ``scipy.sparse.csgraph`` improvements - ------------------------------------- Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching`` for computing reorderings of sparse graphs were added. ``scipy.stats`` improvements - ---------------------------- Added a Dirichlet multivariate distribution, `scipy.stats.dirichlet`. The new function `scipy.stats.median_test` computes Mood's median test. The new function `scipy.stats.combine_pvalues` implements Fisher's and Stouffer's methods for combining p-values. `scipy.stats.describe` returns a namedtuple rather than a tuple, allowing users to access results by index or by name. Deprecated features =================== The `scipy.weave` module is deprecated. It was the only module never ported to Python 3.x, and is not recommended to be used for new code - use Cython instead. In order to support existing code, ``scipy.weave`` has been packaged separately: https://github.com/scipy/weave. It is a pure Python package, and can easily be installed with ``pip install weave``. `scipy.special.bessel_diff_formula` is deprecated. It is a private function, and therefore will be removed from the public API in a following release. ``scipy.stats.nanmean``, ``nanmedian`` and ``nanstd`` functions are deprecated in favor of their numpy equivalents. Backwards incompatible changes ============================== scipy.ndimage - ------------- The functions `scipy.ndimage.minimum_positions`, `scipy.ndimage.maximum_positions`` and `scipy.ndimage.extrema` return positions as ints instead of floats. scipy.integrate - --------------- The format of banded Jacobians in `scipy.integrate.ode` solvers is changed. Note that the previous documentation of this feature was erroneous. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlSyt/cACgkQ6BQxb7O0pWA8SACfXmpUsJcXT5espj71OYpeaj5b JJwAoL10ud3q1f51A5Ij4lgqMeZGnHlj =ZmOl -----END PGP SIGNATURE-----

5 4

[ANN] bcolz v0.8.0
by Valentin Haenel Jan. 9, 2015

Jan. 9, 2015

====================== Announcing bcolz 0.8.0 ====================== What's new ========== This version adds a public API in the form of a Cython definitions file (``carray_ext.pxd``) for the ``carray`` class! This means, other libraries can use the Cython definitions to build more complex programs using the objects provided by bcolz. In fact, this feature was specifically requested and there already exists a nascent application called *bquery* (https://github.com/visualfabriq/bquery) which provides an efficient out-of-core groupby implementation for the ``ctable`` object Because this is a fairly sweeping change, the minor version number was incremented and no additional major features or bugfixes were added to this release. We kindly ask any users of bcolz to try this version carefully and report back any issues, bugs, or even slow-downs you experience. I.e. please, please be careful when deploying this version into production. Many, many kudos to Francesc Elies and Carst Vaartjes of Visualfabriq for their hard work, continued effort to push this feature and their work on bquery which makes use of it! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/) and Quantopian (https://www.quantopian.com/) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!**

1 0

Build doesn't pass tests
by Nikolay Mayorov Jan. 9, 2015

Jan. 9, 2015

Hi all! I'm trying to build numpy on Windows 64 bit, Python 3.4.2 64 bit. I do environment setup by the following command: CMD /K "SET MSSdk=1 && SET DISTUTILS_USE_SDK=1 && "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" x86_amd64" Then I cd to the newly cloned numpy folder and do: python setup.py build_ext --inplace It looks like the build process finishes correctly. But then python -c "import numpy; numpy.test()" crashes the interpreter (some tests pass before the crash). I found out that it is caused by numpy.fromfile function call. What might be the reason of that? Do I use wrong msvc compiler?

2 2

Proceedings of EuroSciPy 2014
by Ralf Gommers Jan. 6, 2015

Jan. 6, 2015

On Tue, Dec 23, 2014 at 7:06 PM, <pdebuyl(a)ulb.ac.be> wrote: > Dear scientist using Python, > > We are glad to announce the publication of the proceedings of the 7th > European > Conference on Python in Science, EuroSciPy 2014, still in 2014! > > The proceedings cover various scientific fields in which Python and its > scientific libraries are used. You may obtain the table of contents and > all the > articles on the arXiv at http://arxiv.org/abs/1412.7030 > For convenience, the articles' titles are listed below. > > It is a useful reference to have as the publication of software-related > scientific work is not always straightforward. > > Thanks go to all authors and reviewers for their contributions. The reviews > were conducted publicly at > https://github.com/euroscipy/euroscipy_proceedings > > Pierre de Buyl & Nelle Varoquaux, editors > > PS: there was no large announcement of the proceedings of EuroSciPy 2013. > In the > hope that this can increase their visibility, here is the URL > Proceedings of EuroSciPy 2013: http://arxiv.org/abs/1405.0166 > > Pierre de Buyl, Nelle Varoquaux: Preface > Jérôme Kieffer, Giannis Ashiotis: PyFAI: a Python library for high > performance azimuthal integration on GPU > Andrew Leonard, Huw Morgan: Temperature diagnostics of the solar > atmosphere using SunPy > Bastian Venthur, Benjamin Blankertz: Wyrm, A Pythonic Toolbox for > Brain-Computer Interfacing > Christophe Pouzat, Georgios Is. Detorakis: SPySort: Neuronal Spike Sorting > with Python > Thomas Cokelaer, Julio Saez-Rodriguez: Using Python to Dive into > Signalling Data with CellNOpt and BioServices > Davide Monari, Francesco Cenni, Erwin Aertbeliën, Kaat Desloovere: > Py3DFreeHandUS: a library for voxel-array reconstruction using > Ultrasonography and attitude sensors > Esteban Fuentes, Hector E. Martinez: SClib, a hack for straightforward > embedded C functions in Python > Jamie A Dean, Liam C Welsh, Kevin J Harrington, Christopher M Nutting, > Sarah L Gulliford: Predictive Modelling of Toxicity Resulting from > Radiotherapy Treatments of Head and Neck Cancer > Rebecca R. Murphy, Sophie E. Jackson, David Klenerman: pyFRET: A Python > Library for Single Molecule Fluorescence Data Analysis > Robert Cimrman: Enhancing SfePy with Isogeometric Analysis > Steve Brasier, Fred Pollard: A Python-based Post-processing Toolset For > Seismic Analyses > Vladimír Lukeš, Miroslav Jiřík, Alena Jonášová, Eduard Rohan, Ondřej > Bublík, Robert Cimrman: Numerical simulation of liver perfusion: from CT > scans to FE model > > _______________________________________________ > euroscipy-org mailing list > euroscipy-org(a)python.org > https://mail.python.org/mailman/listinfo/euroscipy-org >

1 0

edge-cases of ellipsis indexing
by Antony Lee Jan. 6, 2015

Jan. 6, 2015

While trying to reproduce various fancy indexings for astropy's FITS sections (a loaded-on-demand array), I found the following interesting behavior: >>> np.array([1])[..., 0] array(1) >>> np.array([1])[0] 1 >>> np.array([1])[(0,)] 1 The docs say "Ellipsis expand to the number of : objects needed to make a selection tuple of the same length as x.ndim.", so it's not totally clear to me how to explain that difference in the results. Antony

3 3

[ANN] bcolz 0.7.3
by Valentin Haenel Jan. 4, 2015

Jan. 4, 2015

====================== Announcing bcolz 0.7.3 ====================== What's new ========== This release includes the support for pickling persistent carray/ctable objects contributed by Matthew Rocklin. Also, the included version of Blosc is updated to ``v1.5.2``. Lastly, several minor issues and typos have been fixed, please see the release notes for details. ``bcolz`` is a renaming of the ``carray`` project. The new goals for the project are to create simple, yet flexible compressed containers, that can live either on-disk or in-memory, and with some high-performance iterators (like `iter()`, `where()`) for querying them. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… For more detailed info, see the release notes in: https://github.com/Blosc/bcolz/wiki/Release-Notes What it is ========== bcolz provides columnar and compressed data containers. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt ---- **Enjoy data!**

1 0

Regarding np.ma.masked_equal behavior
by Maniteja Nandana Jan. 4, 2015

Jan. 4, 2015

Hello friends, This is an issue related to the working of *masked_equal* method. I was thinking if anyone related to an old ticket #1851 <https://github.com/numpy/numpy/issues/1851>, regarding the modification of *masked_equal *function effect on *fill_value *could clarify the situation, since right now, the documentation and implementation conflict. There is an issue raised regarding this #5408 <https://github.com/numpy/numpy/issues/5408>. Cheers*,* N.Maniteja _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

2 1

numpy.fromiter in numpypy
by Albert-Jan Roskam Jan. 4, 2015

Jan. 4, 2015

Hi, I would like to use the numpy implementation for Pypy. In particular, I would like to use numpy.fromiter, which is available according to this overview: http://buildbot.pypy.org/numpy-status/latest.html. However, contrary to what this website says, this function is not yet available. Conclusion: the website is wrong. Or am I missing something? albertjan@debian:~$ sudo pypy $(which pip) install -U git+https://bitbucket.org/pypy/numpy.git albertjan@debian:~$ sudo pypy -c 'import numpy' # sudo: as per the installation instructions albertjan@debian:~$ pypy Python 2.7.8 (f5dcc2477b97386c11e4b67f08a2d00fbd2fce5d, Sep 19 2014, 10:37:41) [PyPy 2.4.0 with GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> import numpy as np >>>> np.__version__, sys.version ('1.9.0', '2.7.8 (f5dcc2477b97386c11e4b67f08a2d00fbd2fce5d, Sep 19 2014, 10:37:41)\n[PyPy 2.4.0 with GCC 4.8.2]') >>>> np.fromiter <function tmp at 0x00007fdc06e35bc8> >>>> np.fromiter((i for i in range(10)), np.float) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/pypy-2.4/site-packages/numpy/core/multiarray.py", line 55, in tmp raise NotImplementedError("%s not implemented yet" % func) NotImplementedError: fromiter not implemented yet The same also applies to numpy.fromfile Thanks in advance and happy 2015. Regards, Albert-Jan ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2 1

diag, diagonal, ravel and all that
by Charles R Harris Jan. 4, 2015

Jan. 4, 2015

Hi All, The diag, diagonal, and ravel functions have recently been changed to preserve subtypes. However, this causes lots of backward compatibility problems for matrix users, in particular, scipy.sparse. One possibility for fixing this is to special case matrix and so that these functions continue to return 1-d arrays for matrix instances. This is kind of ugly as `a..ravel` will still return a matrix when a is a matrix, an ugly inconsistency. This may be a case where practicality beats beauty. Thoughts? Chuck

4 8

Optimizing multi-tensor contractions in numpy
by Daniel Smith Jan. 3, 2015

Jan. 3, 2015

Hello everyone, I have been working on a chunk of code that basically sets out to provide a single function that can take an arbitrary einsum expression and computes it in the most optimal way. While np.einsum can compute arbitrary expressions, there are two drawbacks to using pure einsum: einsum does not consider building intermediate arrays for possible reductions in overall rank and is not currently capable of using a vendor BLAS. I have been working on a project that aims to solve both issues simultaneously: https://github.com/dgasmith/opt_einsum <https://github.com/dgasmith/opt_einsum> This program first builds the optimal way to contract the tensors together, or using my own nomenclature a “path.” This path is then iterated over and uses tensordot when possible and einsum for everything else. In test cases the worst case scenario adds a 20 microsecond overhead penalty and, in the best case scenario, it can reduce the overall rank of the tensor. The primary (if somewhat exaggerated) example is a 5-term N^8 index transformation that can be reduced to N^5; even when N is very small (N=10) there is a 2,000 fold speed increase over pure einsum or, if using tensordot, a 2,400 fold speed increase. This is somewhat similar to the new np.linalg.multi_dot function. If you are interested in this function please head over to the github repo and check it out. I believe the README is starting to become self-explanatory, but feel free to email me with any questions. This originally started because I was looking into using numpy to rapidly prototype quantum chemistry codes. The results of which can be found here: https://github.com/dgasmith/psi4numpy <https://github.com/dgasmith/psi4numpy> As such, I am very interested in implementing this into numpy. While I think opt_einsum is in a pretty good place, there is still quite a bit to do (see outstanding issues in the README). Even if this is not something that would fit into numpy I would still be very interested in your comments. Thank you for your time, -Daniel Smith

3 3

Correct C string handling in the NumPy C API?
by Sturla Molden Jan. 3, 2015

Jan. 3, 2015

Here is an example: NPY_NO_EXPORT NpyIter_IterNextFunc * NpyIter_GetIterNext(NpyIter *iter, char **errmsg) { npy_uint32 itflags = NIT_ITFLAGS(iter); int ndim = NIT_NDIM(iter); int nop = NIT_NOP(iter); if (NIT_ITERSIZE(iter) < 0) { if (errmsg == NULL) { PyErr_SetString(PyExc_ValueError, "iterator is too large"); } else { *errmsg = "iterator is too large"; } return NULL; } After NpyIter_GetIterNext returns, *errmsg points to a local variable in a returned function. Either I am wrong about C, or this code has undefied behavior... My gutfeeling is that *errmsg = "iterator is too large"; puts the string "iterator is too large" on the stack and points *errmsg to the string. Shouldn't this really be strcpy(*errmsg, "iterator is too large"); and then *errmsg should point to a char buffer allocated before NpyIter_GetIterNext is called? Or will the statement *errmsg = "iterator is too large"; put the string on the stack in the calling C function? Before I open an issue I will ask if my understanding of C is correct or not. I am a bit confused here... Regards, Sturla

2 3

Pass 2d ndarray into C **double using ctypes
by Yuxiang Wang Jan. 3, 2015

Jan. 3, 2015

Dear all, I am currently using a piece of C code, where one of the input argument of a function is **double. So, in numpy, I tried np.ctypeslib.ndpointer(ctypes.c_double), but obviously this wouldn't work because this is only *double, not **double. Then I tried np.ctypeslib.ndpointer(np.ctypeslib.ndpointer(ctypes.c_double)), but this didn't work either because it says "ArgumentError: argument 4: <class 'TypeError'>: array must have data type uint64 ". np.ctypeslib.ndpointer(ctypes.c_double, ndim=2) wound't work too, because **double is not the same with *double[]. Could anyone please give any thoughts to help? Thanks, Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj(a)virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/

3 11

Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
by Stephan Hoyer Jan. 3, 2015

Jan. 3, 2015

I recently wrote function to manually broadcast an ndarray to a given shape according to numpy's broadcasting rules (using strides): https://github.com/xray/xray/commit/7aee4a3ed2dfd3b9aff7f3c5c6c68d51df2e3ff3 The same functionality can be done pretty straightforwardly with np.broadcast_arrays, but that function does both too much (I don't actually have a second array that needs to be broadcast) and not enough (I need to create a dummy array to broadcast against it). This approach is simpler, and also, according to my benchmarks, about 3x faster than np.broadcast_arrays: In [1]: import xray In [2]: import numpy as np In [3]: x = np.random.randn(4) In [4]: y = np.empty((2, 3, 4)) In [5]: %timeit xray.core.utils.as_shape(x, y.shape) 100000 loops, best of 3: 17 µs per loop In [6]: %timeit np.broadcast_arrays(x, y)[0] 10000 loops, best of 3: 47.4 µs per loop Would this be a welcome addition to numpy's lib.stride_tricks? If so, I will put together a PR. In my search, I turned up a Stack Overflow post looking for similar functionality: https://stackoverflow.com/questions/11622692/is-there-a-better-way-to-broad… Cheers, Stephan

8 24

Extracting required indices from the array of tuples
by Simen Langseth Jan. 2, 2015

Jan. 2, 2015

import numpy as np from scipy import signal y = np.array([[2, 1, 2, 3, 2, 0, 1, 0], [2, 1, 2, 3, 2, 0, 1, 0]]) maximas = signal.argrelmax(y, axis=1) print maximas (array([0, 0, 1, 1], dtype=int64), array([3, 6, 3, 6], dtype=int64)) I want to extract only the first maxima of both rows, i.e., [3, 3] using the tuples (maximas). How would you do it?

2 2

npymath on Windows
by Matthew Brett Jan. 2, 2015

Jan. 2, 2015

Hi, Sorry for this ignorant email, but we got confused trying to use 'libnpymath.a' from the mingw builds of numpy: We were trying to link against the mingw numpy 'libnpymath.a' using Visual Studio C, but this give undefined symbols from 'libnpymath.a' like this: npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol _atanf referenced in function _npy_atanf npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol _acosf referenced in function _npy_acosf npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol _asinf referenced in function _npy_asinf (see : http://nipy.bic.berkeley.edu/builders/dipy-bdist32-33/builds/73/steps/shell…) npymath.lib from Christophe Gohlke's (MSVC compiled) numpies does not give such an error. Sure enough, 'npymath.lib' shows these lines from `dumpbin /all npymath.lib`: 00000281 REL32 00000000 4F asinf 00000291 REL32 00000000 51 acosf 000002A1 REL32 00000000 53 atanf whereas `dumpbin /all libnpymath.a` shows these kinds of lines: 000008E5 REL32 00000000 86 _asinf 000008F5 REL32 00000000 85 _acosf 00000905 REL32 00000000 84 _atanf As far as I can see, 'acosf' is defined in the msvc runtime library. I guess that '_acosf' is defined in some mingw runtime library? Is there any way of making a npymath library that will pick up the msvc math and so may work with both msvc and mingw? Sorry again if that's a dumb question, Matthew

4 7

numpy developers
by Charles R Harris Jan. 2, 2015

Jan. 2, 2015

Hi All, I've invited Alex Griffing onto the team to be a numpy developer. He has been contributing fixes and reviews for a while and it is time to give him more opportunity to contribute. I think he will do well. Chuck

2 1

Re: [Numpy-discussion] numpy.fromiter in numpypy
by Matti Picus Jan. 2, 2015

Jan. 2, 2015

1 0

Access dtype kind from cython
by Valentin Haenel Jan. 1, 2015

Jan. 1, 2015

Hi, how do I access the kind of the data from cython, i.e. the single character string: 'b' boolean 'i' (signed) integer 'u' unsigned integer 'f' floating-point 'c' complex-floating point 'O' (Python) objects 'S', 'a' (byte-)string 'U' Unicode 'V' raw data (void) In regular Python I can do: In [7]: d = np.dtype('S') In [8]: d.kind Out[8]: 'S' Looking at the definition of dtype that comes with cython, I see: ctypedef class numpy.dtype [object PyArray_Descr]: # Use PyDataType_* macros when possible, however there are no macros # for accessing some of the fields, so some are defined. Please # ask on cython-dev if you need more. cdef int type_num cdef int itemsize "elsize" cdef char byteorder cdef object fields cdef tuple names I.e. no kind. Also, i looked for an appropriate PyDataType_* macro but couldn't find one. Perhaps there is something simple I could use? best, V-

3 5