Mailman 3 March 2013 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer March 31, 2016

March 31, 2016

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

SVD errors
by mtrumpis＠berkeley.edu June 4, 2013

June 4, 2013

Hello list.. I've run into two SVD errors over the last few days. Both errors are identical in numpy/scipy. I've submitted a ticket for the 1st problem (numpy ticket #990). Summary is: some builds of the lapack_lite module linking against system LAPACK (not the bundled dlapack_lite.o, etc) give a "LinAlgError: SVD did not converge" exception on my matrix. This error does occur using Mac's Accelerate framework LAPACK, and a coworker's Ubuntu LAPACK version. It does not seem to happen using ATLAS LAPACK (nor using Octave/Matlab on said Ubuntu) Just today I've come across a negative singular value cropping up in an SVD of a different matrix. This error does occur on my ATLAS LAPACK based numpy, as well as on the Ubuntu setup. And once again, it does not happen in Octave/Matlab. I'm using numpy 1.3.0.dev6336 -- don't know what the Ubuntu box is running. Here are some npy files for the two different cases: https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/noconverge_operator.npy https://cirl.berkeley.edu/twiki/pub/User/MikeTrumpis/negsval_operator.npy Mike

6 7

pyhdf packaging
by Andreas Hilboll May 23, 2013

May 23, 2013

Hi, I would like to package pyhdf for Ubuntu and make the package publicly available. Since the license is not totally clear to me (I cannot find any information in the sources, and the cheeseshop says "public", which doesn't mean anything to me), I tried to contact the maintainer, Andre Gosselin, however the email bounces, so I guess he's gone. Can anyone point me to how to proceed from here? Cheers, Andreas.

2 2

Any plans for windows 64-bit installer for 1.7?
by Matthew Brett May 6, 2013

May 6, 2013

Hi, I see there is no Windows 64 bit installer for the 1.7 rc1. Is there any prospect of a 64 bit installer for the full release? Can I help? I have a Windows 7 64 bit machine that I use as a build slave; I am happy to give access. Cheers, Matthew

13 57

Scheduling the 1.7.1 and 1.8 releases
by Charles R Harris April 11, 2013

April 11, 2013

Hi All, There are now some 14 non-merge commits in the 1.7.x branch including the critical diagonal leak fix. I think there is maybe one more critical backport and perhaps several low priority fixes, documentation and such, but I think we should start up the release process with a goal of getting 1.7.1 out by the middle of April. The development branch has been accumulating stuff since last summer, I suggest we look to get it out in May, branching at the end of this month. Thoughts? Chuck

5 9

OpenOpt Suite release 0.45
by Dmitrey April 10, 2013

April 10, 2013

Hi all, > I'm glad to inform you about new OpenOpt Suite release 0.45 (2013-March-15): * Essential improvements for FuncDesigner interval analysis (thus affect interalg) * Temporary walkaround for a serious bug in FuncDesigner automatic differentiation kernel due to a bug in some versions of Python or NumPy, may affect optimization problems, including (MI)LP, (MI)NLP, TSP etc * Some other minor bugfixes and improvements > > --------------------------- > Regards, D. > http://openopt.org/Dmitrey

6 27

Growing the contributor base of Numpy
by Jonathan Rocher April 7, 2013

April 7, 2013

Dear all, One recurring question is how to *grow the contributor base* to NumPy and provide help and relief to core developers and maintainers. One way to do this would be to *leverage the upcoming SciPy conference* in 2 ways: 1. Provide an intermediate or advanced level tutorial on NumPy focusing on teaching the C-API and the architecture of the package to help people navigate the source code, and find answers to precise deep questions. I think that many users would be interested in being better able to understand the underlayers to become powerful users (and contributors if they want to). 2. Organize a Numpy sprint to leverage all this freshly graduated students apply what they learned to tackle some of the work under the guidance of core developers. This would be a great occasion to share and grow knowledge that is fundamental to our community. And the fact that the underlayers are in C is fine IMHO: SciPy is about scientific programming in Python and that is done with a lot of C. *Thoughts? Anyone interested in leading a tutorial (can be a team of people)? Anyone willing to coordinate the sprint? Who would be willing to be present and help during the sprint? * Note that there is less than 1 week left until the tutorial submission deadline. I am happy to help brainstorm on this to make it happen. Thanks, Jonathan and Andy, for the SciPy2013 organizers -- Jonathan Rocher, PhD Scientific software developer SciPy2013 conference co-chair Enthought, Inc. jrocher(a)enthought.com 1-512-536-1057 http://www.enthought.com

4 7

ANN: NumPy 1.7.1rc1 release
by Ondřej Čertík April 7, 2013

April 7, 2013

Hi, I'm pleased to announce the availability of the first release candidate of NumPy 1.7.1rc1. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.1rc1/ Please test it and report any bugs. It fixes a few bugs, listed below. I would like to thank everybody who contributed patches to this release: Nathaniel J. Smith, Sebastian Berg, Charles Harris, Bradley M. Froehle, Ralf Gommers, Christoph Gohlke, Mark Wiebe and Maximilian Albert. Cheers, Ondrej ========================= NumPy 1.7.1 Release Notes ========================= This is a bugfix only release in the 1.7.x series. Issues fixed ------------ gh-2973 Fix `1` is printed during numpy.test() gh-2983 BUG: gh-2969: Backport memory leak fix 80b3a34. gh-3007 Backport gh-3006 gh-2984 Backport fix complex polynomial fit gh-2982 BUG: Make nansum work with booleans. gh-2985 Backport large sort fixes gh-3039 Backport object take gh-3105 Backport nditer fix op axes initialization gh-3108 BUG: npy-pkg-config ini files were missing after Bento build. gh-3124 BUG: PyArray_LexSort allocates too much temporary memory. gh-3131 BUG: Exported f2py_size symbol prevents linking multiple f2py modules. gh-3117 Backport gh-2992 gh-3135 DOC: Add mention of PyArray_SetBaseObject stealing a reference gh-3134 DOC: Fix typo in fft docs (the indexing variable is 'm', not 'n'). gh-3136 Backport #3128 Checksums ========= 28c3f3e71b5eaa6bfab6e8340dbd35e7 release/installers/numpy-1.7.1rc1.tar.gz 436f416dee10d157314bd9da7ab95c9c release/installers/numpy-1.7.1rc1-win32-superpack-python2.7.exe a543c8cf69f66ff2b4c9565646105863 release/installers/numpy-1.7.1rc1-win32-superpack-python2.6.exe 6dfcbbd449b7fe4e841c5fd1bfa7af7c release/installers/numpy-1.7.1rc1-win32-superpack-python2.5.exe 22912792a1b6155ae2bdbc30bee8fadc release/installers/numpy-1.7.1rc1-win32-superpack-python3.2.exe 95bc5a5fcce9fcbc2717a774dccae31b release/installers/numpy-1.7.1rc1-win32-superpack-python3.3.exe 33cf283765a148846b49b89fb96d67d5 release/installers/numpy-1.7.1rc1-win32-superpack-python3.1.exe 9761de4b35493fed38c5d177da9c3b37 release/installers/numpy-1.7.1rc1.zip

4 5

Bug in np.records?
by Pierre Barbier de Reuille April 6, 2013

April 6, 2013

Hey, I am trying to use titles for the record arrays. In the documentation, it is specified that any column can set to "None". However, trying this fails on numpy 1.6.2 because in np.core.records, on line 195, the "strip" method is called on the title object. This is really annoying. Could we fix this by replacing line 195 with: self._titles = [n.strip() if n is not None else None for n in titles[:self._nfields]] ? Thank you, -- Barbier de Reuille Pierre

2 1

Raveling, reshape order keyword unnecessarily confuses index and memory ordering
by Matthew Brett April 5, 2013

April 5, 2013

Hi, We were teaching today, and found ourselves getting very confused about ravel and shape in numpy. Summary -------------- There are two separate ideas needed to understand ordering in ravel and reshape: Idea 1): ravel / reshape can proceed from the last axis to the first, or the first to the last. This is "ravel index ordering" Idea 2) The physical layout of the array (on disk or in memory) can be "C" or "F" contiguous or neither. This is "memory ordering" The index ordering is usually (but see below) orthogonal to the memory ordering. The 'ravel' and 'reshape' commands use "C" and "F" in the sense of index ordering, and this mixes the two ideas and is confusing. What the current situation looks like ---------------------------------------------------- Specifically, we've been rolling this around 4 experienced numpy users and we all predicted at least one of the results below wrongly. This was what we knew, or should have known: In [2]: import numpy as np In [3]: arr = np.arange(10).reshape((2, 5)) In [5]: arr.ravel() Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) So, the 'ravel' operation unravels over the last axis (1) first, followed by axis 0. So far so good (even if the opposite to MATLAB, Octave). Then we found the 'order' flag to ravel: In [10]: arr.flags Out[10]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [11]: arr.ravel('C') Out[11]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) But we soon got confused. How about this? In [12]: arr_F = np.array(arr, order='F') In [13]: arr_F.flags Out[13]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [16]: arr_F Out[16]: array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) In [17]: arr_F.ravel('C') Out[17]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) Right - so the flag 'C' to ravel, has got nothing to do with *memory* ordering, but is to do with *index* ordering. And in fact, we can ask for memory ordering specifically: In [22]: arr.ravel('K') Out[22]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [23]: arr_F.ravel('K') Out[23]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9]) In [24]: arr.ravel('A') Out[24]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [25]: arr_F.ravel('A') Out[25]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9]) There are some confusions to get into with the 'order' flag to reshape as well, of the same type. Ravel and reshape use the tems 'C' and 'F" in the sense of index ordering. This is very confusing. We think the index ordering and memory ordering ideas need to be separated, and specifically, we should avoid using "C" and "F" to refer to index ordering. Proposal ------------- * Deprecate the use of "C" and "F" meaning backwards and forwards index ordering for ravel, reshape * Prefer "Z" and "N", being graphical representations of unraveling in 2 dimensions, axis1 first and axis0 first respectively (excellent naming idea by Paul Ivanov) What do y'all think? Cheers, Matthew Paul Ivanov JB Poline

9 79

Re: [Numpy-discussion] Indexing bug
by Ivan Oseledets April 5, 2013

April 5, 2013

Message: 2 Date: Sat, 30 Mar 2013 11:13:35 -0700 From: Jaime Fern?ndez del R?o <jaime.frio(a)gmail.com> Subject: Re: [Numpy-discussion] Indexing bug? To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <CAPOWHWk+mL6KN6F2FHTPn5HTiU0UEQPj6KdxjNK_+T1E-YRiBg(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" On Sat, Mar 30, 2013 at 11:01 AM, Ivan Oseledets <ivan.oseledets(a)gmail.com>wrote: > I am using numpy 1.6.1, > and encountered a wierd fancy indexing bug: > > import numpy as np > c = np.random.randn(10,200,10); > > In [29]: print c[[0,1],:200,:2].shape > (2, 200, 2) > > In [30]: print c[[0,1],:200,[0,1]].shape > (2, 200) > > It means, that here fancy indexing is not working right for a 3d array. > On Sat, Mar 30, 2013 at 11:01 AM, Ivan Oseledets <ivan.oseledets(a)gmail.com>wrote: > I am using numpy 1.6.1, > and encountered a wierd fancy indexing bug: > > import numpy as np > c = np.random.randn(10,200,10); > > In [29]: print c[[0,1],:200,:2].shape > (2, 200, 2) > > In [30]: print c[[0,1],:200,[0,1]].shape > (2, 200) > > It means, that here fancy indexing is not working right for a 3d array. > --> It is working fine, review the docs: http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-ind… In your return, item [0, :] is c[0, :, 0] and item[1, :]is c[1, :, 1]. If you want a return of shape (2, 200, 2) where item [i, :, j] is c[i, :, j] you could use slicing: c[:2, :200, :2] or something more elaborate like: c[np.arange(2)[:, None, None], np.arange(200)[:, None], np.arange(2)] Jaime ---> Oh! So it is not a bug, it is a feature, which is completely incompatible with other array based languages (MATLAB and Fortran). To me, I can not find a single explanation why it is so in numpy. Taking submatrices from a matrix is a common operation and the syntax above is very natural to take submatrices, not a weird diagonal stuff. i.e., c = np.random.randn(100,100) d = c[[0,3],[2,3]] should NOT produce two numbers! (and you can not do it using slices!) In MATLAB and Fortran c(indi,indj) will produce a 2 x 2 matrix. How it can be done in numpy (and why the complications?) So, please consider this message as a feature request. Ivan

4 3

NumPy/SciPy participation in GSoC 2013
by Ralf Gommers April 2, 2013

April 2, 2013

Hi all, It is the time of the year for Google Summer of Code applications. If we want to participate with Numpy and/or Scipy, we need two things: enough mentors and ideas for projects. If we get those, we'll apply under the PSF umbrella. They've outlined the timeline they're working by and guidelines at http://pyfound.blogspot.nl/2013/03/get-ready-for-google-summer-of-code.html. We should be able to come up with some interesting project ideas I'd think, let's put those at http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas. Preferably with enough detail to be understandable for people new to the projects and a proposed mentor. We need at least 3 people willing to mentor a student. Ideally we'd have enough mentors this week, so we can apply to the PSF on time. If you're willing to be a mentor, please send me the following: name, email address, phone nr, and what you're interested in mentoring. If you have time constaints and have doubts about being able to be a primary mentor, being a backup mentor would also be helpful. Cheers, Ralf P.S. as you can probably tell from the above, I'm happy to coordinate the GSoC applications for Numpy and Scipy

4 7

Indexing bug?
by Ivan Oseledets March 30, 2013

March 30, 2013

I am using numpy 1.6.1, and encountered a wierd fancy indexing bug: import numpy as np c = np.random.randn(10,200,10); In [29]: print c[[0,1],:200,:2].shape (2, 200, 2) In [30]: print c[[0,1],:200,[0,1]].shape (2, 200) It means, that here fancy indexing is not working right for a 3d array. Is this bug fixed with higher versions of numpy? I do not check, since mine is from EPD and is compiled with MKL (and I can consider recompiling myself only under strong circumstances) Ivan

2 1

Polar/spherical coordinates handling
by Todd March 29, 2013

March 29, 2013

>From what I can see, numpy doesn't have any functions for handling polar or spherical coordinate to/from cartesian coordinate conversion. I think such methods would be pretty useful. I am looking now and it doesn't look that hard to create functions to convert between n-dimensional cartesian and n-spherical coordinates. Would anyone be interested in me adding methods for this?

2 2

inheriting from recarray with nested dtypes
by Geoffrey Irving March 28, 2013

March 28, 2013

I have the following two structured dtypes: rotation (quaternion) = dtype([('s','f8'),('v','3f8')]) frame = dtype([('t','3f8'),('r',rotation)]) For various reasons, I usually store rotation arrays in a class Rotations deriving from ndarray, and frames in a class Frames deriving from ndarray. Currently I am defining .s, .v, .t, and .r properties manually, and I'd like to switch to inheriting from recarray. However, the .r property should return an array of class Rotations. I.e., f = Frames(...) # f.dtype = frame r = f.r # r.dtype = rotation, type(r) = Rotations Is there a clean way to tell recarray to adjust the type returned? It already has a bit of intelligence there, since it returns ndarray vs. recarray based on whether the returned dtype has fields. The full code is here in case anyone is curious: https://github.com/otherlab/core/blob/master/vector/Frame.py https://github.com/otherlab/core/blob/master/vector/Rotation.py Thanks, Geoffrey

1 0

Implementing a "find first" style function
by Phil Elson March 28, 2013

March 28, 2013

The ticket https://github.com/numpy/numpy/issues/2269 discusses the possibility of implementing a "find first" style function which can optimise the process of finding the first value(s) which match a predicate in a given 1D array. For example: >>> a = np.sin(np.linspace(0, np.pi, 200)) >>> print find_first(a, lambda a: a > 0.9) ((71, ), 0.900479032457) This has been discussed in several locations: https://github.com/numpy/numpy/issues/2269 https://github.com/numpy/numpy/issues/2333 http://stackoverflow.com/questions/7632963/numpy-array-how-to-find-index-of… *Rationale* For small arrays there is no real reason to avoid doing: >>> a = np.sin(np.linspace(0, np.pi, 200)) >>> ind = (a > 0.9).nonzero()[0][0] >>> print (ind, ), a[ind] (71,) 0.900479032457 But for larger arrays, this can lead to massive amounts of work even if the result is one of the first to be computed. Example: >>> a = np.arange(1e8) >>> print (a == 5).nonzero()[0][0] 5 So a function which terminates when the first matching value is found is desirable. As mentioned in #2269, it is possible to define a consistent ordering which allows this functionality for >1D arrays, but IMHO it overcomplicates the problem and was not a case that I personally needed, so I've limited the scope to 1D arrays only. *Implementation* My initial assumption was that to get any kind of performance I would need to write the *find* function in C, however after prototyping with some array chunking it became apparent that a trivial python function would be quick enough for my needs. The approach I've implemented in the code found in #2269 simply breaks the array into sub-arrays of maximum length *chunk_size* (2048 by default, though there is no real science to this number), applies the given predicating function, and yields the results from *nonzero()*. The given function should be a python function which operates on the whole of the sub-array element-wise (i.e. the function should be vectorized). Returning a generator also has the benefit of allowing users to get the first *n*matching values/indices. *Results* I timed the implementation of *find* found in my comment at https://github.com/numpy/numpy/issues/2269#issuecomment-14436725 with an obvious test: In [1]: from np_utils import find In [2]: import numpy as np In [3]: import numpy.random In [4]: np.random.seed(1) In [5]: a = np.random.randn(1e8) In [6]: a.min(), a.max() Out[6]: (-6.1194900990552776, 5.9632246301166321) In [7]: next(find(a, lambda a: np.abs(a) > 6)) Out[7]: ((33105441,), -6.1194900990552776) In [8]: (np.abs(a) > 6).nonzero() Out[8]: (array([33105441]),) In [9]: %timeit (np.abs(a) > 6).nonzero() 1 loops, best of 3: 1.51 s per loop In [10]: %timeit next(find(a, lambda a: np.abs(a) > 6)) 1 loops, best of 3: 912 ms per loop In [11]: %timeit next(find(a, lambda a: np.abs(a) > 6, chunk_size=100000)) 1 loops, best of 3: 470 ms per loop In [12]: %timeit next(find(a, lambda a: np.abs(a) > 6, chunk_size=1000000)) 1 loops, best of 3: 483 ms per loop This shows that picking a sensible *chunk_size* can yield massive speed-ups (nonzero is x3 slower in one case). A similar example with a much smaller 1D array shows similar promise: In [41]: a = np.random.randn(1e4) In [42]: %timeit next(find(a, lambda a: np.abs(a) > 3)) 10000 loops, best of 3: 35.8 us per loop In [43]: %timeit (np.abs(a) > 3).nonzero() 10000 loops, best of 3: 148 us per loop As I commented on the issue tracker, if you think this function is worth taking forward, I'd be happy to open up a pull request. Feedback greatfully received. Cheers, Phil

3 5

variables not defined in numpy.random __init.py__ ?
by Dinesh B Vadhia March 27, 2013

March 27, 2013

Using PyInstaller, the following error occurs: Traceback (most recent call last): File "<string>", line 9, in <module> File "//usr/lib/python2.7/dist-packages/PIL/Image.py", line 355, in init __import__(f, globals(), locals(), []) File "//usr/lib/python2.7/dist-packages/PIL/IptcImagePlugin.py", line 23, in <module> import os, tempfile File "/usr/lib/python2.7/tempfile.py", line 34, in <module> from random import Random as _Random File "//usr/lib/python2.7/dist-packages/numpy/random/__init__.py", line 90, in <module> ranf = random = sample = random_sample NameError: name 'random_sample' is not defined Is line 90 in __init.py__ valid?

3 4

Re: [Numpy-discussion] Growing the contributor base of Numpy
by Andrea Cimatoribus March 27, 2013

March 27, 2013

Oh, I didn't even know it existed! > > Not sure if this is really relevant to the original message, but here is my opinion. I think that the numpy/scipy community would greatly benefit from a platform enabling easy sharing of code written by users. This would provide a database of solved problems, where people could dig without having to ask. I think that something like this exists for matlab, but I have no experience with it. If it exists for python, then it must be seriously under-advertised. The web provides many answers, but they are scattered in all sorts of places, and it is often impossible to contribute improvements to code found online. If such a database could enable some sort of collaborative development it would be a great added value for numpy, and would provide a natural source of new features or improvements for scipy and numpy. Supposedly that's what scipy-central is for, but it's somehow not yet reached critical mass and become a household name; I haven't looked hard enough to have any hypotheses about why not. Surya Kasturi is working on spiffing it up (see discussion on scipy-dev); I bet they could use some help if you want to scratch this itch.

1 0

Re: [Numpy-discussion] Growing the contributor base of Numpy
by Andrea Cimatoribus March 27, 2013

March 27, 2013

Not sure if this is really relevant to the original message, but here is my opinion. I think that the numpy/scipy community would greatly benefit from a platform enabling easy sharing of code written by users. This would provide a database of solved problems, where people could dig without having to ask. I think that something like this exists for matlab, but I have no experience with it. If it exists for python, then it must be seriously under-advertised. The web provides many answers, but they are scattered in all sorts of places, and it is often impossible to contribute improvements to code found online. If such a database could enable some sort of collaborative development it would be a great added value for numpy, and would provide a natural source of new features or improvements for scipy and numpy.

2 1

ANN: matplotlib 1.2.1 release
by Michael Droettboom March 27, 2013

March 27, 2013

I'm pleased to announce the release of matplotlib 1.2.1. This is a bug release and improves stability and quality over the 1.2.0 release from four months ago. All users on 1.2.0 are encouraged to upgrade. Since github no longer provides download hosting, our tarballs and binaries are back on SourceForge, and we have a master index of downloads here: http://matplotlib.org/downloads <http://matplotlib.org/downloads.html> Highlights include: - Usage of deprecated APIs in matplotlib are now displayed by default on all Python versions - Agg backend: Cleaner rendering of rectilinear lines when snapping to pixel boundaries, and fixes rendering bugs when using clip paths - Python 3: Fixes a number of missed Python 3 compatibility problems - Histograms and stacked histograms have a number of important bugfixes - Compatibility with more 3rd-party TrueType fonts - SVG backend: Image support in SVG output is consistent with other backends - Qt backend: Fixes leaking of window objects in Qt backend - hexbin with a log scale now works correctly - autoscaling works better on 3D plots - ...and numerous others. Enjoy! As always, there are number of good ways to get help with matplotlib listed on the homepage at http://matplotlib.org/ and I thank everyone for their continued support of this project. Mike Droettboom

1 0

Re: [Numpy-discussion] howto reduce along arbitrary axis
by Chao YUE March 26, 2013

March 26, 2013

Oh sorry, my fault... here is the answer by Nathaniel Smith: def retrieve_data(a, ax, idx): full_idx = [slice(None)] * a.ndim full_idx[ax] = idx return a[tuple(full_idx)] Or for the specific case where you do know the axis in advance, you just don't know how many trailing axes there are, use a[:, :, 0, ...] and the ... will expand to represent the appropriate number of :'s. probably you can sue something simlaer. Chao On Tue, Mar 26, 2013 at 3:33 PM, Neal Becker <ndbecker2(a)gmail.com> wrote: > Thank you, but do you also have an answer to this question? I only see > the question. > > > On Tue, Mar 26, 2013 at 10:23 AM, Chao YUE <chaoyuejoy(a)gmail.com> wrote: > >> Hi Neal, >> >> I forward you this mail which I think might be of help to your question. >> >> Chao >> >> ---------- Forwarded message ---------- >> From: Chao YUE <chaoyuejoy(a)gmail.com> >> Date: Sat, Mar 16, 2013 at 5:40 PM >> Subject: indexing of arbitrary axis and arbitrary slice? >> To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> >> >> >> Dear all, >> >> Is there some way to index the numpy array by specifying arbitrary axis >> and arbitrary slice, while >> not knowing the actual shape of the data? >> For example, I have a 3-dim data, data.shape = (3,4,5) >> Is there a way to retrieve data[:,0,:] by using something like >> np.retrieve_data(data,axis=2,slice=0), >> by this way you don't have to know the actual shape of the array. >> for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually >> be data[:,0,:,:] >> >> thanks in advance, >> >> Chao >> >> -- >> >> *********************************************************************************** >> Chao YUE >> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) >> UMR 1572 CEA-CNRS-UVSQ >> Batiment 712 - Pe 119 >> 91191 GIF Sur YVETTE Cedex >> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 >> >> ************************************************************************************ >> >> >> >> -- >> >> *********************************************************************************** >> Chao YUE >> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) >> UMR 1572 CEA-CNRS-UVSQ >> Batiment 712 - Pe 119 >> 91191 GIF Sur YVETTE Cedex >> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 >> >> ************************************************************************************ >> > > -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 0

howto reduce along arbitrary axis
by Chao YUE March 26, 2013

March 26, 2013

Hi Neal, I forward you this mail which I think might be of help to your question. Chao ---------- Forwarded message ---------- From: Chao YUE <chaoyuejoy(a)gmail.com> Date: Sat, Mar 16, 2013 at 5:40 PM Subject: indexing of arbitrary axis and arbitrary slice? To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 0