Mailman 3 March 2014 - NumPy-Discussion

ANN: NumPy 1.8.1 release
by Julian Taylor 05 Apr '14

05 Apr '14

Hello, I'm happy to announce the of Numpy 1.8.1. This is a bugfix only release supporting Python 2.6 - 2.7 and 3.2 - 3.4. More than 48 issues have been fixed, the most important issues are listed in the release notes: https://github.com/numpy/numpy/blob/maintenance/1.8.x/doc/release/1.8.1-not… Compared to the last release candidate we have fixed a regression of the 1.8 series that prevented using some gufunc based linalg functions on larger matrices on 32 bit systems. This implied a few changes in the NDIter C-API which might expose insufficient checks for error conditions in third party applications. Please check the release notes for details. Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.1 Cheers, Julian Taylor

11 36

Transparently reading complex arrays from netcdf4
by G Jones 30 Mar '14

30 Mar '14

Hi, I am using netCDF4 to store complex data using the recommended strategy of creating a compound data type with the real and imaginary parts. This all works well, but reading the data into a numpy array is a bit clumsy. Typically I do: nc = netCDF4.Dataset('my.nc') cplx_data = nc.groups['mygroup'].variables['cplx_stuff'][:].view('complex') which directly gives a nice complex numpy array. This is OK for small arrays, but is wasteful if I only need some chunks of the array because it reads all the data in, reducing the utility of the mmap feature of netCDF. I'm wondering if there is a better way to directly make a numpy array view that uses the netcdf variable's memory mapped buffer directly. Looking at the Variable class, there is no access to this buffer directly which could then be passed to np.ndarray(buffer=...). Any ideas of simple solutions to this problem? Thanks, Glenn

2 5

Changes to np.vander
by Jaime Fernández del Río 29 Mar '14

29 Mar '14

Hi, I have submitted a PR (https://github.com/numpy/numpy/pull/4568) that speeds up `np.vander` by using accumulated multiplication instead of exponentiation to compute the Vandermonde matrix. For largish matrices the speed-ups can be quite dramatic, over an order of magnitude. Julian has raised concerns on numerical stability and loss of precision, which don't seem to be all that relevant. Do speak up if you think otherwise. We are also discussing replacing a recently added kwarg, "order", which now accepts a string, either "increasing" or "decreasing", to indicate the ordering of the matrix columns. This was not present in 1.8, so can still be modified. The proposal is to replace it with a "reversed" boolean flag. Unfortunately, the return of np.vander in 1.8 and before is the opposite (i.e. its reversed) from the standard definition, which has powers increasing from left to right. So it is not clear what the reversed keyword should refer to: 1. If it refers to the standard definition, then it would default to False for backwards compatibility, but be consistent with the conventional definition. 2. If it refers to the existing behavior of numpy's vander, then it would default to True, and not be consistent with the conventional definition. I prefer option 1, but would like to hear other's opinions. Which could of course include naming the boolean flag more ingeniously, or keeping the string flag. If he's reading, I'd specially like to hear Warren Weckesser's thoughts, as he is the one who added the "order" kwarg. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 3

Windows wheels using MKL?
by Matthew Brett 27 Mar '14

27 Mar '14

Hi, Can I check what is stopping us building official numpy binary wheels for Windows using the Intel Math Kernel Library? * We'd need developer licenses, but those sound like they would be easy to come by * We'd have to add something to the license for the wheel on the lines of the Canopy license [1], derived from the MKL license [2] - is that a problem? Are there other problems for numpy? * I believe we would also need the Intel Fortran compiler when building 64-bit scipy with MSVC. Is that correct? If we have a license, is that a problem? If we did static linking to MKL for numpy and scipy, is there anything stopping us building wheels that would work for XP and above, for 32 and 64 bit? Maybe this is not the ideal solution, but perhaps it's the right thing to do for now? Cheers, Matthew [1] https://www.enthought.com/products/canopy/canopy-license/ [2] http://software.intel.com/en-us/articles/intel-software-development-product…

2 6

Re: [Numpy-discussion] Is there a pure numpy recipe for this?
by RayS 27 Mar '14

27 Mar '14

Thanks for all of the suggestions; we are migrating to 64bit Python soon as well. The environments are Win7 and Mac Maverics. carray sounds like what you said Chris - more I just found at http://kmike.ru/python-data-structures/ - Ray Schumacher At 12:31 PM 3/27/2014, you wrote: >On Thu, Mar 27, 2014 at 7:42 AM, RayS ><<mailto:rays@blue-cove.com>rays(a)blue-cove.com> wrote: >I find this interesting, since I work with medical data sets of 100s >of MB, and regularly run into memory allocation problems when doing a >lot of Fourrier analysis, waterfalls etc. The per-process limit seems >to be about 1.3GB on this 6GB quad-i7 with Win7. > > >This sounds like 32 bit -- have you tried a 64 >bit Python_numpy? Nt that you wont have issues >anyway, but you should be abel to do better than 1.3GB... >Â >Â memmaps are also limited to RAM, > > >I don't think so, no -- but are limited to 2GB >(I think) Â if you're using a 32 bit process > >There is also a compressed array package out >there -- I can't remember what it's called -- >but if you have largeÂ compressibleÂ arrays -- that might help. >Â >-CHB > > >-- > >Christopher Barker, Ph.D. >Oceanographer > >Emergency Response Division >NOAA/NOS/OR&R Â Â Â Â Â Â (206) 526-6959Â Â voice >7600 Sand Point Way NE Â Â (206) 526-6329Â Â fax >Seattle, WA Â 98115 Â Â Â Â (206) 526-6317Â Â main reception > ><mailto:Chris.Barker@noaa.gov>Chris.Barker(a)noaa.gov >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion(a)scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion

2 1

Is there a pure numpy recipe for this?
by Slaunger 27 Mar '14

27 Mar '14

I am working on solving a recent recreational mathematical problem on Project Euler <http://projecteuler.net> . I have a solution, which works fine for small N up to 10^5 but it takes too long to compute for the actual problem, where N is of the order 2*10^7. The problem is nested loops, and I am hoping to avoid one level in the loop by using clever numpy magic (as I have done often before). However, there is one step, which I cannot figure out how to do using numpy operations alone, and I am hoping for some help The subproblem is that I have in principle k = 1, ..., N sets of boolean arrays f_k and g_k each of length N. For each k I need to find the number of elements i where both f_k[i] and g_k[i] are True and sum that up over all N values of k. A problem of the order 4*10^14 if I just do it brute force. This takes way too long (there is a one minute rule). However, after a lot of thinking and by using some properties of the f_k and g_k I have managed to construct using only pure numpy function and only a single loop over k, arrays f_k_changes_at g_k_changes_at which contain the indices i at which the functions change it boolean value from True to False or False to True. It so happens that the number of changes is only a small fraction of N, the fraction decreases with larger N, so the size of these changes_at arrays contains perhaps only 1000 elements instead of 10000000 for each k, a significant reduction of complexity. Now, my problem is to figure out for how many values of i both f_k and g_k are True given the changes_at arrays. As this may be a little hard to understand here is a specific example of how these arrays can look like for k = 2 and N = 150 f_2_changes_at = [ 2 3 39 41 58 59 65 66 93 102 145] g_2_changes_at = [ 2 94 101 146 149] with the boundary condition that f_2[0] = g_2[0] = False Which expands to i f_2 g_2 f_2 and g_2 0 F F F 1 F F F <- 2 T T T <- 3 F T F 4 F T F ... 38 F T F <- 39 T T T 40 T T T <- 41 F T F 42 F T F ... 57 F T F <- 58 T T T <- 59 F T F 60 F T F ... 64 F T F <- 65 T T T <- 66 F T F ... 92 F T F <- 93 T T T <- 94 T F F ... 100 T F F <- 101 T T T <- 102 F T F ... 144 F T F <- 145 T T T <- 146 T F F 147 T F F 148 T F F <- 149 T T T <- With the sum of elements fulfilling the condition being (see arrows) (2 - 1) + (40 - 38) + (58 - 57) + (65 - 64) + (93 - 92) + (101 - 100) + (145 - 144) + (149 - 148) = 1 + 2 + 1 + 1 + 1 + 1 + 1 + 1 = 9 So, is there a numpy recipe for doing the equivalent process without expanding it into the full arrays? I have tried looping over each element in the changes_at arrays and build up the sums, but that is too inefficient as I then have an inner for loop containing conditional branching code Thanks in advance, Slaunger -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Is-there-a-pure-numpy-recipe-fo… Sent from the Numpy-discussion mailing list archive at Nabble.com.

9 16

Re: [Numpy-discussion] Windows wheels using MKL?
by RayS 26 Mar '14

26 Mar '14

I've often wondered the particulars of the MKL; I have licensed via Enthought and distributed compiled works to client(s), and often use C. Gohkle's distros myself. - Ray At 05:29 PM 3/26/2014, you wrote: >Hi, > >On Wed, Mar 26, 2014 at 4:48 PM, Matthew Brett ><matthew.brett(a)gmail.com> wrote: > > Hi, > > > > Can I check what is stopping us building official numpy binary wheels > > for Windows using the Intel Math Kernel Library? > > > > * We'd need developer licenses, but those sound like they would be > > easy to come by > > * We'd have to add something to the license for the wheel on the lines > > of the Canopy license [1], derived from the MKL license [2] - is that > > a problem? > > > > Are there other problems for numpy? > >Talking with Fernando, we identified these as being the key problem >clauses in the MKL license [1]: > ><start quote> >D. DISTRIBUTION: Distribution of the Redistributables is also subject >to the following limitations: >[snipped clauses] > (iv) shall use a license agreement >that prohibits disassembly and reverse engineering of the >Redistributables, (v) shall indemnify, hold >harmless, and defend Intel and its suppliers from and against any >claims or lawsuits, including >attorney's fees, that arise or result from your distribution of any product. ></end quote> > >The first is a problem that might conceivably be adequately solved by >adding a paragraph to the Pypi page for numpy ("If you download and >install the windows binaries, you also agree... ") and copying a new >clause into the license in the installed tree. Maybe. The second >looks like it would be very hard to deal with for open source project >like us.... > >Cheers (sadly), > >Matthew > >[1] >http://software.intel.com/en-us/articles/intel-software-development-product… >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion(a)scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion

1 0

Missing Data
by T J 26 Mar '14

26 Mar '14

What is the status of: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst and of missing data in Numpy, more generally? Is np.ma.array still the "state-of-the-art" way to handle missing data? Or has something better and more comprehensive been put together?

3 2

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 90, Issue 83
by Colin J. Williams 26 Mar '14

26 Mar '14

On 25-Mar-2014 1:00 PM, numpy-discussion-request(a)scipy.org wrote: > Message: 3 Date: Mon, 24 Mar 2014 17:58:57 -0600 From: Charles R > Harris <charlesr.harris(a)gmail.com> Subject: Re: [Numpy-discussion] > Resolving the associativity/precedence debate for @ To: Discussion of > Numerical Python <numpy-discussion(a)scipy.org> Message-ID: > <CAB6mnxLyjNA5BHgOHO+U8+p3UmvXdjGg+ZUqfwi+vjFHfosxxQ(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" On Mon, Mar 24, 2014 at > 5:56 PM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >On Sat, Mar 22, 2014 at 6:13 PM, Nathaniel Smith<njs(a)pobox.com> wrote: >>> > >After 88 emails we don't have a conclusion in the other thread (see >>> > >[1] for background). But we have to come to some conclusion or another >>> > >if we want @ to exist:-). So I'll summarize where the discussion >>> > >stands and let's see if we can find some way to resolve this. >> > >> >Response in this thread so far seems (AFAICT) to have pretty much >> >converged on same-left. >> > >> >If you think that this would be terrible and there is some compelling >> >argument against it, then please speak up! Otherwise, if no-one >> >objects, then I'll go ahead in the next few days and put same-left >> >into the PEP. >> > > I think we should take a close look at broadcasting before deciding on the > precedence. > > Chuck >

2 1

Any numpy core devs on gittip (or similar)?
by Robert McGibbon 25 Mar '14

25 Mar '14

Hey, I've just been reading the discussion<http://code.activestate.com/lists/python-dev/129419/>on python-dev about PEP466, the meta-theme of which is that building and maintaining mission-critical software is just really tough. I don't really use SSL from python (the major topic of that discussion), but I depend on numpy and the awesome scientific stack every day. Anyways, that conversation made me think about what we take for granted in terms of software tools. If any of the numpy core devs are on gittip <https://www.gittip.com/> or similar, I think many of us would be happy to pitch in a few bucks for beer money or the like. Keep up the awesome, -Robert

1 0