Mailman 3 March 2013 - NumPy-Discussion

fast numpy.fromfile skipping data chunks
by Andrea Cimatoribus 20 Mar '13

20 Mar '13

Hi everybody, I hope this has not been discussed before, I couldn't find a solution elsewhere. I need to read some binary data, and I am using numpy.fromfile to do this. Since the files are huge, and would make me run out of memory, I need to read data skipping some records (I am reading data recorded at high frequency, so basically I want to read subsampling). At the moment, I came up with the code below, which is then compiled using cython. Despite the significant performance increase from the pure python version, the function is still much slower than numpy.fromfile, and only reads one kind of data (in this case uint32), otherwise I do not know how to define the array type in advance. I have basically no experience with cython nor c, so I am a bit stuck. How can I try to make this more efficient and possibly more generic? Thanks import numpy as np #For cython! cimport numpy as np from libc.stdint cimport uint32_t def cffskip32(fid, int count=1, int skip=0): cdef int k=0 cdef np.ndarray[uint32_t, ndim=1] data = np.zeros(count, dtype=np.uint32) if skip>=0: while k<count: try: data[k] = np.fromfile(fid, count=1, dtype=np.uint32) fid.seek(skip, 1) k +=1 except ValueError: data = data[:k] break return data

12 26

numpy reference array
by Matt U 18 Mar '13

18 Mar '13

Is it possible to create a numpy array which points to the same data in a different numpy array (but in different order etc)? For example: Code: ------------------------------------------------------------------------------ import numpy as np a = np.arange(10) ids = np.array([0,0,5,5,9,9,1,1]) b = a[ids] a[0] = -1 b[0] #should be -1 if b[0] referenced the same data as a[0] 0 ------------------------------------------------------------------------------ ctypes almost does it for me, but the access is inconvenient. I would like to access b as a regular numpy array: Code: ------------------------------------------------------------------------------ import numpy as np import ctypes a = np.arange(10) ids = np.array([0,0,5,5,9,9,1,1]) b = [a[id:id+1].ctypes.data_as(ctypes.POINTER(ctypes.c_long)) for id in ids] a[0] = -1 b[0][0] #access is inconvenient -1 ------------------------------------------------------------------------------ Some more information: I've written a finite-element code, and I'm working on optimizing the python implementation. Profiling shows the slowest operation is the re-creation of an array that extracts edge degrees of freedom from the volume of the element (similar to b above). So, I'm trying to avoid copying the data every time, and just setting up 'b' once. The ctypes solution is sub-optimal since my code is mostly vectorized, that is, later I'd like to something like Code: ------------------------------------------------------------------------------ c[ids] = b[ids] + d[ids] ------------------------------------------------------------------------------ where c, and d are the same shape as b but contain different data. Any thoughts? If it's not possible that will save me time searching.

2 2

Memory issue with memory-mapped array assignment
by Thomas Robitaille 18 Mar '13

18 Mar '13

Hi everyone, I've come across a memory issue when trying to assign data to slices of a Numpy memory-mapped array. The short story is that if I create a memory mapped array and try to add data to subsets of the array many times in a loop, the memory usage of my code grows over time, suggesting there is some kind of memory leak. More specifically, if I run the following script: import random import numpy as np image = np.memmap('image.np', mode='w+', dtype=np.float32, shape=(10000, 10000)) print("Before assignment") for i in range(1000): x = random.uniform(1000, 9000) y = random.uniform(1000, 9000) imin = int(x) - 128 imax = int(x) + 128 jmin = int(y) - 128 jmax = int(y) + 128 data = np.random.random((256,256)) image[imin:imax, jmin:jmax] = image[imin:imax, jmin:jmax] + data del x, y, imin, imax, jmin, jmax, data the memory usage goes up to ~300Mb after 1000 iterations (and proportionally more if I increase the number of iterations). I've written up a more detailed overview of the issue on stackoverflow (with memory profiling): http://stackoverflow.com/questions/15473377/memory-issue-with-numpy-memory-… Does anyone have any idea what is going on, and how I can avoid this issue? Thanks! Tom

1 0

indexing of arbitrary axis and arbitrary slice?
by Chao YUE 18 Mar '13

18 Mar '13

Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 2

Possible bug in numpy 1.6.1
by Ake Sandgren 15 Mar '13

15 Mar '13

Hi! Found this thing that looks like a bug in core/src/multiarray/dtype_transfer.c diff -ru site/numpy/core/src/multiarray/dtype_transfer.c amd64_ubuntu1004-intel-acml/numpy/core/src/multiarray/dtype_transfer.c --- site/numpy/core/src/multiarray/dtype_transfer.c 2011-07-20 20:25:28.000000000 +0200 +++ amd64_ubuntu1004-intel-acml/numpy/core/src/multiarray/dtype_transfer.c 2013-03-14 22:09:38.000000000 +0100 @@ -1064,7 +1064,7 @@ _one_to_n_data *d = (_one_to_n_data *)data; PyArray_StridedTransferFn *subtransfer = d->stransfer, *stransfer_finish_src = d->stransfer_finish_src; - void *subdata = d->data, *data_finish_src = data_finish_src; + void *subdata = d->data, *data_finish_src = d->data_finish_src; npy_intp subN = d->N, dst_itemsize = d->dst_itemsize; while (N > 0) { -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: ake(a)hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se

2 3

Fox News
by Kevin Christman 15 Mar '13

15 Mar '13

http://ezinsurance.org/bpln/iebpmnsmrme.htm

1 0

Any help from Numpy community?
by Birdada Simret 14 Mar '13

14 Mar '13

* Any help from Numpy community [[ 0. 1.54 0. 0. 0. 1.08 1.08 1.08 ] [ 1.54 0. 1.08 1.08 1.08 0. 0. 0. ] [ 0. 1.08 0. 0. 0. 0. 0. 0. ] [ 0. 1.08 0. 0. 0. 0. 0. 0. ] [ 0. 1.08 0. 0. 0. 0. 0. 0. ] [ 1.08 0. 0. 0. 0. 0. 0. 0. ] [ 1.08 0. 0. 0. 0. 0. 0. 0. ] [ 1.08 0. 0. 0. 0. 0. 0. 0. ]] the above is the numpy array matrix. the numbers represents: C-C: 1.54 and C-H=1.08 So I want to write this form as C of index i is connected to C of index j C of index i is connected to H of index j (C(i),C(j)) # key C(i) and value C(j) (C(i),H(j)) # key C(i) and value H(j) ; the key C(i) can be repeated to fulfil as much as the values of H(j) To summarize, the out put may look like: C1 is connected to C2 C1 is connected to H1 C1 is connected to H3 C2 is connected to H2 etc.... Any guide is greatly appreciated, thanks birda *

2 5

Adopt Mersenne Twister 64bit?
by Siu Kwan Lam 14 Mar '13

14 Mar '13

Hi all, I am redirecting a discussion on github issue tracker here. My original post (https://github.com/numpy/numpy/issues/3137): "The current implementation of the RNG seems to be MT19937-32. Since 64-bit machines are common nowadays, I am suggesting adding or upgrading to MT19937-64. Thoughts?" Let me start by answering to njsmith's comments on the issue tracker: > Would it be faster? Although I have not benchmarked the 64-bit implementation, it is likely that it will be faster on a 64-bit machine since the number of iteration (controlled by NN and MM in the reference implementation http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/VERSIONS/C-LANG/mt19937-64.c) is reduced by half. In addition, each generation in the 64-bit implementation produces a 64-bit random int which can be used to generate double precision random number. Unlike the 32-bit implementation which requires generating a pair of 32-bit random int. But, on a 32-bit machine, a 64-bit instruction is translated into 4 32-bit instructions; thus, it is likely to be slower. (1) > Use less memory? The amount of memory use will remain the same. The size of the RNG state is the same. > Provide higher quality randomness? My naive answer is that 32-bit and 64-bit implementation have the same 2^19937-1 period. Need to do some research and experiments. > Would it change the output of this program: > import numpy > numpy.random.seed(0) > print numpy.random.random() > ? Unfortunately, yes. The 64-bit implementation generates a different random number sequence with the same seed. (2) My suggestion to overcome (1) and (2) is to allow the user to select between the two implementations (and possibly different algorithms in the future). If user does not provide a choice, we use the MT19937-32 by default. numpy.random.set_state("MT19937_64", …) # choose the 64-bit implementation Thoughts? Best, Siu

5 15

can't run cython on mtrand.pyx
by Neal Becker 14 Mar '13

14 Mar '13

Grabbed numpy-1.7.0 source. Cython is 0.18 cython mtrand.pyx produces lots of errors.

2 3

Yes, this one again "ImportError: No module named multiarray"
by Dinesh B Vadhia 14 Mar '13

14 Mar '13

I've been using Numpy/Scipy for >5 years so know a little on how to get around them. Recently, I've needed to either freeze or create executables with tools such as PyInstaller, Cython, Py2exe and others on both Windows (XP 32-bit, 7 64-bit) and Ubuntu (12.04) Linux (64-bit). The test program (which runs perfectly with the Python interpreter) is very simple: import numpy def main(): print numpy.array([12, 23, 34, 45, 56, 67, 78, 89, 90]) return if __name__ == '__main__': main() The software versions are Python 2.7.3, Numpy 1.7.0, and Scipy 0.11. The "import numpy" causes an "ImportError: No module named multiarray". After endless Googling, I am none the wiser about what (really) causes the ImportError let alone what the solution is. The Traceback, similar to others found on the web, is: Traceback (most recent call last): File "test.py", ... File "C:\Python27\lib\site-packages\numpy\__init__.py", line 154, in <module> import add_newdocs File "C:\Python27\lib\site-packages\numpy\add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "C:\Python27\lib\site-packages\numpy\lib\__init__.py", line 4, in <module> from type_check import * File "C:\Python27\lib\site-packages\numpy\lib\type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "C:\Python27\lib\site-packages\numpy\core\__init__.py", line 5, in <module> import multiarray ImportError: No module named multiarray. Could someone shed some light on this - please? Thx.

4 5