Mailman 3 January 2014 - NumPy-Discussion

Speedup by avoiding memory alloc twice in scalar array
by Arink Verma Jan. 8, 2014

Jan. 8, 2014

Hi, I am working on performance parity between numpy scalar/small array and python array as GSOC mentored By Charles. Currently I am looking at PyArray_Return, which allocate separate memory just for scalar return. Unlike python which allocate memory once for returning result of scalar operations; numpy calls malloc twice once for the array object itself, and a second time for the array data. These memory allocations are happening in PyArray_NewFromDescr and PyArray_Scalar. Stashing both within a single allocation would be more efficient. In, PyArray_Scalar, new struct (PyLongScalarObject) need allocation in case of scalar arrays. Instead, can we just some how convert/cast PyArrayObject to PyLongScalarObject.?? -- Arink Verma www.arinkverma.in

5 12

an indexing question
by Neal Becker Jan. 8, 2014

Jan. 8, 2014

I have a 1d vector d. I want compute the means of subsets of this vector. The subsets are selected by looking at another vector s or same shape as d. This can be done as: [np.mean (d[s == i]) for i in range (size)] But I think this could be done directly with numpy addressing, without resorting to list comprehension?

2 1

LLVM
by Charles R Harris Jan. 7, 2014

Jan. 7, 2014

Has anyone tried using LLVM with Visual Studio? It is supposed to work with Visual Studio >= 2010 and might provide an alternative to MinGw64. Chuck

2 1

C99 compatible complex number tests fail
by Matti Picus Jan. 4, 2014

Jan. 4, 2014

Hi. I started to port the stdlib cmath C99 compatible complex number tests to numpy, after noticing that numpy seems to have different complex number routines than cmath. The work is available on a "retest_complex" branch of numpy https://github.com/mattip/numpy/tree/retest_complex The tests can be run by pulling the branch (no need to rebuild numpy) and running python <path-to-branch>/numpy/core/tests/test_umath_complex.py > test.log 2>&1 So far it is just a couple of commits that run the tests on numpy, I did not dive into modifying the math routines. If I did the work correctly, failures point to some differences, most due to edge cases with inf and nan, but there are a number of failures due to different finite values (for some small definition of different). I guess my first question is "did I do the tests properly". Assuming I did, the next question is "are the inconsistencies intentional" i.e. are they that way in order to be compatible with Matlab or some other non-C99 conformant library? For instance, a comparison between the implementation of cmath's sqrt and numpy's sqrt shows that numpy does not check for subnormals. And I am probably mistaken since I am new to the generator methods of numpy, but could it be that trigonometric functions like acos and acosh are generated in umath/funcs.inc.src, using a very different algorithm than cmathmodule.c? Would there be interest in a pull request that changed the routines to be more compatible with results from cmath? Matti

3 2

proposal: min, max of complex should give warning
by Neal Becker Jan. 4, 2014

Jan. 4, 2014

I propose the following change: min, max applied to complex should give a warning. The rationale is, when the user applies min or max to complex, it's probably a mistake.

3 5

ArrayList object
by Nicolas Rougier Jan. 4, 2014

Jan. 4, 2014

Hi all, I've coding an ArrayList object based on a regular numpy array. This objects allows to dynamically append/insert/delete/access items. I found it quite convenient since it allows to manipulate an array as if it was a list with elements of different sizes but with same underlying type (=array dtype). # Creation from a nested list L = ArrayList([ [0], [1,2], [3,4,5], [6,7,8,9] ]) # Creation from an array + common item size L = ArrayList(np.ones(1000), 3) # Empty list L = ArrayList(dype=int) # Creation from an array + individual item sizes L = ArrayList(np.ones(10), 1+np.arange(4)) # Access to elements: print L[0], L[1], L[2], L[3] [0] [1 2] [3 4 5] [6 7 8 9] # Operations on elements L[:2] += 1 print L.data [1 2 3 3 4 5 6 7 8 9] Source code is available from: https://github.com/rougier/array-list I wonder is there is any interest in having such object within core numpy (np.list ?) ? Nicolas

1 0

Re: [Numpy-discussion] Quaternion type @ rosettacode.org
by David Goldsmith Jan. 3, 2014

Jan. 3, 2014

Thanks Anthony and Paul! OlyDLG

1 0

Padding An Array Along A Single Axis
by Freddie Witherden Jan. 3, 2014

Jan. 3, 2014

Hi all, This should be an easy one but I can not come up with a good solution. Given an ndarray with a shape of (..., X) I wish to zero-pad it to have a shape of (..., X + K), presumably obtaining a new array in the process. My best solution this far is to use np.zeros(curr.shape[:-1] + (curr.shape[-1] + K,)) followed by an assignment. However, this seems needlessly cumbersome. I looked at np.pad but it does not seem to provide a means of just padding a single axis easily. Regards, Freddie.

2 1

Loading large NIfTI file -> MemoryError
by Amira Chekir Jan. 3, 2014

Jan. 3, 2014

Hello together, I try to load a (large) NIfTI file (DMRI from Human Connectome Project, about 1 GB) with NiBabel. import nibabel as nib img = nib.load("dmri.nii.gz") data = img.get_data() The program crashes during "img.get_data()" with an "MemoryError" (having 4 GB of RAM in my machine). Any suggestions? Best regards, AMIRA

3 2

Quaternion type @ rosettacode.org
by David Goldsmith Jan. 2, 2014

Jan. 2, 2014

Anyone here use/have an opinion about the Quaternion type @ rosettacode.org<http://rosettacode.org/wiki/Simple_Quaternion_type_and_operations#Python>? Or have an opinion about it having derived the type from collections.namedtuple? Anyone have an open-source, numpy-based alternative? Ditto last question for Octonion and/or general n-basis Grassmann (exterior) and/or Clifford Algebras? (rosettacode appears to have none of these). Thanks! David Goldsmith

3 2

Altering/initializing NumPy array in C
by Bart Baker Jan. 2, 2014

Jan. 2, 2014

Hello, I'm having issues with performing operations on an array in C and passing it back to Python. The array values seem to become unitialized upon being passed back to Python. My first attempt involved initializing the array in C as so: double a_fin[max_mth]; where max_mth is an int. I fill in the values of a_fin and then, before returning back to Python, I create a NumPy array in C and fill it in using a pointer to the a_fin array: npy_intp a_dims[2] = {max_mth, 1}; a_fin_array = (PyArrayObject *) PyArray_SimpleNewFromData(2, a_dims, NPY_DOUBLE, a_fin); I update the flags as so: PyArray_UpdateFlags(a_fin_array, NPY_OWNDATA); and return using: PyObject *Result = Py_BuildValue("OO", a_fin_array, b_fin_array); Py_DECREF(a_fin_array); (there is another array, b_bin_array that I create in this way and it suffers from the same issues). Immediately upon returning to Python, all of a_fin_array appears unitilized. This only happens in certain situations and sometime only part of the arrary will be unitilized. I check the values of a_dim and a_fin_array in C using gdb and they appear as expected, but are over-written with unitialized values upon returning to Python. I've tried initializing in Python and then passing the NumPy array in instead of initializing in C, but the effects of the calculations in C are still not kept. My feeling is that, with the Numpy C-API, this should be a simple process, but I'm having a lot of trouble with it. Any help would be much appreciated. I didn't want to give too much information in the post, but please let me know what other information would be useful. -Bart

2 2

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 88, Issue 1
by Amira Chekir Jan. 1, 2014

Jan. 1, 2014

On 31.12.2013 14:13, Amira Chekir wrote: > > Hello together, > > > > I try to load a (large) NIfTI file (DMRI from Human Connectome Project, > > about 1 GB) with NiBabel. > > > > import nibabel as nib > > img = nib.load("dmri.nii.gz") > > data = img.get_data() > > > > The program crashes during "img.get_data()" with an "MemoryError" > > (having 4 GB of RAM in my machine). > > > > Any suggestions? > > are you using a 64 bit operating system? > which version of numpy? > > assuming nibabel uses np.load under the hood you could try it with numpy > 1.8 which reduces excess memory usage when loading compressed files. Hi, Thanks for your answer. I use ubuntu 12.04 32 bits and python 2.7 I upgrade numpy to 1.8, but the error persists I think that the problem is in gzip.py : max_read_chunk = 10 * 1024 * 1024 # 10Mb What do you think? Best regards, AMIRA 2014/1/1 <numpy-discussion-request(a)scipy.org> > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: proposal: min, max of complex should give warning (Ralf > Gommers) (David Goldsmith) > 2. Re: NumPy-Discussion Digest, Vol 87, Issue 35 (Amira Chekir) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 31 Dec 2013 11:43:49 -0800 > From: David Goldsmith <d.l.goldsmith(a)gmail.com> > Subject: Re: [Numpy-discussion] proposal: min, max of complex should > give warning (Ralf Gommers) > To: numpy-discussion(a)scipy.org > Message-ID: > <CAFtPsZr9FN6wwr85ScDNhqdkWZhG7C+= > rWU6EVuBMG+mY-XJdA(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > > > > As for your proposal, it would be good to know if adding a warning would > > actually catch any bugs. For the truncation warning it caught several in > > scipy and other libs IIRC. > > > > Ralf > > > > In light of this, perhaps the pertinent unit tests should be modified (even > if the warning suggestion isn't adopted, about which I'm neutral...but I'm > a little surprised that there isn't a generic way to globally turn off > specific warnings). > > DG >

2 1

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 87, Issue 35
by Amira Chekir Jan. 1, 2014

Jan. 1, 2014

Hi, Thanks for your answer. I use ubuntu 12.04 32 bits and python 2.7 I upgrade numpy to 1.8, but the error persists I think that the problem is in gzip.py : max_read_chunk = 10 * 1024 * 1024 # 10Mb What do you think? Best regards, AMIRA 2013/12/31 <numpy-discussion-request(a)scipy.org> > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Loading large NIfTI file -> MemoryError (Amira Chekir) > 2. Re: Loading large NIfTI file -> MemoryError (Julian Taylor) > 3. Re: proposal: min, max of complex should give warning (Cera, > Tim) > 4. Re: proposal: min, max of complex should give warning > (Neal Becker) > 5. Re: proposal: min, max of complex should give warning > (Ralf Gommers) > 6. Re: proposal: min, max of complex should give warning > (Neal Becker) > 7. ANN: NumPy 1.7.2 release (Julian Taylor) > 8. Re: ANN: NumPy 1.7.2 release (Charles R Harris) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 31 Dec 2013 14:13:57 +0100 > From: Amira Chekir <chekir.amira(a)gmail.com> > Subject: [Numpy-discussion] Loading large NIfTI file -> MemoryError > To: numpy-discussion(a)scipy.org > Message-ID: > <CAB-foYgW4HiL37aPchrbYkgPNJLY1Dt55M8Ki30UaTg= > EQ29Zw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hello together, > > I try to load a (large) NIfTI file (DMRI from Human Connectome Project, > about 1 GB) with NiBabel. > > import nibabel as nib > img = nib.load("dmri.nii.gz") > data = img.get_data() > > The program crashes during "img.get_data()" with an "MemoryError" (having 4 > GB of RAM in my machine). > > Any suggestions? > > Best regards, > AMIRA >

1 0