Mailman 3 December 2012 - NumPy-Discussion

Re: [Numpy-discussion] how do I specify maximum line length when using savetxt?
by Mark Bakker 06 Dec '12

06 Dec '12

I guess I wasn't explicit enough. Say I have an array with 100 numbers and I want to write it to a file with 6 numbers on each line (and hence, only 4 on the last line). Can I use savetxt to do that? What other easy tool does numpy have to do that? Thanks, Mark On 5. des. 2012, at 22:35, Mark Bakker wrote: > Hello List, > > I want to write a large array to file, and each line can only be 80 > characters long. > Can I use savetxt to do that? Where would I specify the maximum line length? If you specify the format, %10.3f for instance, you will know the max line length if you also know the array shape. > Or is there a better way to do this? Probably 1000 ways to accomplish the same thing out there, sure. Cheers Paul

3 2

how do I specify maximum line length when using savetxt?
by Mark Bakker 05 Dec '12

05 Dec '12

Hello List, I want to write a large array to file, and each line can only be 80 characters long. Can I use savetxt to do that? Where would I specify the maximum line length? Or is there a better way to do this? Thanks, Mark

2 1

Weird Travis-CI bugs in the release 1.7.x branch
by Ondřej Čertík 04 Dec '12

04 Dec '12

Hi, I started to work on the release again and noticed weird failures at Travis-CI: https://github.com/numpy/numpy/pull/2782 The first commit (8a18fc7) should not trigger this failure: ====================================================================== FAIL: test_iterator.test_iter_array_cast ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/core/tests/test_iterator.py", line 836, in test_iter_array_cast assert_equal(i.operands[0].strides, (-96,8,-32)) File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py", line 252, in assert_equal assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg), verbose) File "/home/travis/virtualenv/python2.5/lib/python2.5/site-packages/numpy/testing/utils.py", line 314, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: item=0 ACTUAL: 96 DESIRED: -96 So I pushed a whitespace commit into the PR (516b478) yet it has the same failure. So it's there, it's not some random fluke at Travis. I created this testing PR: https://github.com/numpy/numpy/pull/2783 to try to nail it down. But I can't see what could have caused this, because the release branch was passing all tests last time I worked on it. Any ideas? Btw, I managed to reproduce the SPARC64 bug: https://github.com/numpy/numpy/issues/2668 so that's good. Now I just need to debug it. Ondrej P.S. My thesis was finally approved by the grad school today, doing some final changes took more time than expected, but I think that I am done now.

2 3

Numpy's definition of contiguous arrays
by Sebastian Berg 04 Dec '12

04 Dec '12

Hi, maybe someone has an opinion about how this can be handled and was not yet aware of this. In current numpy master (probably being reverted), the definition for contiguous arrays is changed such that it means "Contiguous in memory" and nothing more. What this means is this: 1. An array of size (1,3,1) is both C- and F-contiguous (Assuming `arr.strides[1] == arr.itemsize`). 2. However it is incorrect that `arr.strides[-1] == arr.itemsize` because the corresponding axes dimension is 1 so it does not matter for the memory layout. Also other similar assumptions about "clean strides" are incorrect. (This was always incorrect in corner cases) I think most will agree that this change reflects what these flags should indicate, because the exact value of the strides is not really important for the memory layout and for example for a row vector there is no reason to say it cannot be both C- and F-contiguous. However the change broke some code in scipy as well as sk-learn, that relied on `arr.strides[-1] == arr.itemsize` (for C-contiguous arrays). The fact that it was never noticed that this isn't quite correct indicates that there is certainly more code out there just like it. There is more discussion here: https://github.com/numpy/numpy/pull/2735 with suggestions for a possible deprecation process of having both definitions next to each other and deprecating the current, etc. I was personally wondering if it is good enough to ensure strides are cleaned up when an array is explicitly requested as contiguous which means: np.array(arr, copy=False, order='C').strides[-1] == arr.itemsize is always True, but: if arr.flags.c_contiguous: # It is possible that: arr.strides[-1] != arr.itemsize Which fixes the problems found yet since typically if you want to use the fact that an array is contiguous, you use this kind of command to make sure it is. But I guess it is likely too dangerous to assume that nobody only checks the flags and then continuous to do unwanted assumptions about strides. Best Regards, Sebastian

1 0

Allowing 0-d arrays in np.take
by Sebastian Berg 04 Dec '12

04 Dec '12

Hey, Maybe someone has an opinion about this (since in fact it is new behavior, so it is undefined). `np.take` used to not allow 0-d/scalar input but did allow any other dimensions for the indices. Thinking about changing this, meaning that: np.take(np.arange(5), 0) works. I was wondering if anyone has feelings about whether this should return a scalar or a 0-d array. Typically numpy prefers scalars for these cases (indexing would return a scalar too) for good reasons, so I guess that is correct. But since I noticed this wondering if maybe it returns a 0-d array, I thought I would ask here. Regards, Sebastian

2 1

Speed bottlenecks on simple tasks - suggested improvement
by Raul Cota 03 Dec '12

03 Dec '12

Hello, First a quick summary of my problem and at the end I include the basic changes I am suggesting to the source (they may benefit others) I am ages behind in times and I am still using Numeric in Python 2.2.3. The main reason why it has taken so long to upgrade is because NumPy kills performance on several of my tests. I am sorry if this topic has been discussed before. I tried parsing the mailing list and also google and all I found were comments related to the fact that such is life when you use NumPy for small arrays. In my case I have several thousands of lines of code where data structures rely heavily on Numeric arrays but it is unpredictable if the problem at hand will result in large or small arrays. Furthermore, once the vectorized operations complete, the values could be assigned into scalars and just do simple math or loops. I am fairly sure the core of my problems is that the 'float64' objects start propagating all over the program data structures (not in arrays) and they are considerably slower for just about everything when compared to the native python float. Conclusion, it is not practical for me to do a massive re-structuring of code to improve speed on simple things like "a[0] < 4" (assuming "a" is an array) which is about 10 times slower than "b < 4" (assuming "b" is a float) I finally decided to track down the problem and I started by getting Python 2.6 from source and profiling it in one of my cases. By far the biggest bottleneck came out to be PyString_FromFormatV which is a function to assemble a string for a Python error caused by a failure to find an attribute when "multiarray" calls PyObject_GetAttrString. This function seems to get called way too often from NumPy. The real bottleneck of trying to find the attribute when it does not exist is not that it fails to find it, but that it builds a string to set a Python error. In other words, something as simple as "a[0] < 3.5" internally result in a call to set a python error . I downloaded NumPy code (for Python 2.6) and tracked down all the calls like this, ret = PyObject_GetAttrString(obj, "__array_priority__"); and changed to if (PyList_CheckExact(obj) || (Py_None == obj) || PyTuple_CheckExact(obj) || PyFloat_CheckExact(obj) || PyInt_CheckExact(obj) || PyString_CheckExact(obj) || PyUnicode_CheckExact(obj)){ //Avoid expensive calls when I am sure the attribute //does not exist ret = NULL; } else{ ret = PyObject_GetAttrString(obj, "__array_priority__"); ( I think I found about 7 spots ) I also noticed (not as bad in my case) that calls to PyObject_GetBuffer also resulted in Python errors being set thus unnecessarily slower code. With this change, something like this, for i in xrange(1000000): if a[1] < 35.0: pass went down from 0.8 seconds to 0.38 seconds. A bogus test like this, for i in xrange(1000000): a = array([1., 2., 3.]) went down from 8.5 seconds to 2.5 seconds. Altogether, these simple changes got me half way to the speed I used to get in Numeric and I could not see any slow down in any of my cases that benefit from heavy array manipulation. I am out of ideas on how to improve further though. Few questions: - Is there any interest for me to provide the exact details of the code I changed ? - I managed to compile NumPy through setup.py but I am not sure how to force it to generate pdb files from my Visual Studio Compiler. I need the pdb files such that I can run my profiler on NumPy. Anybody has any experience with this ? (Visual Studio) - The core of my problems I think boil down to things like this s = a[0] assigning a float64 into s as opposed to a native float ? Is there any way to hack code to change it to extract a native float instead ? (probably crazy talk, but I thought I'd ask :) ). I'd prefer to not use s = a.item(0) because I would have to change too much code and it is not even that much faster. For example, for i in xrange(1000000): if a.item(1) < 35.0: pass is 0.23 seconds (as opposed to 0.38 seconds with my suggested changes) I apologize again if this topic has already been discussed. Regards, Raul

6 9

Apparently Non-deterministic behaviour of complex-array instantiation values
by Karl Kappler 03 Dec '12

03 Dec '12

Hello, This is a continuation of a problem I had last year, http://old.nabble.com/Apparently-non-deterministic-behaviour-of-complex-arr… at least it seems to have similar symptoms. I am working again with complex valued arrays in numpy (python version 2.7.3). This time however, the dataset is not very large, and I am able to post a snippet of the code. I became aware of the problem working in and IDE: Spyder 2.1.9, where I was repeatedly running code by pushing f5 and checking that my numerical results were what I expect. What I found was that the output in spyder varied somewhat randomly. In particular, when initializing a 2x2 complex-valued numpy array on line 32 of the code called “lowTri”. When I print the value of the upper right element (the one that should be zero) I often see 1.789+1.543j, or 0+1.543j, or 0+0j. This behavior happens when I run the code using f5 in spyder, and I thought it may be a Spyder issue, but further investigation has shown equally strange behavior on the command line as well. When I run this script on the command line the output is usually the same from run to run (although I have seen some variations, which I do not understand), but most remarkable, and reproducible, is that if I comment out line 17 (where the complex-valued array zzz is populated), the behavior of the initialization of lowTri varies. With Line 17 uncommented I usually get (on the command line): Lower Triangular [0,1]: 1.543j [[ 670.9 +1.22400000e-05j 0.0 +1.54300000e+00j] [ 195.8 -1.17300000e+02j 391.2 +1.46900000e-05j]] Lower Triangular [0,1]: 1.543j Real Part 0.0 and with line 17 it commented: Lower Triangular [0,1]: 0j [[ 670.9 +1.22400000e-05j 0.0 +0.00000000e+00j] [ 195.8 -1.17300000e+02j 391.2 +1.46900000e-05j]] Lower Triangular [0,1]: 0j Real Part 0.0 I.e.. the imaginary part is initialized to a different value. From reading up on forums I think I understand that when an array is allocated without specific values, it will be given random values which are very small, ie. ~1e-316 or so. But it would seem that sometimes initallization is done to a finite quantity. I know I can try to initialize the array using np.zeros() instead of np.ndarray(), but it is the principle I am concerned about. Last year it had been suggested that I had bad RAM, but these issues are reproducing on four computers, one of which has a new motherboard/RAM and AMD processor, and the others are Intel. Memtest has been run recently on at least two of the machines. Could someone try running this script with and without line 17 commented out and tell me if they are getting the same sort of behaviour? Thanks, Karl ********************************** import numpy as np if __name__ == "__main__": a={} a[0] = '0.1537E+00 0.1610E+01 -0.4801E+01 -0.3175E+01' a[1] = '0.1789E+01 0.1543E+01 -0.5524E+00 -0.8423E+00' c = '0.6709E+03 0.1224E-04 0.1958E+03 -0.1173E+03 0.3912E+03 0.1469E-04' ztmp = np.zeros((4,2)) zzz = np.zeros((2,2)) + complex(0,1)*np.zeros((2,2)); line = [] for iE in range(2): line = a[iE].split() for iElt, elt in enumerate(line): ztmp[iElt,iE] = float(elt) zzz[:,0:2] = ztmp[[0,2],:] + complex(0,1)*ztmp[[1,3],:] #commenting this line seems to affect value of lowTri stemp = np.zeros((2,3)) nElts = np.prod(stemp.shape) v = [] line = c.split() for l in line: v.append(float(l)) N=len(v)/2 cVec = np.ndarray(shape=(N), dtype=complex) for i in range(N): cVec[i] = complex(float(v[2*i]),float(v[2*(i+1)-1])) lowTri = np.ndarray(shape=(2,2), dtype=complex) #lowTri = np.zeros(shape=(2,2), dtype=complex) print("Lower Triangular [0,1]: {}".format(lowTri[0,1])) TI = np.tril_indices(2) rows = TI[0] cols = TI[1] for iCell in range(len(rows)): lowTri[rows[iCell],cols[iCell]] = cVec[iCell] print lowTri print("Lower Triangular [0,1]: {}".format(lowTri[0,1])) print("Real Part {}".format(np.real(lowTri[0,1])))

2 1

scalars and strange casting
by josef.pktd＠gmail.com 03 Dec '12

03 Dec '12

A followup on the previous thread on scalar speed. operations with numpy scalars I can *maybe* understand this >>> np.array(2)[()] * [0.5, 1] [0.5, 1, 0.5, 1] but don't understand this >>> np.array(2.+0.1j)[()] * [0.5, 1] __main__:1: ComplexWarning: Casting complex values to real discards the imaginary part [0.5, 1, 0.5, 1] The difference in behavior compared to the other operators, +,-, /,**, looks, at least, like an inconsistency to me. Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.array(2.+0.1j)[()] * [0.5, 1] __main__:1: ComplexWarning: Casting complex values to real discards the imaginary part [0.5, 1, 0.5, 1] >>> np.array(2.+0.1j)[()] ** [0.5, 1] array([ 1.41465516+0.0353443j, 2.00000000+0.1j ]) >>> np.array(2.+0.1j)[()] + [0.5, 1] array([ 2.5+0.1j, 3.0+0.1j]) >>> np.array(2.+0.1j)[()] / [0.5, 1] array([ 4.+0.2j, 2.+0.1j]) >>> np.array(2)[()] * [0.5, 1] [0.5, 1, 0.5, 1] >>> np.array(2)[()] / [0.5, 1] array([ 4., 2.]) >>> np.array(2)[()] ** [0.5, 1] array([ 1.41421356, 2. ]) >>> np.array(2)[()] - [0.5, 1] array([ 1.5, 1. ]) >>> np.__version__ '1.5.1' or >>> np.array(-2.+0.1j)[()] * [0.5, 1] [] >>> np.multiply(np.array(-2.+0.1j)[()], [0.5, 1]) array([-1.+0.05j, -2.+0.1j ]) >>> np.array([-2.+0.1j])[0] * [0.5, 1] [] >>> np.multiply(np.array([-2.+0.1j])[0], [0.5, 1]) array([-1.+0.05j, -2.+0.1j ]) Josef defensive programming = don't use python, use numpy arrays, or at least remember which kind of animals you have

1 0

Euler-Mascheroni constant
by Charles R Harris 02 Dec '12

02 Dec '12

Hi All, I put in a PR to expose the Euler-Mascheroni constant as 'euler_gamma'. The name is open to discussion. Suggestions for alternatives welcome. Chuck

1 0

allclose changed behaviour in 1.6.2 ?
by Martin Raspaud 02 Dec '12

02 Dec '12

Hi, We noticed that comparing arrays of different shapes with allclose doesn't work anymore in numpy 1.6.2. Is this a feature or a bug ? :) See the output in both 1.6.1 and 1.6.2 at the end of this mail. Best regards, Martin 1.6.1:: In [1]: import numpy as np In [2]: np.__version__ Out[2]: '1.6.1' In [3]: a = np.array([1, 2, 3]) In [4]: b = np.array([1, 2, 3, 4]) In [5]: np.allclose(a, b) Out[5]: False 1.6.2:: In[1]: import numpy as np In[2]: np.__version__ Out[2]: '1.6.2' In [3]: a = np.array([1, 2, 3]) In[4]: b = np.array([1, 2, 3, 4]) In[5]: np.allclose(a, b) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/maarten/pytroll/local/lib/python2.7/site-packages/numpy-1.6.2-py2.7-linux-x86_64.egg/numpy/core/numeric.py", line 1936, in allclose return all(less_equal(abs(x-y), atol + rtol * abs(y))) ValueError: operands could not be broadcast together with shapes (3) (4)

3 2