Mailman 3 May 2009 - NumPy-Discussion

Benchmak on record arrays
by Nicolas Rougier 29 May '09

29 May '09

Hi, I've written a very simple benchmark on recarrays: import numpy, time Z = numpy.zeros((100,100), dtype=numpy.float64) Z_fast = numpy.zeros((100,100), dtype=[('x',numpy.float64), ('y',numpy.int32)]) Z_slow = numpy.zeros((100,100), dtype=[('x',numpy.float64), ('y',numpy.bool)]) t = time.clock() for i in range(10000): Z*Z print time.clock()-t t = time.clock() for i in range(10000): Z_fast['x']*Z_fast['x'] print time.clock()-t t = time.clock() for i in range(10000): Z_slow['x']*Z_slow['x'] print time.clock()-t And got the following results: 0.23 0.37 3.96 Am I right in thinking that the last case is quite slow because of some memory misalignment between float64 and bool or is there some machinery behind that makes things slow in this case ? Should this be mentioned somewhere in the recarray documentation ? Nicolas

4 8

sparse matrix dot product
by Nicolas Rougier 29 May '09

29 May '09

Hi, I'm now testing dot product and using the following: import numpy as np, scipy.sparse as sp A = np.matrix(np.zeros((5,10))) B = np.zeros((10,1)) print (A*B).shape print np.dot(A,B).shape A = sp.csr_matrix(np.zeros((5,10))) B = sp.csr_matrix((10,1)) print (A*B).shape print np.dot(A,B).shape A = sp.csr_matrix(np.zeros((5,10))) B = np.zeros((10,1)) print (A*B).shape print np.dot(A,B).shape I got: (5, 1) (5, 1) (5, 1) (5, 1) (5, 1) (10, 1) Obviously, the last computation is not a dot product, but I got no warning at all. Is that the expected behavior ? By the way, I wrote a speed benchmark for dot product using the different flavors of sparse matrices and I wonder if it should go somewhere in documentation (in anycase, if anyone interested, I can post the benchmark program and result). Nicolas

3 3

Numpy vs PIL in image statistics
by cp 29 May '09

29 May '09

Testing the PIL vs numpy in calculating the mean value of each color channel of an image I timed the following. impil = Image.open("10.tif") imnum = asarray(impil) #in PIL for i in range(1,10): stats = ImageStat.Stat(impil) stats.mean # for numpy for i in range(1,10): imnum.reshape(-1,3).mean(axis=0) The image I tested initially is 2000x2000 RGB tif ~11mb in size. I set a timer in each for loop and measured the performance of numpy 7 times slower than PIL. When I did the the same with an 10x10 RGB tif and with 1000 cycles in for, numpy was 25 times faster than PIL. Why is that? Does mean or reshape, make a copy?

4 5

numpy ufuncs and COREPY - any info?
by dmitrey 28 May '09

28 May '09

hi all, has anyone already tried to compare using an ordinary numpy ufunc vs that one from corepy, first of all I mean the project http://socghop.appspot.com/student_project/show/google/gsoc2009/python/t124… It would be interesting to know what is speedup for (eg) vec ** 0.5 or (if it's possible - it isn't pure ufunc) numpy.dot(Matrix, vec). Or any another example.

6 17

Nested recarrays with subarrays and loadtxt: a bug in loadtxt?
by Fernando Perez 28 May '09

28 May '09

Howdy, I'm wondering if the code below illustrates a bug in loadtxt, or just a 'live with it' limitation. I'm inlining it for ease of discussion, but the same code is attached to ensure that anyone willing to look at this can just download and run without pasting/whitespace issues. The code is, I hope, sufficiently commented to explain the problem in full detail. Thanks for any input! Cheers, f ### """Simple illustration of nested record arrays. Note: possible numpy.loadtxt bug?""" from StringIO import StringIO import numpy as np from numpy import array, dtype, loadtxt, recarray # Consider the task of loading data that is stored in plain text in a file such # as the string below, where the last block of numbers is meant to be # interpreted as a single 2x3 int array, whose field name in the resulting # structured array will be 'block'. txtdata = StringIO(""" # name x y block - 2x3 ints aaaa 1.0 8.0 1 2 3 4 5 6 aaaa 2.0 7.4 2 11 22 3 4 5 6 bbbb 3.5 8.5 3 0 22 44 5 6 aaaa 6.4 4.0 4 1 3 33 54 65 aaaa 8.8 4.1 5 5 3 4 44 77 bbbb 5.5 9.1 6 3 4 5 0 55 bbbb 7.7 8.5 7 2 3 4 5 66 """) # We make the dtype for it: dt = dtype(dict(names=['name','x','y','block'], formats=['S4',float,float,(int,(2,3))])) # And we load it with loadtxt and make a recarray version for convenience data = loadtxt(txtdata,dt) rdata = data.view(recarray) # Unfortunately, if we look at the block data, it repeats the first number # found. This seems to be a loadtxt bug: # In [176]: rdata.block[0,1] # Out[176]: array([1, 1, 1]) # we'd expect array([4, 5, 6]) if np.any(rdata.block[0,1] != array([4, 5, 6])): print 'WARNING: loadtxt bug??' # A workaround can be used by doing a second pass on the file, loading the # columns corresponding to the block as plain ints and doing a reassignment of # that data into the original data. # Rewind the data and reload only the 'block' of ints: txtdata.seek(0) block_data = loadtxt(txtdata,int,usecols=range(3,9)) # Let's work with a copy of the original so we can compare interactively... rdata2 = rdata.copy() # We assign to the block field in our real array the block_data one, # appropriately reshaped rdata2.block[:] = block_data.reshape(rdata.block.shape) # Same check as before, with the new one if np.any(rdata2.block[0,1] != array([4, 5, 6])): print 'WARNING: loadtxt bug??' else: print 'Second pass - data loaded OK.'

3 10

convert between structure arrays with different record orderings?
by roger peppe 28 May '09

28 May '09

hi, sorry, i'm new to the list, and if this is a frequently asked question, please point me in the right direction. say, for some reason i've got two numpy structure arrays that both contain the same fields with the same types but in a different order, is there a simple way to convert one to the other type? i'd have thought that astype() might do the job, but it seems to ignore the field names entirely, thus values are lost in the following transcript: > t1 = np.dtype([('foo', np.float64), ('bar', '?')]) > t2 = np.dtype([('bar', '?'), ('foo', np.float64)]) > a1 = np.array([(123, False), (654, True)], dtype=t1) > a1.astype(t2) array([(True, 0.0), (True, 1.0)], dtype=[('bar', '|b1'), ('foo', '<f8')]) > thanks for any help, rog.

1 0

Failure with 1.3
by Matthieu Brucher 28 May '09

28 May '09

Hi, I've just tested the latest numpy with my new configuration (Opteron 2220, 64bits with RH5.2, compiled with ICC 10.1.018) and I got this failure. ====================================================================== FAIL: test_umath.TestLogAddExp2.test_logaddexp2_values [...] assert_almost_equal(np.logaddexp2(xf, yf), zf, decimal=dec) [...] raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 80.0%) x: array([ 2.32090838, 2.00127574, 2.5849625 , 2.00127574, 2.32090838]) y: array([ 2.5849625, 2.5849625, 2.5849625, 2.5849625, 2.5849625]) Is it fixed in the current trunk? Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher

2 2

arrray/matrix nonzero() type
by Nicolas Rougier 27 May '09

27 May '09

Hi again, I have a problem with the nonzero() function for matrix. The following test program: import numpy, scipy.sparse Z = numpy.zeros((10,10)) Z[0,0] = Z[1,1] = 1 i = Z.nonzero() print i Zc = scipy.sparse.coo_matrix((Z[i],i)) Z = numpy.matrix(Z) i = Z.nonzero() print i Zc = scipy.sparse.coo_matrix((Z[i],i)) gives me: (array([0, 1]), array([0, 1])) (matrix([[0, 1]]), matrix([[0, 1]])) Traceback (most recent call last): File "test.py", line 13, in <module> Zc = scipy.sparse.coo_matrix((Z[i],i)) File "/Volumes/Data/Local/lib/python2.6/site-packages/scipy/sparse/ coo.py", line 179, in __init__ self._check() File "/Volumes/Data/Local/lib/python2.6/site-packages/scipy/sparse/ coo.py", line 194, in _check nnz = self.nnz File "/Volumes/Data/Local/lib/python2.6/site-packages/scipy/sparse/ coo.py", line 187, in getnnz raise ValueError('row, column, and data arrays must have rank 1') ValueError: row, column, and data arrays must have rank 1 Is that the intended behavior ? How can I use nonzero with matrix to build the coo one ? Nicolas

2 1

asarray() and PIL
by cp 27 May '09

27 May '09

Hi, I'm using PIL for image processing, but lately I also try numpy for the flexibility and superior speed it offers. The first thing I noticed is that for an RGB image with height=1600 and width=1900 while img=Image.open('something.tif') img.size (1900,1600) then arr=asarray(img) arr.shape (1600,1900,3) This means that the array-image has 1600 color channels, 1900 image pixel rows and 3 image pixel columns. Why is that? if I reshape with arr.reshape(3,1900,1600) will there be a mix-up in pixel values and coordinates when compared to the initial PIL image?

4 4

casting bug
by Charles سمير Doutriaux 27 May '09

27 May '09

Hi there, One of our users just found a bug in numpy that has to do with casting. Consider the attached example. The difference at the end should be 0 (zero) everywhere. But it's not by default. Casting the data to 'float64' at reading and assiging to the arrays works Defining the arrays "at creation time" as "float32" works But simply reading in the data (flaot32) and putting them into the default arrays (float64) leads to differences! Thanks for looking into this, C.

2 2