Mailman 3 May 2009 - NumPy-Discussion

Re: [Numpy-discussion] How to merge or SQL join record arrays in Python?
by Wei Su 11 May '09

11 May '09

Hi, Pierre: Thanks for the reply. I can now actually turn a big list into a record array. My question is actually how to join related record arrays in Python. This is done in SAS by MERGE and PROC SQL and by merge() in R. But I have no idea how to do it in Python. Thanks. Wei Su --- On Mon, 5/11/09, Pierre GM <pgmdevlist(a)gmail.com> wrote: From: Pierre GM <pgmdevlist(a)gmail.com> Subject: Re: [Numpy-discussion] How to merge or SQL join record arrays in Python? To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Date: Monday, May 11, 2009, 10:03 PM On May 11, 2009, at 5:44 PM, Wei Su wrote: > > Coming from SAS and R, this is probably the first thing I want to do > now that I can convert my data into record arrays. But I could not > find any clues after googling for a while. Any hint or suggestions > will be great! That depends what you want, actually, ut this should get you started http://docs.scipy.org/doc/numpy/user/basics.rec.html Note the slight difference between a structured array (fields accessible as items) and a recarray (fields accessible as items and attributes). _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

3 3

How to merge or SQL join record arrays in Python?
by Wei Su 11 May '09

11 May '09

Hi, All, Coming from SAS and R, this is probably the first thing I want to do now that I can convert my data into record arrays. But I could not find any clues after googling for a while. Any hint or suggestions will be great! Thanks a lot. Wei Su

2 1

strange behavior convolving via fft
by Chris Colbert 11 May '09

11 May '09

at least I think this is strange behavior. When convolving an image with a large kernel, its know that its faster to perform the operation as multiplication in the frequency domain. The below code example shows that the results of my 2d filtering are shifted from the expected value a distance 1/2 the width of the filter in both the x and y directions. Can anyone explain why this occurs? I have been able to find the answer in any of my image processing books. The code sample below is an artificial image of size (100, 100) full of zeros, the center of the image is populated by a (10, 10) square of 1's. The filter kernel is also a (10,10) square of 1's. The expected result of the convolution would therefore be a peak at location (50,50) in the image. Instead, I get (54, 54). The same shifting occurs regardless of the image and filter (assuming the filter is symetric, so flipping isnt necessary). I came across this behavior when filtering actual images, so this is not a byproduct of this example. The same effect also occurs using the full FFT as opposed to RFFT. I have links to the images produced by this process below. Thanks for any insight anyone can give! Chris In [12]: a = np.zeros((100,100)) In [13]: a[45:55,45:55] = 1 In [15]: k = np.ones((10,10)) In [16]: afft = np.fft.rfft2(a, s=(256,256)) In [19]: kfft = np.fft.rfft2(k, s=(256,256)) In [21]: result = np.fft.irfft2(afft*kfft).real[0:100,0:100] In [23]: result.argmax() Out[23]: 5454 www.therealstevencolbert.com/dump/a.png www.therealstevencolbert.com/dump/afft.png www.therealstevencolbert.com/dump/kfft.png www.therealstevencolbert.com/dump/result.png

3 7

OS-X binary name...
by Christopher Barker 11 May '09

11 May '09

Hi all, The binary for OS-X on sourceforge is called: numpy-1.3.0-py2.5-macosx10.5.dmg However, as far as I can tell, it works just fine on OS-X 10.4, and maybe even 10.3.9. Perhaps a re-naming is in order? But to what? I'd say: numpy-1.3.0-py2.5-macosx10.4.dmg but would folks think that it's only for 10.4? maybe: numpy-1.3.0-py2.5-macosx-python.org.dmg to indicate that it's for the python.org build of python2.5, though I'v never seen anyone use that convention. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

3 4

Re: [Numpy-discussion] String manipulation
by Sebastien Binet 11 May '09

11 May '09

On Monday 11 May 2009 14:36:17 Nils Wagner wrote: > On Mon, 11 May 2009 14:25:46 +0200 > > Francesc Alted <faltet(a)pytables.org> wrote: > > A Monday 11 May 2009, Nils Wagner escrigué: > >> Hi all, > >> > >> Please consider two strings > >> > >> >>> line_a > >> > >> '12345678abcdefgh12345678' > >> > >> >>> line_b > >> > >> '12345678 abcdefgh 12345678' > >> > >> >>> line_b.split() > >> > >> ['12345678', 'abcdefgh', '12345678'] > >> > >> Is it possible to split line_a such that the output > >> is > >> > >> ['12345678', 'abcdefgh', '12345678'] > > > > Mmh, your question is a bit too generic. > > Indeed. > I would like to split strings made of digits after eight > characters each. > > >>> line_a > > '111111.1222222.2333333.3' > > >>> line_b > > '111111.1 222222.2 333333.3' > > >>> line_b.split() > > ['111111.1', '222222.2', '333333.3'] > > How can I accomplish that ? would this suit you ? >>> np.asarray(line_b,dtype=[('hdr','|S8'),('mid','|S8'),('tail','|S8')]) array(('111111.1', '222222.2', '333333.3'), dtype=[('hdr', '|S8'), ('mid', '|S8'), ('tail', '|S8')]) hth, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay #########################################

2 2

List of arrays
by Nils Wagner 11 May '09

11 May '09

Hi all, How can I convert a list of arrays into one array ? Nils >>> data [array([ 40. , 285.6, 45. , 285.3, 50. , 285.1, 55. , 284.8]), array([ 60. , 284.5, 65. , 282.8, 70. , 281.1, 75. , 280. ]), array([ 80. , 278.8, 85. , 278.1, 90. , 277.4, 95. , 276.9]), array([ 100. , 276.3, 105. , 276.1, 110. , 275.9, 115. , 275.7]), array([ 120. , 275.5, 125. , 275.2, 130. , 274.8, 135. , 274.5]), array([ 140. , 274.1, 145. , 273.7, 150. , 273.2, 155. , 272.7]), array([ 160. , 272.2, 165. , 272.1, 170. , 272. , 175. , 271.8]), array([ 180. , 271.6, 185. , 271. , 190. , 270.3, 195. , 269.5]), array([ 200. , 268.5, 205. , 267.4, 210. , 266.1, 215. , 263.5]), array([ 220. , 260.1, 225. , 256.1, 230. , 249.9, 235. , 239.3]), array([ 238.7, 186.2, 240., 160. , 245. , 119.7, 250. , 111.3])] newdata=array([ 40. , 285.6, 45. , 285.3, 50. , 285.1, 55. , 284.8, 60. , 284.5, 65. , 282.8, ..., 111.3]) Nils

3 3

Subcripts for arrays of nested structures
by Robert Radocinski 10 May '09

10 May '09

In order to illustrate my question which involves two related subscript expressions, on arrays of nested structures, I have created a short code example which is given below. In the code example, I create a nested "structure" data type (called dtype3) and two numpy.ndarray's (called arecords and brecords) which are identical in shape and dtype. The remaining code consists of the replacement statement (lines 34 and 49) at the heart of my question and print statements. Specifically, my question centers on my expectation that the statement on line 34 (arecords["data"][1] = data_values) would have the same effect on arecords as the statement on line 49 (brecords[1]["data"]=data_values) would have on brecords. From running the code example, this is obviously not the case. In the former case, all four of the "data" arrays of the second record in arecords are set to the values found in data_values. In the latter case, only the first "data" array of the second record in brecords is set to a value in data_values and the other three "data" values remain unchanged. I am baffled by the latter case involving brecords. If you examine the left-hand side and righthand side of the statement on line 49 (brecords[1]["data"]=data_values), both sides have the same shape(1-dim with 4 elements) and dtype. I am having a great deal of difficulty trying to understand why the replacement statement only effects the first "data" array and not all 4 "data" arrays. What am I overlooking? Any help in explaining this behavior would be appreciated. Thanks, RR CODE EXAMPLE ---------------------------------------------------------------------------- import numpy type1 = numpy.dtype([("a", numpy.int32), ("b", numpy.int32)]) type2 = numpy.dtype([("alpha", numpy.float64), ("beta", numpy.float64), ("gamma", numpy.float64)]) type3 = numpy.dtype([("header", type1), ("data", type2, 4)]) header_values = numpy.empty(1, dtype=type1) data_values = numpy.empty(4, dtype=type2) header_values["a"] = 1000 header_values["b"] = 2000 data_values["alpha"] = [11.0, 21.0, 31.0, 41.0] data_values["beta"] = [12.0, 22.0, 32.0, 42.0] data_values["gamma"] = [13.0, 23.0, 33.0, 43.0] arecords = numpy.empty(2, dtype=type3) brecords = numpy.empty(2, dtype=type3) print print "Case A:" print print arecords, ': arecords:'.upper() print data_values, ': data_values'.upper() print arecords["data"][1].shape, ': arecords["data"][1].shape'.upper() print arecords[1]["data"].shape, ': arecords[1]["data"].shape'.upper() print data_values.shape, ': data_values.shape' print arecords["data"][1].dtype, ': arecords["data"][1].dtype'.upper() print arecords[1]["data"].dtype, ': arecords[1]["data"].dtype'.upper() print data_values.dtype, ': data_values.dtype'.upper() arecords["header"][1] = header_values arecords["data"][1] = data_values print arecords, ': arecords:'.upper() print print "Case B:" print print brecords, ': brecords:'.upper() print data_values, ': data_values'.upper() print brecords["data"][1].shape, ': brecords["data"][1].shape'.upper() print brecords[1]["data"].shape, ': brecords[1]["data"].shape'.upper() print data_values.shape, ': data_values.shape' print brecords["data"][1].dtype, ': brecords["data"][1].dtype'.upper() print brecords[1]["data"].dtype, ': brecords[1]["data"].dtype'.upper() print data_values.dtype, ': data_values.dtype'.upper() brecords[1]["header"] = header_values brecords[1]["data"] = data_values print brecords, ': brecords:'.upper()

1 0

Re: [Numpy-discussion] Numpy Trac site redirecting in a loop?
by Ken Basye 08 May '09

08 May '09

Hi Pauli, and thanks. I tried clearing the cache (no luck), and then logging in from a completely different machine with a completely re-initialized browser. I can get to the site, but as soon as I try to log in, the same thing (redirect loop) happens. I should point out that just before this happened I had the site reset my password. The URL it seems to be trying to get to after the login dialog is dismissed is: http://projects.scipy.org/numpy/prefs/account Back on the original machine, I was able to get the main page loading again by clearing all the cookies and stored passwords from scipy.org, but the same thing happened when I tried to log into the site. Thanks again, Ken > > Just now when I tried to access http://projects.scipy.org/numpy, > > Firefox complains (after a bit) that there's a redirect loop. Is anyone > > else seeing this problem, and/or does anyone know how I can fix my side > > of things to get me to the site? I don't see this. Can you try emptying your browser's cache? -- Pauli Virtanen Wed, 15 Apr 2009 17:21:59 -0400, Ken Basye wrote:

4 6

David does C code coverage!
by Stéfan van der Walt 08 May '09

08 May '09

Hi all, David Cournapeau got gcov working with NumPy! Well done, David! http://cournape.wordpress.com/2009/05/08/first-steps-toward-c-code-coverage… Regards Stéfan

2 2

efficient 3d histogram creation
by Chris Colbert 08 May '09

08 May '09

in my endless pursuit of perfomance, i'm searching for a quick way to create a 3d histogram from a 3d rgb image. Here is what I have so far for a (16,16,16) 3d histogram: def hist3d(imgarray): histarray = N.zeros((16, 16, 16)) temp = imgarray.copy() (i, j) = imgarray.shape[0:2] temp = (temp - temp % 16) / 16 for a in range(i): for b in range(j): (b1, b2, b3) = temp[a, b, :] histarray[b1, b2, b3] += 1 return histarray this works, but takes about 4 seconds for a 640x480 image. I tried doing the inverse of my previous post, namely replacing the nested for loop with: histarray[temp[:,:,0], temp[:,:,1], temp[:,:,2]] += 1 but that doesn't work for whatever reason. It gives me number, but they're incorrect. Any ideas? Chris

6 21