Mailman 3 May 2003 - NumPy-Discussion

fftshift
by Cliff Martin 29 May '03

29 May '03

All, First I'd like to thank Paul Dubois, Todd Miller and Perry Greenfield for the help you gave me on setting defined values based on index values in an array. Your fixes worked and let me get done what was needed. This leads me to my second problem. In Matlab there is a function called fftshift that lets one shift the fft values for more easy visualization. Numeric doesn't seem to have a function like that explicitly although I suppose one could use the flip function in the MatPy module. Has anyone implemented something like the fftshift function or could you suggest the best way to do this? Thanks for your help. Oh I'm doing this on a 2D array. Cliff Martin

2 1

Re: Numeric -> numarray gives MemoryError
by christopheranderson＠shaw.ca 27 May '03

27 May '03

Todd, thanks very much for the quick reply. > >Well... the faster part is good. :) Yes indeed. > > > What might be causing this problem? > >It sounds to me like a memory leak (reference counting error) in >numarray. > >Did you run the same code with older versions of numarray, or did you >start with numarray-0.5? > >How long does it take to fail? This was my first stab at using numarray, so I haven't tried with prior versions. Not that this means anything, but it usually dies around 500 or so iterations through the optimization routine. For comparison, Numeric will cheerfully run for several thousands of iterations, multiple times, without complaint. >Can you mail me your code (or better, the smallest derivative of it >which reproduces the problem)? This might be a little complicated. I'll try stripping things down to isolate the problem. Chris

1 0

Numeric -> numarray gives MemoryError
by christopheranderson＠shaw.ca 27 May '03

27 May '03

Hi, I've started to move some of my Numeric code to numarray 0.5 and have run into a problem. This is a relatively large optimization problem that requires finite difference estimation of gradients. The implementation uses either Numeric/LinearAlgebra/RandomArray or numarray/LinearAlgebra2/RandomArray2; these are the only extensions used. The code itself is fairly straightforward and ports from Numeric to numarray with no modifications. Typical run times are on the order of hours with Numeric. When using Numeric, I am able to repeat multiple optimization runs (> 10) with no problems. When the same code is run with numarray instead, I get this error part of the way through the first run: Traceback (most recent call last): File "runFD2.py", line 81, in ? W1, W2 = estimFD2(inputs, outputs, Nh, Ne, alpha) File "optimFD2.py", line 171, in estimFD2 gradW1, gradW2 = dEdW(inputs, outputs, MSE, alpha, W1, W2, f1, f2) File "optimFD2.py", line 83, in dEdW y_plus = evalFD2(inputs, W1_plus, W2, f1, f2) File "optimFD2.py", line 63, in evalFD2 h1 = dot(inputs, W1) File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 940, in dot return innerproduct(a, swapaxes(inputarray(b), -1, -2)) File "F:\Python22\Lib\site-packages\numarray\ufunc.py", line 1866, in innerproduct a = a.astype(rtype) File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 478, in astype return self.copy() File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 553, in copy c = ndarray.NDArray.copy(self) File "F:\Python22\Lib\site-packages\numarray\ndarray.py", line 571, in copy arr._data = memory.new_memory(arr._itemsize * arr.nelements()) MemoryError >>> This is a little frustrating, because numarray is clearly much faster. Checking the results along the way shows that numarray gives the same outputs as Numeric, so it doesn't appear to be a porting issue. The datasets involved are large (~10000x20). Initial memory usage with Numeric and numarray is similar, and is well below the machine's limit. The code is running on a Windows 2000 machine using the binary release of numarray. What might be causing this problem? Should numarray be less tolerant than Numeric in a situation like this? Thanks. Chris

2 1

faster way for calculating distance matrix...
by Kasper Souren 21 May '03

21 May '03

Hi, I was happy to see solutions coming up for Cliff's problems and it made me think: maybe I can mention my problem here as well. From an array of feature vectors I want to calculate it's distance matrix. Something like [1]. Currently it can take quite a while to calculate the stuff for a long array. Some questions: 1) Is there a smart speed up possible? Like, a way to avoid the double loop? It's no problem if this would lead to less generality (like the choice for a distance function). I know there is a little speed up to be gained by leaving out the array(f) thing, but that's not what I'm looking for. 2) Is it possible (in Numeric or numarray) to define a class DiagonalMatrix that at least saves half of the memory? 3) If 1) is not possible, what would be the way to go for speeding it up by writing it in C? weave because of its availability in scipy or would pyrex be more interesting, or are there even more options..? bye, Kasper [1] Example program: import Numeric def euclidean_dist(a, b): diff = a - b return Numeric.dot(diff, diff) def calc_dist_matrix(f, distance_function): W = Numeric.array(f) length = W.shape[0] S = Numeric.zeros((length, length)) * 1.0 for i in range(length): for j in range(i): S[j, i] = S[i, j] = distance_function(W[i], W[j]) return S print calc_dist_matrix(Numeric.sin(Numeric.arange(30)), euclidean_dist)

2 1

Finding indices and setting values
by Cliff Martin 21 May '03

21 May '03

Hi, I have a problem where I want to set values in a 2D array based on conditions I’ve established. The variables are the NA of the system, its wavelength and the correct units on the device I’m modeling. Using these variables I define a variable N. I set up a 512 by 512 array [x,y]. Then I set up a radius , r = x^2 +y^2 and ask it to give me all r’s(to the closest integer) <= N(the variable I’ve defined above). This gives all the index values in that radius and then I set all those locations to a value of 1. After this I do some FFT ‘s, etc. So how do I do this in Numerical Python without having to index through i,j steps which would be incredibly tedious. This works fairly well in MatLab but I want to port my program to Python(for lots of reasons). If you’d rather I write the MatLab code snippet I can do that. Thanks. Cliff Martin

4 4

Numarray 0.5
by Peter Verveer 20 May '03

20 May '03

Hi All, I found the following problems after testing my software with numarray 0.5: 1) Following works fine if both a and b are arrays: >>> a = array([2]) >>> b = array([1, 2]) >>> print a + b [3 4] However, if b is an python sequence: >>> a = array([2]) >>> b = [1, 2] >>> print a + b [3] Apparently broadcasting does not work with python sequeces anymore. This used to work fine in version 0.4. Is this a bug? 2) It is not possible to compare an array type object to the 'Any' object: >>> Float64 == Any Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.2/site-packages/numarray/numerictypes.py", line 112, in __cmp__ return (genericTypeRank.index(self.name) - ValueError: list.index(x): x not in list I am not sure if this is a bug, or intended behaviour, but the possibilty to compare an array type object to 'Any' would be very useful for me. 3) The NA_typeNoToTypeObject() function fails if it is called before any arrays are created. It looks to me as if the pNumType array in libnumarraymodule.c is not initialized until an array is created. I don't know if any other functions are affected in the same way. Could this be fixed? Cheers, Peter -- Dr. Peter J. Verveer Cell Biology and Cell Biophysics Programme EMBL Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. : +49 6221 387245 Fax : +49 6221 387242 Email: Peter.Verveer(a)embl-heidelberg.de

3 5

Re: change in semantics of arange?
by Todd Miller 16 May '03

16 May '03

On Fri, 2003-05-16 at 13:03, Paul Dubois wrote: > C:\numpy\numarray\Packages\MA2>python > Python 2.3b1 (#40, Apr 25 2003, 19:06:24) [MSC v.1200 32 bit (Intel)] on > win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numarray > >>> x=numarray.arange(5,3,2) > Traceback (most recent call last): > File "<stdin>", line 1, in ? > File "c:\python23\Lib\site-packages\numarray\numarray.py", line 924, in > arange > r = _fillarray(size, start, stride, type) > File "c:\python23\Lib\site-packages\numarray\numarray.py", line 120, in > _fillarray > outarr = NumArray((size,), outtype) > libnumarray.error: NA_updateDataPtr: error getting read buffer data ptr > >>> import Numeric > >>> Numeric.arange(5,3,2) > zeros((0,), 'l') > >>> > > Is this change intentional? No. It's bugs in arange and also the memory object which are creating a negative length buffer. The buffer API then reports a negative length which is fortuitously interpreted as an error. Ya gotta smile... It's fixed in CVS. I modified the memory module so that: >>> import memory >>> memory.new_memory(-10) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: new_memory: invalid region size: -10. And I modified arange so that negative sizes are clipped to 0. So now: >>> import numarray >>> numarray.arange(5,3,2) array([]) > -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

1 0

numarray repackaging, last call...
by Todd Miller 15 May '03

15 May '03

Based on the comments I have received, the numarray repackaging is shaping up as shown below: Here are the planned renamings for the numarray modules as we transition to a package: numarray --> numarray.numeric recarray --> numarray.records chararray --> numarray.strings ndarray --> numarray.generic * --> numarray.* Note that class and function names will remain unchanged, so recarray.RecArray will become numarray.records.RecArray. Here are the renamings for the current and planned add-on packages consolidated into a single add-ons distribution: LinearAlgrebra2 --> numarray.linear_algebra FFT2 --> numarray.fft RandomArray2 --> numarray.random_array Convolve --> numarray.convolve Convolve.Image --> numarray.image MA2 --> numarray.ma Future 3rd party packages that require numarray specific C extensions should locate themselves directly in the numarray package. Doing so makes it easy to figure out what to delete when extensions need to be rebuilt. If you don't want to, that's OK too. Barring a sudden burst of interest or superior name choice, I think the top level package name should just remain "numarray". Stub modules will be included in the first few repackaged releases to make it possible to import the primary modules (numarray, recarray, chararray, and ndarray) as you do with numarray-0.5. This will relax the requirement for synchronous change of numarray and software which uses numarray. The stub modules will be activated either through the creation of a .pth file or via additions to PYTHONPATH to make them visible. This is a backwards compatability mode only, not recommended usage following the repackaging. Thanks to everyone who responded. Final comments? -- Todd Miller jmiller(a)stsci.edu STSCI / ESS / SSB

1 0

Re: Numpy-discussion digest, Vol 1 #656 - 11 msgs
by verveer＠embl.de 15 May '03

15 May '03

> As has been mentioned before, we're planning to repackage numarray as > a package rather than as a collection of modules. We're soliciting > comments now because we only want to do this once. [some stuff deleted] > Convolve.Image --> numarray.Image Should this not become "numarray.Convolve.Image" instead of "numarray.Image"? Peter

2 1

Zeroing an existing array
by Andrew P. Lentvorski, Jr. 15 May '03

15 May '03

What is the official way to zero out an array in numarray/Numeric? While I can create a new array of all zeros and then assign it to the old variable, this is extremely wasteful of memory. Currently, I am just using: >>> import numarray >>> a = numarray.arange(6) >>> a array([0, 1, 2, 3, 4, 5]) >>> a[:] = 0.0 >>> a array([0, 0, 0, 0, 0, 0]) I looked through the manual for a function or array mathod which would accomplish the same thing, but I didn't find an obvious one. Did I miss something obvious? Thanks, -a

3 4