Hello- I have a function to generate a multi-dimensional array, which then gets summed over one axis. The problem is that the dimensions are large, and I run out of memory when I create the entire array, so I'm trying to do the sum *within* the function. Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT; functions f1(x,y,z,t), f2(y,z,t); want to calculate f1*f2 and sum over t to get out[x,y,z]. With loops, I could do it like-- out = zeros((numX,numY,numZ)) for x in range(numX): for y in range(numY): for z in range(numZ): for t in range(numT): tempval = f1(x,y,z,t) * f2(y,z,t) out[x,y,z] = out[x,y,z] + tempval With numarray, if I had enough memory, I could just do-- temp1 = fromfunction(f1,(numX,numY,numZ,numT)) temp2 = resize(fromfunction(f2,(numY,numZ,numT)),(numX,numY,numZ,numT)) out = sum(temp1 * temp2, axis = 3) Instead, I'm trying to do something like-- def f3(x,y,z): for t in range(numT): tempval = f1(x,y,z,t) * f2(y,z,t) outval = sum(tempval,axis = 3) return outval out = fromfunction(f3,(numX,numY,numZ)) I've been trying various slicing and indexing, but I can't seem to get that *extra* dimension within the 3-D function. I've scoured the documentation and list archives, but haven't found exactly what I need. Any suggestions? Am I stuck with generating the entire 4-D array? Thanks in advance, Jim Cser
Jim Cser wrote:
Instead, I'm trying to do something like-- def f3(x,y,z): for t in range(numT): tempval = f1(x,y,z,t) * f2(y,z,t)
outval = sum(tempval,axis = 3) return outval
Perhaps I'm confused but it seems to me the for loop is wrong. Don't you want something like: def f3(x,y,z): tempval = 0.*x for t in range(numT): tempval += f1(x,y,z,t) * f2(y,z,t) return tempval Instead? Not that I've tried this, but so long as the function will work given 3-d arrays for x,y,z, it should. (though the dependence on numT as a global is a bit worrisome) Perry GReenfield
Perry Greenfield wrote:
Jim Cser wrote:
Instead, I'm trying to do something like-- def f3(x,y,z): for t in range(numT): tempval = f1(x,y,z,t) * f2(y,z,t)
outval = sum(tempval,axis = 3) return outval
Perhaps I'm confused but it seems to me the for loop is wrong.
Don't you want something like:
def f3(x,y,z): tempval = 0.*x for t in range(numT): tempval += f1(x,y,z,t) * f2(y,z,t) return tempval
Instead?
Not that I've tried this, but so long as the function will work given 3-d arrays for x,y,z, it should. (though the dependence on numT as a global is a bit worrisome)
Perry GReenfield
The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, thanks. Unfortunately, iterating over t is still extremely slow. Ideally, there would be a way to use fromfunction() with slices as arguments. -Jim
Jim Cser wrote:
The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, thanks. Unfortunately, iterating over t is still extremely slow. Ideally, there would be a way to use fromfunction() with slices as arguments.
If the other 3 dimensions are small (e.g., each time sample involves an array whose size is less than 1000 elements total), then the slowness is due to the small arrays. In that event you may want iterate over a different dimension than time, or construct the for loop to iterate over ranges of times rather than one at a time. Perry
Jim Cser wrote:
The loop was indeed wrong, and "tempval = 0.*x" was the trick I needed, thanks. Unfortunately, iterating over t is still extremely slow. Ideally, there would be a way to use fromfunction() with slices as arguments.
If the other 3 dimensions are small (e.g., each time sample involves an array whose size is less than 1000 elements total), then the slowness is due to the small arrays. In that event you may want iterate over a different dimension than time, or construct the for loop to iterate over ranges of times rather than one at a time.
Perry
I was just using T as a sample name, so I am not actually doing a time series of anything. My dimesions are on the order of numT = 500, numX = 500, numY = 100, numZ = 10, so smallness is not the problem. Unfortunately, I don't need to sum over Z, the short dimension. If my system didn't hang, I would have done it the easier way in the first place. Perhaps I need to learn more about memory management-- I'm sure that many Python users are crunching bigger even arrays than this. For what it's worth, I'm using Python 2.3 / Numarray 1.0 with PythonWin on an XP box, 512M RAM. -Jim
Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT; functions f1(x,y,z,t), f2(y,z,t); want to calculate f1*f2 and sum over t to get out[x,y,z].
Cobbling together a number of suggestions, what finally worked was-- def f3(x, y, z, t_range=arange(numT)): tempval = 0.* x for t in t_range: tempval += f1(x,y,z,t) + f2(y,z,t) return tempval out = fromfunction(f3,(numX,numY,numZ,1)) I couldn't quite get sum() to work inside the function, but this is definitely good enough for now. Thanks to all for your help. -Jim Cser
participants (3)
-
Jim Cser -
jimcserï¼ pacifier.com
-
Perry Greenfield