Numeric -> numarray gives MemoryError
![](https://secure.gravatar.com/avatar/7662cef50830e9bfb1c82046ca12763b.jpg?s=120&d=mm&r=g)
Hi, I've started to move some of my Numeric code to numarray 0.5 and have run into a problem. This is a relatively large optimization problem that requires finite difference estimation of gradients. The implementation uses either Numeric/LinearAlgebra/RandomArray or numarray/LinearAlgebra2/RandomArray2; these are the only extensions used. The code itself is fairly straightforward and ports from Numeric to numarray with no modifications. Typical run times are on the order of hours with Numeric. When using Numeric, I am able to repeat multiple optimization runs (> 10) with no problems. When the same code is run with numarray instead, I get this error part of the way through the first run: Traceback (most recent call last): File "runFD2.py", line 81, in ? W1, W2 = estimFD2(inputs, outputs, Nh, Ne, alpha) File "optimFD2.py", line 171, in estimFD2 gradW1, gradW2 = dEdW(inputs, outputs, MSE, alpha, W1, W2, f1, f2) File "optimFD2.py", line 83, in dEdW y_plus = evalFD2(inputs, W1_plus, W2, f1, f2) File "optimFD2.py", line 63, in evalFD2 h1 = dot(inputs, W1) File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 940, in dot return innerproduct(a, swapaxes(inputarray(b), -1, -2)) File "F:\Python22\Lib\site-packages\numarray\ufunc.py", line 1866, in innerproduct a = a.astype(rtype) File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 478, in astype return self.copy() File "F:\Python22\Lib\site-packages\numarray\numarray.py", line 553, in copy c = ndarray.NDArray.copy(self) File "F:\Python22\Lib\site-packages\numarray\ndarray.py", line 571, in copy arr._data = memory.new_memory(arr._itemsize * arr.nelements()) MemoryError
This is a little frustrating, because numarray is clearly much faster. Checking the results along the way shows that numarray gives the same outputs as Numeric, so it doesn't appear to be a porting issue. The datasets involved are large (~10000x20). Initial memory usage with Numeric and numarray is similar, and is well below the machine's limit. The code is running on a Windows 2000 machine using the binary release of numarray. What might be causing this problem? Should numarray be less tolerant than Numeric in a situation like this? Thanks. Chris
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Tue, 2003-05-27 at 11:52, christopheranderson@shaw.ca wrote:
Well... the faster part is good. :)
What might be causing this problem?
Should numarray be less tolerant than Numeric in a situation like
It sounds to me like a memory leak (reference counting error) in numarray. Did you run the same code with older versions of numarray, or did you start with numarray-0.5? How long does it take to fail? Can you mail me your code (or better, the smallest derivative of it which reproduces the problem)? this? No, it's just newer and less mature.
Thanks.
You're welcome! Todd
-- Todd Miller jmiller@stsci.edu STSCI / ESS / SSB
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Tue, 2003-05-27 at 11:52, christopheranderson@shaw.ca wrote:
Well... the faster part is good. :)
What might be causing this problem?
Should numarray be less tolerant than Numeric in a situation like
It sounds to me like a memory leak (reference counting error) in numarray. Did you run the same code with older versions of numarray, or did you start with numarray-0.5? How long does it take to fail? Can you mail me your code (or better, the smallest derivative of it which reproduces the problem)? this? No, it's just newer and less mature.
Thanks.
You're welcome! Todd
-- Todd Miller jmiller@stsci.edu STSCI / ESS / SSB
participants (2)
-
christopheranderson@shaw.ca
-
Todd Miller