MemoryError using NumPy

[Chris Barker]

Emile van Sebille wrote:

> Numeric.sum(Numeric.take(myDataArray, indices)).astype(Numeric.Float64)

> This causes a temporary jump in memory usage (from ~800Mb to ~1.2Gb)
> before settling down again, but it delivers the speed I'm looking for
> (~7 secs vs ~30 summing in loop).  All starts well, but soon I get
> MemoryError "can't allocate memory for array" only when indices is
> large.  I suspect that memory has fragmented such that a sufficiently
> large contiguous block is not available to set up the array.  Is this
> the likely cause, and if so is there a way to un-fragment memory?  Or
> might there be an alternative to Numeric.take that gets to the same
> place?

I think you have two functions in one line that create a copy (take()
and astype()). I have never gotten an answer as to when CPython frees
the memory for temporary variables in one statement, but if you can
delete myDataArray before calling astype(), you might save something
there.

If not, I'm interested in finding oput what the options are for memory
defragmenting...

I'd also be interested in what happens under Linux, I still have a much
better feeling about the robustness of Linux over Win2k, I'd like to see
if they are different in this respect.

-Chris






-- 
Christopher Barker,
Ph.D.                                                           
ChrisHBarker at attbi.net                ---           ---           ---
                                     ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------