MemoryError using NumPy

[Chris Barker]

Gustavo Cordova wrote:
> 
> > > I suspect that memory has fragmented such that a sufficiently
> > > large contiguous block is not available to set up the array.
> > > Is this the likely cause, and if so is there a way to un-fragment
> > > memory? Or might there be an alternative to Numeric.take that
> > > gets to the same place?
> 
> Er... This isn't an Amiga, you know. :-)
> 
> Really. Under UNIX, or anything "Unixy", where there's
> a memory management unit on the CPU and a virtual memory
> subsystem in the OS, memory fragmentation really doesn't
> mean much, because the OS remaps the memory layout
> so that each and every process receives (virtually) the
> same memory space, in a contiguous manner.
> 
> I remember having to reboot my Ami because I didn't have
> enought ram left, when in fact it was all fragmented.
> Virtual memory would have been good then.  :-)
> 
> (regardless, they rocked)
> 
> > I think you have two functions in one line that create a copy (take()
> > and astype()). I have never gotten an answer as to when CPython frees
> > the memory for temporary variables in one statement, but if you can
> > delete myDataArray before calling astype(), you might save something
> > there.
> 
> The problem is the C library, seems like it doesn't ever return
> allocated-and-then-freed memory to the OS, so, CPython never
> returns unused memory to the OS neither.

OK, so the Python process keeps the memory. Does the memory that the
Python process holds get fragmented? If so, same problem. If not, then
what is going on?

Also, a few years back there was a discussion on the MATLAB newsgroup
about a similar problem, and The Mathworks claimed that it was not
possible to return memory to the os once a process has used it. This was
the case on Linux and I'm pretty sure, all the other *nixs that they
supported. Perhaps it's a library issue, but is there a different one(s)
that we all should be using to compile Python?

-Chris




-- 
Christopher Barker,
Ph.D.                                                           
ChrisHBarker at attbi.net                ---           ---           ---
                                     ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------