MemoryError using NumPy

[Gustavo Cordova]

> 
> OK, so the Python process keeps the memory. Does the memory that the
> Python process holds get fragmented? If so, same problem. If not, then
> what is going on?
> 

Whenever there isn't enough memory (reserved by the Python
runtime) it simply asks for another block, at least as
large as is needed.

Linux's VM system is responsible for that request. If there's
enough virtual memory, then it answers positivly. As the memory
is being physically used, it's being swapped in and out of ram.

So. If you need 1Gb of ram for an allocation, you could make
a 2Gb swap partition. Most probably, the memory request would
succeed, if there's enough VM free.

> Also, a few years back there was a discussion on the MATLAB newsgroup
> about a similar problem, and The Mathworks claimed that it was not
> possible to return memory to the os once a process has used 
> it. This was
> the case on Linux and I'm pretty sure, all the other *nixs that they
> supported. Perhaps it's a library issue, but is there a 
> different one(s)
> that we all should be using to compile Python?
> 
> -Chris
> 

I wouldn't really know. Allocated memory becomes fragmented,
so maybe it isn't possible to return it to the OS, since not
the whole page/block/whatever is unused.

On the other hand, maybe the Python runtime can be smart
enough to periodically shrink it's allocated memory size,
by freeing the top area of the heap whenever possible. But,
since it isn't possible to relocate an object once it's
been instantiated, it'd be up to luck to wait for a moment
when the heap has enough free memory on the top to free.

Is it possible to relocate an active object?

If not, what would be necessary to make that happen?

If it can be done, then a garbage-collector can just
pack everything at the bottom of the heap and free
all allocated memory above the new heap-top.

Anyhow.

That's just my 2 ct.

-gustavo