[Numpy-discussion] How to limit the numpy.memmap's RAM usage?

[Simon Lyngby Kokkendorff]

Hi List,

  I had similar problems on windows. I tried to use memmaps to buffer a
large amount of data and process it in chunks. But I found that whenever I
tried to do this, I always ended filling up RAM completely which led to
crashes of my python script with a MemoryError. This led me to consider,
actually from an advice via this list, the module h5py, which has a nice
numpy interface to the hdf5 file format. It seemed more clear to me with the
h5py-module, what was being buffered on disk and what was stored in RAM.

  Cheers,
  Simon


On Sun, Oct 24, 2010 at 2:15 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Sun, Oct 24, 2010 at 12:44 AM, braingateway <braingateway at gmail.com>
> wrote:
>
> >
> > I agree with you about the point of using memmap.
>
> > That is why the behavior
> > is so strange to me.
>
> I think it is expected. What kind of behavior were you expecting ? To
> be clear, if I have a lot of available ram, I expect memmap arrays to
> take almost all of it (virtual memroy ~ resident memory). Now, if at
> the same time, another process starts taking a lot of memory, I expect
> the OS to automatically lower resident memory for the process using
> memmap.
>
> I did a small experiment on mac os x, creating a giant mmap'd array in
> numpy, and at the same time running a small C program using mlock (to
> lock pages into physical memory). As soon as I lock a big area (where
> big means most of my physical ram), the python process dealing with
> the mmap area sees its resident memory decrease. As soon as I kill the
> C program locking the memory, the resident memory starts increasing
> again.
>
> cheers,
>
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20101025/d15676a5/attachment.html>