[Numpy-discussion] multiprocessing shared arrays and numpy

[Francesc Alted]

A Sunday 07 March 2010 20:03:21 Gael Varoquaux escrigué:
> On Sun, Mar 07, 2010 at 07:00:03PM +0000, René Dudfield wrote:
> > 1. Mmap'd files are useful since you can reuse disk cache as program
> > memory.  So large files don't waste ram on the disk cache.
> 
> I second that. mmaping has worked very well for me for large datasets,
> especialy in the context of reducing memory pressure.

As far as I know, memmap files (or better, the underlying OS) *use* all 
available RAM for loading data until RAM is exhausted and then start to use 
SWAP, so the "memory pressure" is still there.  But I may be wrong...

-- 
Francesc Alted