March 11, 2010
1:04 a.m.
A Sunday 07 March 2010 20:03:21 Gael Varoquaux escrigué:
On Sun, Mar 07, 2010 at 07:00:03PM +0000, René Dudfield wrote:
1. Mmap'd files are useful since you can reuse disk cache as program memory. So large files don't waste ram on the disk cache.
I second that. mmaping has worked very well for me for large datasets, especialy in the context of reducing memory pressure.
As far as I know, memmap files (or better, the underlying OS) *use* all available RAM for loading data until RAM is exhausted and then start to use SWAP, so the "memory pressure" is still there. But I may be wrong... -- Francesc Alted