[Numpy-discussion] multiprocessing shared arrays and numpy
gael.varoquaux at normalesup.org
Thu Mar 11 04:36:42 EST 2010
On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote:
> As far as I know, memmap files (or better, the underlying OS) *use* all
> available RAM for loading data until RAM is exhausted and then start to use
> SWAP, so the "memory pressure" is still there. But I may be wrong...
I believe that your above assertion is 'half' right. First I think that
it is not SWAP that the memapped file uses, but the original disk space,
thus you avoid running out of SWAP. Second, if you open several times the
same data without memmapping, I believe that it will be duplicated in
memory. On the other hand, when you memapping, it is not duplicated, thus
if you are running several processing jobs on the same data, you save
memory. I am very much in this case.
More information about the NumPy-Discussion