[Numpy-discussion] How to limit the numpy.memmap's RAM usage?
braingateway
braingateway at gmail.com
Sat Oct 23 06:40:11 EDT 2010
Hi everyone,
I noticed the numpy.memmap using RAM to buffer data from memmap files.
If I get a 100GB array in a memmap file and process it block by block,
the RAM usage is going to increasing with the process running until
there is no available space in RAM (4GB), even though the block size is
only 1MB.
for example:
####
a = numpy.memmap(‘a.bin’, dtype='float64', mode='r')
blocklen=1e5
b=npy.zeros((len(a)/blocklen,))
for i in range(0,len(a)/blocklen):
b[i]=npy.mean(a[i*blocklen:(i+1)*blocklen])
####
Is there any way to restrict the memory usage in numpy.memmap?
LittleBigBrain
More information about the NumPy-Discussion
mailing list