[Numpy-discussion] Memory mapping and NPZ files

Mathieu Dubois mathieu.dubois at icm-institute.org
Wed Dec 9 09:51:55 EST 2015


Dear all,

If I am correct, using mmap_mode with Npz files has no effect i.e.:
f = np.load("data.npz", mmap_mode="r")
X = f['X']
will load all the data in memory.

Can somebody confirm that?

If I'm correct, the mmap_mode argument could be passed to the NpzFile 
class which could in turn perform the correct operation. One way to 
handle that would be to use the ZipFile.extract method to write the Npy 
file on disk and then load it with numpy.load with the mmap_mode 
argument. Note that the user will have to remove the file to reclaim 
disk space (I guess that's OK).

One problem that could arise is that the extracted Npy file can be large 
(it's the purpose of using memory mapping) and therefore it may be 
useful to offer some control on where this file is extracted (for 
instance /tmp can be too small to extract the file here). numpy.load 
could offer a new option for that (passed to ZipFile.extract).

Does it make sense?

Thanks in advance,
Mathieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20151209/8ad09e6f/attachment.html>


More information about the NumPy-Discussion mailing list