Le 11/12/2015 11:22, Sturla Molden a écrit :

Mathieu Dubois <mathieu.dubois@icm-institute.org> wrote:

...

The point is precisely that, you can't do memory mapping with Npz files (while it works with Npy files). The operating system can memory map any file. But as npz-files are compressed, you will need to uncompress the contents in your memory mapping to make sense of it. We agree on that. The goal is to be able to create a np.memmap array from an Npz file.

I would suggest you use PyTables instead of npz-files. It allows on the fly compression and uncompression (via blosc) and will probably do what you want. Yes I know I can use other solutions. The point is that np.load silently ignore the mmap option so I wanted to discuss ways to improve this.

Mathieu

Le 11/12/2015 11:22, Sturla Molden a écrit :

Mathieu Dubois <mathieu.dubois@icm-institute.org> wrote:

...

The point is precisely that, you can't do memory mapping with Npz files (while it works with Npy files). The operating system can memory map any file. But as npz-files are compressed, you will need to uncompress the contents in your memory mapping to make sense of it. We agree on that. The goal is to be able to create a np.memmap array from an Npz file.

I would suggest you use PyTables instead of npz-files. It allows on the fly compression and uncompression (via blosc) and will probably do what you want. Yes I know I can use other solutions. The point is that np.load silently ignore the mmap option so I wanted to discuss ways to improve this.

Mathieu