On Dec 12, 2015 10:53 AM, "Mathieu Dubois" <mathieu.dubois@icm-institute.org> wrote:
>
> Le 11/12/2015 11:22, Sturla Molden a écrit :
>>
>> Mathieu Dubois <mathieu.dubois@icm-institute.org> wrote:
>>
>>> The point is precisely that, you can't do memory mapping with Npz files
>>> (while it works with Npy files).
>>
>> The operating system can memory map any file. But as npz-files are
>> compressed, you will need to uncompress the contents in your memory mapping
>> to make sense of it.
>
> We agree on that. The goal is to be able to create a np.memmap array from an Npz file.
>
>
>> I would suggest you use PyTables instead of npz-files.
>> It allows on the fly compression and uncompression (via blosc) and will
>> probably do what you want.
>
> Yes I know I can use other solutions. The point is that np.load silently ignore the mmap option so I wanted to discuss ways to improve this.

I can see a good argument for transitioning to a rule where mmap=False doesn't mmap, mmap=True mmaps if the file is uncompressed and raises an error for compressed files, and mmap="if-possible" gives the current behavior.

(It's even possible that the current code would already accept "if-possible" as a alias for True, which would make the transition easier.)

Or maybe "never"/"always"/"if-possible" would be better for type consistency reasons, while deprecating the use of bools altogether. But this transition might be a bit more of a hassle, since these definitely won't work on older numpy's.

Silently creating a massive temporary file doesn't seem like a great idea to me in any case. Creating a temporary file + mmaping it is essentially equivalent to just loading the data into swappable RAM, except that the swap case is guaranteed not to accidentally leave a massive temp file lying around afterwards.

-n