[Numpy-discussion] np.array, copy=False and memmap

Allan Haldane allanhaldane at gmail.com
Thu Aug 10 12:27:57 EDT 2017


On 08/07/2017 05:01 PM, Nisoli Isaia wrote:
> Dear all,
> I have a question about the behaviour of
> 
> y = np.array(x, copy=False, dtype='float32')
> 
> when x is a memmap. If we check the memmap attribute of mmap
> 
> print "mmap attribute", y._mmap
> 
> numpy tells us that y is not a memmap.
> But the following code snippet crashes the python interpreter
> 
> # opens the memmap
> with open(filename,'r+b') as f:
>       mm = mmap.mmap(f.fileno(),0)
>       x = np.frombuffer(mm, dtype='float32')
> 
> # builds an array from the memmap, with the option copy=False
> y = np.array(x, copy=False, dtype='float32')
> print "before", y
> 
> # closes the file
> mm.close()
> print "after", y
> 
> In my code I use memmaps to share read-only objects when doing parallel
> processing
> and the behaviour of np.array, even if not consistent, it's desirable.
> I share scipy sparse matrices over many processes and if np.array would
> make a copy
> when dealing with memmaps this would force me to rewrite part of the sparse
> matrices
> code.
> Would it be possible in the future releases of numpy to have np.array
> check,
> if copy is false, if y is a memmap and in that case return a full memmap
> object
> instead of slicing it?

This does appear to be a bug in numpy or mmap.

Probably the solution isn't to make mmaps a special case, rather we
should fix a bug somewhere in the use of the PEP3118 interface.

I've opened an issue on github for your issue:
https://github.com/numpy/numpy/issues/9537

It seems to me that the "correct" behavior may be for it to me
impossible to close the memmap while pointers to it exist; this is the
behavior for `memoryview`s of mmaps. That is, your line `mm.close()`
shoud raise an error `BufferError: cannot close exported pointers exist`.


> Best wishes
> Isaia
> 
> P.S. A longer account of the issue may be found on my university blog
> http://www.im.ufrj.br/nisoli/blog/?p=131
> 
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
> 



More information about the NumPy-Discussion mailing list