[Numpy-discussion] resizeable arrays using shared memory?

[Elliot Hallmark]

Hi all,

I have a program that uses resize-able arrays.  I already over-provision
the arrays and use slices, but every now and then the data outgrows that
array and it needs to be resized.

Now, I would like to have these arrays shared between processes spawned via
multiprocessing (for fast interprocess communication purposes, not for
parallelizing work on an array).  I don't care about mapping to a file on
disk, and I don't want disk I/O happening.  I don't care (really) about
data being copied in memory on resize.  I *do* want the array to be resized
"in place", so that the child processes can still access the arrays from
the object they were initialized with.


I can share arrays easily using arrays that are backed by memmap.  Ie:

    ```
    #Source: http://github.com/rainwoodman/sharedmem


    class anonymousmemmap(numpy.memmap):
        def __new__(subtype, shape, dtype=numpy.uint8, order='C'):

            descr = numpy.dtype(dtype)
            _dbytes = descr.itemsize

            shape = numpy.atleast_1d(shape)
            size = 1
            for k in shape:
                size *= k

            bytes = int(size*_dbytes)

            if bytes > 0:
                mm = mmap.mmap(-1,bytes)
            else:
                mm = numpy.empty(0, dtype=descr)
            self = numpy.ndarray.__new__(subtype, shape, dtype=descr,
buffer=mm, order=order)
            self._mmap = mm
            return self

        def __array_wrap__(self, outarr, context=None):
            return numpy.ndarray.__array_wrap__(self.view(numpy.ndarray),
outarr, context)
    ```

This cannot be resized because it does not own it's own data (ValueError:
cannot resize this array: it does not own its data).  (numpy.memmap has
this same issue [0], even if I set refcheck to False and even though the
docs say otherwise [1]).

arr._mmap.resize(x) fails because it is annonymous (error: [Errno 9] Bad
file descriptor).  If I create a file and use that fileno to create the
memmap, then I can resize `arr._mmap` but the array itself is not resized.

Is there a way to accomplish what I want?  Or, do I just need to figure out
a way to communicate new arrays to the child processes?

Thanks,
  Elliot

[0] https://github.com/numpy/numpy/issues/4198.

[1]
http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.resize.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160206/6466d5dc/attachment.html>