I was trying to parallelize some algorithms and needed a writable
array shared between processes. It turned out to be quite simple and
gave a nice speed up almost linear in number of cores. Of course you
need to know what you are doing to avoid segfaults and such. But I
still think something like this should be included with NumPy for
This works by inheriting anonymous mmaped memory. Not sure if this
works on windows.
import numpy as np
import multiprocessing as mp
"""Shared writable array"""
def __new__(subtype, shape, interface=None):
size = np.prod(shape)
if interface == None:
buffer = mp.RawArray('d', size)
self = np.ndarray.__new__(subtype, shape, float, buffer)
class Dummy(object): pass
buffer = Dummy()
buffer.__array_interface__ = interface
a = np.asarray(buffer)
self = np.ndarray.__new__(subtype, shape=a.shape, buffer=a)
def __reduce_ex__(self, protocol):
return shared, (self.shape, self.__array_interface__)
return __reduce_ex__(self, 0)
Also see attached file for example usage.