[SciPy-user] shared memory machines

[Sturla Molden]

> On Mon, Feb 02, 2009 at 12:51:51AM -0600, Robert Kern wrote:

> Well, you know way more than I do about this. But I fear I am
> miss-understanding something. Does what you are saying means that an
> 'empty_shmem', that would create a multiprocessing Array, and expose it
> as a numpy array, is bound to fail under windows?

Linux:

You can create shared memory using BSD mmap or System V IPC.
Multiprocessing does the former. Shared memory created via BSD mmap is
"unnamed". Thus, it has to be created in the parent prior prior to the
call to fork(), otherwise the child cannot get access to it. That is why
mp.Array must be created prior to mp.Process (the latter calls os.fork).


Windows:

There is no fork(). Shared memory can be named or unnamed. In the second
case, it is passed to the spawned process via handle inheritance. This is
what multiprocessing does. Again, the consequence is that it must med
created prior to the creation of mp.Process. In this case it must actually
be passed as an argument to to mp.Process when it is instantiated.

However:

If we had an ndarray that used named shared memory as buffer, it would be
more convinient on Windows and Linux alike. Any process can map this
segment if it knows its name. It would only pickle the name of the shared
segment (as well as dtype and shape), and could thus be messaged between
processes using mp.Queue. Currently we can only send copies of private
memory arrays via mp.Queue.


> My experiments seem to
> show that this works under Linux, and this would be a very simple way of
> doing shared memory. We could have a numpy.multiprocessing, with all kind
> of constructors for arrays (empty, zeros, ones, *_like, and maybe
> 'array') that would be shared between process.
>
> Am I out of my mind, and will this fail utterly?

It will work. But we should use named shared memory (which requires some C
or Cython coding), not BSD mmap as mp.Array currently does. Also we must
override how ndarrays are pickled.


Sturla Molden