[Numpy-discussion] numpy array sharing between processes?

[Charles R Harris]

On 5/12/07, Andrew Straw <strawman at astraw.com> wrote:
>
> Charles R Harris wrote:
> >
> >     I'll pitch in a few donuts (and my eternal gratitude) for an
> >     example of
> >     shared memory use using numpy arrays that is cross platform, or at
> >     least
> >     works in linux, mac, and windows.
> >
> >
> > I wonder if you could mmap a file and use it as common memory?
> Yes, that's the basic idea. Now for the example that works on those
> platforms...
> > Forking in python under linux leads to copies because anything that
> > accesses an object changes its reference count.
> I'm not sure what you're trying to say here. If it's shared memory, it's
> not copied


Ah, but the child receives a *copy* of the parent memory, right down to the
heap and stack. In linux this is implemented as copy-on-write for
efficiency, but that won't do much good, for as soon as you touch a python
object it's reference count changes and the copy is underway. The one thing
you do get is a copy of any open file descriptors and shared memory handles,
so you can exchange data that way if you are careful to synchronize your
accesses. Pipes are like that. However, when the child exits the files are
closed, so some care is needed. Share memory, however, persists.

Shared memory sounds like what you want. There are python modules shm and
shm_wrapper for unix systems, but windows has that capacity also so the
functionality could probably be extended. I've never used the modules, so
caveat emptor. I don't quite see how one would handle reference counting to
shared memory, so you probably have to explicitly destroy it when you are
done. Hmmm, what would be nice is numpy arrays created out of shared memory.

Boost seems to have stuff for shared memory, so that might be another
starting point. Some info on the python modules is at
http://nikitathespider.com/python/shm/

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070512/97e92589/attachment.html>