[Numpy-discussion] numpy array sharing between processes?

[Charles R Harris]

On 5/12/07, Andrew Straw <strawman at astraw.com> wrote:
>
> Ray Schumacher wrote:
> >
> > After Googling for examples on this, in the Cookbook
> > http://www.scipy.org/Cookbook/Multithreading
> > MPI and POSH (dead?), I don't think I know the answer...
> > We have a data collection app running on dual core processors; I start
> > one thread collecting/writing new data directly into a numpy circular
> > buffer, another thread does correlation on the newest data and
> > occasional FFTs, both now use 50% CPU, total.
> > The threads never need to access the same buffer slices.
> > I'd prefer to have two processes, forking the FFT process off and
> > utilizing the second core. The processes would only need to share two
> > variables (buffer insert position and a short_integer result from the
> > FFT process, each process would only read or write), in addition to
> > the numpy array itself.
> >
> > Should I pass the numpy address to the second process and just create
> > an identical array there, as in
> >
> http://projects.scipy.org/pipermail/numpy-discussion/2006-October/023647.html
> > ?
> >
> > Use a file-like object to share the other variables? mmap?
> > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413807
> >
> > I also thought ctypes
> > ctypes.string_at(address[, size])
> > might do both easily enough, although would mean a copy. We already
> > use it for the collection thread.
> > Does anyone have a lightweight solution to this relatively simple sort
> > of problem?
> I'll pitch in a few donuts (and my eternal gratitude) for an example of
> shared memory use using numpy arrays that is cross platform, or at least
> works in linux, mac, and windows.


I wonder if you could mmap a file and use it as common memory? Forking in
python under linux leads to copies because anything that accesses an object
changes its reference count. Pipes are easy and could be used for
synchronization. Would python threading work for you? It might be the
easiest way to have the fft going on while doing something else.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20070512/49cdf7c1/attachment.html>