[Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation

[Antony Lee]

2013/9/19 Robert Kern <robert.kern at gmail.com>

> On Thu, Sep 19, 2013 at 5:58 PM, Antony Lee <antony.lee at berkeley.edu>
> wrote:
> >
> > Henry: thanks a lot, that would be very appreciated regardless of
> whether I end up using it in this specific project or not.
> > Other replies below.
> >
> > Antony
> >
> > 2013/9/19 Robert Kern <robert.kern at gmail.com>
> >>
> >> On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee <antony.lee at berkeley.edu>
> wrote:
> >> >
> >> > Thanks, I didn't know that multiprocessing Managers could be used
> with processes not started by multiprocessing itself...  I will give them a
> try.
> >> > I just need to compute FFTs, but speed is a real issue for me (I am
> using the results for real-time feedback).
> >>
> >> I am pretty sure that the overhead of communicating a large array from
> one process to another will vastly overwhelm any speed gains you get by
> using pyFFTW over numpy.fft.
> >
> > I would have hoped that the large arrays are simply written (from the
> beginning) to shared memory (what multiprocessing.sharedctypes.Array seems
> to do(?)) and that interprocess communication would be cheap enough (but
> what do I know about that).
>
> It certainly won't be automatic just by passing a numpy array to the
> manager. You will have to manually create the shared memory, pass its
> handle to the other process, and copy into it. But even the copy of the
> array may overwhelm the speed gains between PyFFTW and numpy.fft. If you
> can set it up such that the subprocess owns the shared memory for both
> input and output and the GUI process always writes into the input shared
> array directly and reads out the output shared array, then might work out
> okay. This works well when the inputs/outputs are always the same size.
>
The arrays would always be the same size, and there is no array copy
involved, as (I think that) I can have the C dll directly write whatever
data needs to be analyzed to the shared memory array -- basically what
you're suggesting.

>
> --
> Robert Kern
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130919/e31f8e70/attachment.html>