[Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation

[Robert Kern]

On Thu, Sep 19, 2013 at 5:58 PM, Antony Lee <antony.lee at berkeley.edu> wrote:
>
> Henry: thanks a lot, that would be very appreciated regardless of whether
I end up using it in this specific project or not.
> Other replies below.
>
> Antony
>
> 2013/9/19 Robert Kern <robert.kern at gmail.com>
>>
>> On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee <antony.lee at berkeley.edu>
wrote:
>> >
>> > Thanks, I didn't know that multiprocessing Managers could be used with
processes not started by multiprocessing itself...  I will give them a try.
>> > I just need to compute FFTs, but speed is a real issue for me (I am
using the results for real-time feedback).
>>
>> I am pretty sure that the overhead of communicating a large array from
one process to another will vastly overwhelm any speed gains you get by
using pyFFTW over numpy.fft.
>
> I would have hoped that the large arrays are simply written (from the
beginning) to shared memory (what multiprocessing.sharedctypes.Array seems
to do(?)) and that interprocess communication would be cheap enough (but
what do I know about that).

It certainly won't be automatic just by passing a numpy array to the
manager. You will have to manually create the shared memory, pass its
handle to the other process, and copy into it. But even the copy of the
array may overwhelm the speed gains between PyFFTW and numpy.fft. If you
can set it up such that the subprocess owns the shared memory for both
input and output and the GUI process always writes into the input shared
array directly and reads out the output shared array, then might work out
okay. This works well when the inputs/outputs are always the same size.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130919/e2fe3648/attachment.html>