[Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation

Antony Lee antony.lee at berkeley.edu
Thu Sep 19 12:58:37 EDT 2013


Henry: thanks a lot, that would be very appreciated regardless of whether I
end up using it in this specific project or not.
Other replies below.

Antony

2013/9/19 Robert Kern <robert.kern at gmail.com>

> On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee <antony.lee at berkeley.edu>
> wrote:
> >
> > Thanks, I didn't know that multiprocessing Managers could be used with
> processes not started by multiprocessing itself...  I will give them a try.
> > I just need to compute FFTs, but speed is a real issue for me (I am
> using the results for real-time feedback).
>
> I am pretty sure that the overhead of communicating a large array from one
> process to another will vastly overwhelm any speed gains you get by using
> pyFFTW over numpy.fft.
>
I would have hoped that the large arrays are simply written (from the
beginning) to shared memory (what multiprocessing.sharedctypes.Array seems
to do(?)) and that interprocess communication would be cheap enough (but
what do I know about that).

>
>
> > To be honest I don't know yet if the FFTs are going to be the limiting
> step but I thought I may as well give pyFFTW a try and ran into that
> issue...
>
> In that case, thinking about multiprocessing or even pyFFTW is far too
> premature. Implement your code with numpy.fft and see what performance you
> actually get.
>
There is another (and, in fact, main) reason for me to use multiprocessing:
the main app runs a GUI and running the data analysis in the same process
just makes it painfully slow (I have tried that).  Instead, running the
data analysis in a separate process keeps the GUI responsive.  Now whether
the data analysis process should use numpy.fft or pyFFTW is a separate
question; I realize that the gains from pyFFTW may probably be negligible
compared to the other costs (... including the costs of tweaking
multiprocessing beyond its specifications) but I was just giving it a try
when I ran into the issue and was just puzzled by the error message I had
never seen before.

> --
> Robert Kern
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130919/fd7804f6/attachment.html>


More information about the NumPy-Discussion mailing list