[Numpy-discussion] multiprocessing shared arrays and numpy

[Francesc Alted]

Yeah, 10% of improvement by using multi-cores is an expected figure for memory 
bound problems.  This is something people must know: if their computations are 
memory bound (and this is much more common that one may initially think), then 
they should not expect significant speed-ups on their parallel codes.

Thanks for sharing your experience anyway,
Francesc

A Thursday 04 March 2010 18:54:09 Nadav Horesh escrigué:
> I can not give a reliable answer yet, since I have some more improvement to
>  make. The application is an analysis of a stereoscopic-movie raw-data
>  recording (both channels are recorded in the same file). I treat the data
>  as a huge memory mapped file. The idea was to process each channel (left
>  and right) on a different core. Right now the application is IO bounded
>  since I do classical numpy operation, so each channel (which is handled as
>  one array) is scanned several time. The improvement now over a single
>  process is 10%, but I hope to achieve 10% ore after trivial optimizations.
> 
>  I used this application as an excuse to dive into multi-processing. I hope
>  that the code I posted here would help someone.
> 
>   Nadav.
> 
> 
> -----Original Message-----
> From: numpy-discussion-bounces at scipy.org on behalf of Francesc Alted
> Sent: Thu 04-Mar-10 15:12
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
> 
> What kind of calculations are you doing with this module?  Can you please
>  send some examples and the speed-ups you are getting?
> 
> Thanks,
> Francesc
> 
> A Thursday 04 March 2010 14:06:34 Nadav Horesh escrigué:
> > Extended module that I used for some useful work.
> > Comments:
> >   1. Sturla's module is better designed, but did not work with very large
> >  (although sub GB) arrays 2. Tested on 64 bit linux (amd64) +
> > python-2.6.4 + numpy-1.4.0
> >
> >   Nadav.
> >
> >
> > -----Original Message-----
> > From: numpy-discussion-bounces at scipy.org on behalf of Nadav Horesh
> > Sent: Thu 04-Mar-10 11:55
> > To: Discussion of Numerical Python
> > Subject: RE: [Numpy-discussion] multiprocessing shared arrays and numpy
> >
> > Maybe the attached file can help. Adpted and tested on amd64 linux
> >
> >   Nadav
> >
> >
> > -----Original Message-----
> > From: numpy-discussion-bounces at scipy.org on behalf of Nadav Horesh
> > Sent: Thu 04-Mar-10 10:54
> > To: Discussion of Numerical Python
> > Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
> >
> > There is a work by Sturla Molden: look for multiprocessing-tutorial.pdf
> > and sharedmem-feb13-2009.zip. The tutorial includes what is dropped in
> > the cookbook page. I am into the same issue and going to test it today.
> >
> >   Nadav
> >
> > On Wed, 2010-03-03 at 15:31 +0100, Jesper Larsen wrote:
> > > Hi people,
> > >
> > > I was wondering about the status of using the standard library
> > > multiprocessing module with numpy. I found a cookbook example last
> > > updated one year ago which states that:
> > >
> > > "This page was obsolete as multiprocessing's internals have changed.
> > > More information will come shortly; a link to this page will then be
> > > added back to the Cookbook."
> > >
> > > http://www.scipy.org/Cookbook/multiprocessing
> > >
> > > I also found the code that used to be on this page in the cookbook but
> > > it does not work any more. So my question is:
> > >
> > > Is it possible to use numpy arrays as shared arrays in an application
> > > using multiprocessing and how do you do it?
> > >
> > > Best regards,
> > > Jesper
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > NumPy-Discussion at scipy.org
> > > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

-- 
Francesc Alted