Advice regarding multiprocessing module

Dave Angel davea at davea.name
Mon Mar 11 12:27:28 CET 2013


On 03/11/2013 01:57 AM, Abhinav M Kulkarni wrote:
>
>
>   <SNIP>
>   * My laptop has quad-core Intel i5 processor, so I thought using
>     multiprocessing module I can parallelize my code (basically
>     calculate gradient in parallel on multiple cores simultaneously).
    <SNIP>
>   * As a result I end up creating a process for each data point (instead
>     of a thread that I would ideally like to do, so as to avoid process
>     creation overhead).

Seems you only need 4 processes, as you have 4 cores.  Instead of 
creating a new one each time, reuse the same 4 processes, letting each 
do a quarter of the data.

It's not the process creation that's particularly slow, but all the 
initialization of starting another instance of Python.  If you're on 
Linux, you might be able to speed that up by using fork, but I don't 
specifically know.



-- 
DaveA



More information about the Python-list mailing list