[BangPypers] Multiprocessing takes higher execution time

Anand Balachandran Pillai abpillai at gmail.com
Wed Jan 7 11:02:18 CET 2009


On Wed, Jan 7, 2009 at 3:22 PM, Amit k. Saha <amitsaha.in at gmail.com> wrote:
> On Wed, Jan 7, 2009 at 3:10 PM, Sibtey Mehdi <sibteym at infotechsw.com> wrote:
>> Hi,
>>
>>
>>
>> I use multiprocessing to compare more then one set of files.
>>
>> For comparison each set of files (i.e. Old file1 Vs New file1) I create a
>> process,
>>
>> Process(target=compare, args=(oldFile, newFile)).start()
>>
>> It takes 61 seconds execution time.
>>
>>
>>
>> When I do the same comparison without implementing multiprocessing, it takes
>> 52 seconds execution time.
>>
>>
>>
>> The parallel processing time should be lesser.
>>
>>
>>
>> I am not able to get advantage of multiprocessing here.

No surprise here. A process is a heavyweight entity. The design
of spawning a process per file comparison won't scale and as
the number of comparisons increase, will asymptotically
perform worse than doing all in a single process/thread.

The flaw is in the design. A parallel programming task as above
is about distributing m jobs onto n resources. There are two
trivial cases for it -> where n is 1 (m jobs in 1 resource, i.e a single
process/thread) and where m is 1 (i.e 1 job per resource - spawing
a resource for every job as you are doing).

These two trivial cases are two extremes and will always
yield poorer results when compared to choosing the right value
of m and n. In this case you should group your comparison tasks
and execute say for example 10 comparisons per thread/process.
Design a thread/process pool to scale your solution.

This is simply Concurrent/Parallel Programming 101...

>
> Depends on the overheads of spawning multiple threads/processes v$
> your actual processing. The same way, things are compared in old
> school when they used to compare macros v$ functions in C/C++.
>
> -Amit
>
>
>>
>>
>>
>> Any suggestions can be very helpful.
>>
>>
>>
>> Thanks,
>>
>> Gopal
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> BangPypers mailing list
>> BangPypers at python.org
>> http://mail.python.org/mailman/listinfo/bangpypers
>>
>>
>
>
>
> --
> Amit Kumar Saha
> http://amitksaha.blogspot.com
> http://amitsaha.in.googlepages.com/
> Skype: amitkumarsaha
> _______________________________________________
> BangPypers mailing list
> BangPypers at python.org
> http://mail.python.org/mailman/listinfo/bangpypers
>



-- 
-Anand


More information about the BangPypers mailing list