[BangPypers] Multiprocessing takes higher execution time

Jeff Rush jeff at taupro.com
Wed Jan 7 11:02:00 CET 2009


Amit k. Saha wrote:
> On Wed, Jan 7, 2009 at 3:10 PM, Sibtey Mehdi <sibteym at infotechsw.com> wrote:
>>
>> I use multiprocessing to compare more then one set of files.
>>
>> The parallel processing time should be lesser.
>>
>> I am not able to get advantage of multiprocessing here.
> 
> Depends on the overheads of spawning multiple threads/processes v$
> your actual processing. The same way, things are compared in old
> school when they used to compare macros v$ functions in C/C++.

Simple file comparison is limited more by disk I/O than CPU.  So by trying to
read more than one file at a time, you are causing the disk to seek back and
forth, losing time.

Try it again but this time place the files to be compared in a RAM disk, and
run both the multiprocess and single-process versions of your program.  I'll
best you see a difference then.  Or spread them across multiple physical (not
logical) disk drives.

-Jeff



More information about the BangPypers mailing list