[BangPypers] Multiprocessing takes higher execution time
jeff at taupro.com
Wed Jan 7 11:02:00 CET 2009
Amit k. Saha wrote:
> On Wed, Jan 7, 2009 at 3:10 PM, Sibtey Mehdi <sibteym at infotechsw.com> wrote:
>> I use multiprocessing to compare more then one set of files.
>> The parallel processing time should be lesser.
>> I am not able to get advantage of multiprocessing here.
> Depends on the overheads of spawning multiple threads/processes v$
> your actual processing. The same way, things are compared in old
> school when they used to compare macros v$ functions in C/C++.
Simple file comparison is limited more by disk I/O than CPU. So by trying to
read more than one file at a time, you are causing the disk to seek back and
forth, losing time.
Try it again but this time place the files to be compared in a RAM disk, and
run both the multiprocess and single-process versions of your program. I'll
best you see a difference then. Or spread them across multiple physical (not
logical) disk drives.
More information about the BangPypers