multiprocessing problems

[Adam Tauno Williams]

> I decided to play around with the multiprocessing module, and I'm
> having some strange side effects that I can't explain.  It makes me
> wonder if I'm just overlooking something obvious or not.  Basically, I
> have a script parses through a lot of files doing search and replace
> on key strings inside the file.  I decided the split the work up on
> multiple processes on each processor core (4 total).  I've tried many
> various ways doing this form using pool to calling out separate
> processes, but the result has been the same: computer crashes from
> endless process spawn.

Are you hitting a ulimit error?  The number of processes you can create
is probably limited. 

TIP: close os.stdin on your subprocesses.

> Here's the guts of my latest incarnation.
> def ProcessBatch(files):
>     p = []
>     for file in files:
>         p.append(Process(target=ProcessFile,args=file))
>     for x in p:
>         x.start()
>     for x in p:
>         x.join()
>     p = []
>     return
> Now, the function calling ProcessBatch looks like this:
> def ReplaceIt(files):
>     processFiles = []
>     for replacefile in files:
>         if(CheckSkipFile(replacefile)):
>             processFiles.append(replacefile)
>             if(len(processFiles) == 4):
>                 ProcessBatch(processFiles)
>                 processFiles = []
>     #check for left over files once main loop is done and process them
>     if(len(processFiles) > 0):
>         ProcessBatch(processFiles)

According to this you will create files is sets of four, but an unknown
number of sets of four.