multiprocessing problems

Nils Ruettershoff nils at ccsg.de
Thu Jan 21 00:45:32 CET 2010


Hi,

Adam Tauno Williams wrote:

[...]
>> Here's the guts of my latest incarnation.
>> def ProcessBatch(files):
>>     p = []
>>     for file in files:
>>         p.append(Process(target=ProcessFile,args=file))
>>     for x in p:
>>         x.start()
>>     for x in p:
>>         x.join()
>>     p = []
>>     return
>> Now, the function calling ProcessBatch looks like this:
>> def ReplaceIt(files):
>>     processFiles = []
>>     for replacefile in files:
>>         if(CheckSkipFile(replacefile)):
>>             processFiles.append(replacefile)
>>             if(len(processFiles) == 4):
>>                 ProcessBatch(processFiles)
>>                 processFiles = []
>>     #check for left over files once main loop is done and process them
>>     if(len(processFiles) > 0):
>>         ProcessBatch(processFiles)
>>     
>
> According to this you will create files is sets of four, but an unknown
> number of sets of four.
>
>   
This is not correct, 'cause of the x.join(). This will block the parent 
process until all processes have been terminated. So as soon as the 
current set of processes have finished their job, a new set will be spawned.

Cheers,
Nils
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100121/8c4ec86d/attachment.html>


More information about the Python-list mailing list