multiprocessing Pool workers cannot create subprocesses

[John L. Stephens]

On 3/18/2011 7:54 PM, Jason Grout wrote:
>
> Right; thanks.  Let me rephrase my questions:
>
> 1. Why is important that the multiprocessing Pool worker processors 
> have daemon=True (I think this is the same as asking: why is it 
> important that they be terminated with terminate() rather than join() )?
>
> 2. Is it safe for us to reset a Pool worker process to have 
> daemon=False before starting a subprocess from that worker, like in 
> the code from the original message?
>
> Thanks,
>
> Jason
Jason,

I just happen to be dealing with a project that uses multiprocessing.

What I have learned is this...

If a child thread (pool worker) is not set to daemon (daemon=False), if 
for some reason the parent thread terminates either normally or 
abnormally and the worker thread has not completed its task, the child 
thread will terminate by throwing all sorts of nasty errors.  However, 
in daemon mode, multiprocessing will terminate the child thread 
'cleanly'.  That's not to say that the worker has a chance to complete 
its work or shut itself down.  Multiprocessing will absorb the 
exceptions and not pass them along.

You may terminate a child thread using join().  That is probably the 
safest way to do it.   terminate() will just kill the worker thread 
immediately without any regard to whether or not it has completed its 
tasks.  I believe multiprocessing uses terminate() as well to kill a 
daemon thread if the parent thread disappears.

join() will, however, block until the task has competed and returned.  
If you want to continue doing work in the parent thread while the child 
thread is off doing its thing, then another means of syncing up the 
parent and children threads is needed.

As for allowing children threads to spawn off children of its own using 
subprocess runs the risk of creating a little army of zombie 
'grandchildren' if either the parent or child threads terminate before 
the subprocess completes and returns.

Hope that helps....

John