multiprocessing deadlock

[Brian Quinlan]

On 24 Oct 2009, at 21:37, larudwer wrote:

>
> "Brian Quinlan" <brian at sweetapp.com> schrieb im Newsbeitrag
> news:mailman.1895.1256264717.2807.python-list at python.org...
>>
>> Any ideas why this is happening?
>>
>> Cheers,
>> Brian
>
> IMHO your code is buggy. You run in an typical race condition.
>
> consider following part in your code:
>
>> def _make_some_processes(q):
>>    processes = []
>>    for _ in range(10):
>>        p = multiprocessing.Process(target=_process_worker, args=(q,))
>>        p.start()
>>        processes.append(p)
>>    return processes
>
>
> p.start() may start an process right now, in 5 seconds or an week  
> later,
> depending on how the scheduler of your OS works.

Agreed.

> Since all your processes are working on the same queue it is -- very  
> --
> likely that the first process got started, processed all the input and
> finished, while all the others haven't even got started.

Agreed.

> Though your first
> process exits, and your main process also exits, because the queue  
> is empty
> now ;).
>

The main process shouldn't (and doesn't exit) - the _do function exits  
(with some processes possibly still running) and the next iteration in

for i in range(100):
     _do(i)

is evaluated.


>>    while not q.empty():
>>        pass
>
> If you where using p.join() your main process wourd terminate when  
> the last
> process terminates !
> That's an different exit condition!
>

When you say "your main process would terminate", you mean that the  
_do function would exit, right? Because process.join() has nothing to  
do with terminating the calling process - it just blocks until process  
terminates.

> When the main process terminates all the garbage collection fun  
> happens. I
> hope you don't wonder that your Queue and the underlaying pipe got  
> closed
> and collected!

I expected the queue and underlying queue and pipe to get collected.

> Well now that all the work has been done, your OS may remember that  
> someone
> sometimes in the past told him to start an process.

Sure, that could happen at this stage. Are you saying that it is the  
user of the multiprocessing module's responsibility to ensure that the  
queue is not collected in the parent process until all the child  
processes using it have exited? Actually, causing the queues to never  
be collected fixes the deadlock:

+ p = []
def _do(i):
     print('Run:', i)
     q = multiprocessing.Queue()
+  p.append(q)
     print('Created queue')
     for j in range(30):
         q.put(i*30+j)
     processes = _make_some_processes(q)
     print('Created processes')

     while not q.empty():
         pass
     print('Q is empty')

This behavior is counter-intuitive and, as far as I can tell, not  
documented anywhere. So it feels like a bug.

Cheers,
Brian

>> def _process_worker(q):
>>    while True:
>>        try:
>>            something = q.get(block=True, timeout=0.1)
>>        except queue.Empty:
>>            return
>>        else:
>>            print('Grabbed item from queue:', something)
>
> The line
>
> something = q.get(block=True, timeout=0.1)
>
> should cause some kind of runtime error because q is already  
> collected at
> that time.
> Depending on your luck and the OS this bug may be handled or not.  
> Obviously
> you are not lucky on OSX ;)
>
> That's what i think happens.
>
>
>
>
>
>
>
> -- 
> http://mail.python.org/mailman/listinfo/python-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20091025/ce14d1f0/attachment.html>