multiprocessing deadlock
Brian Quinlan
brian at sweetapp.com
Fri Oct 23 09:45:02 EDT 2009
On 24 Oct 2009, at 00:02, paulC wrote:
> On Oct 23, 3:18 am, Brian Quinlan <br... at sweetapp.com> wrote:
>> My test reduction:
>>
>> import multiprocessing
>> import queue
>>
>> def _process_worker(q):
>> while True:
>> try:
>> something = q.get(block=True, timeout=0.1)
>> except queue.Empty:
>> return
>> else:
>> print('Grabbed item from queue:', something)
>>
>> def _make_some_processes(q):
>> processes = []
>> for _ in range(10):
>> p = multiprocessing.Process(target=_process_worker,
>> args=(q,))
>> p.start()
>> processes.append(p)
>> return processes
>>
>> def _do(i):
>> print('Run:', i)
>> q = multiprocessing.Queue()
>> for j in range(30):
>> q.put(i*30+j)
>> processes = _make_some_processes(q)
>>
>> while not q.empty():
>> pass
>>
>> # The deadlock only occurs on Mac OS X and only when these lines
>> # are commented out:
>> # for p in processes:
>> # p.join()
>>
>> for i in range(100):
>> _do(i)
>>
>> --------------
>>
>> Output (on Mac OS X using the svn version of py3k):
>> % ~/bin/python3.2 moprocessmoproblems.py
>> Run: 0
>> Grabbed item from queue: 0
>> Grabbed item from queue: 1
>> Grabbed item from queue: 2
>> ...
>> Grabbed item from queue: 29
>> Run: 1
>>
>> At this point the script produces no additional output. If I
>> uncomment
>> the lines above then the script produces the expected output. I don't
>> see any docs that would explain this problem and I don't know what
>> the
>> rule would be e.g. you just join every process that uses a queue
>> before
>> the queue is garbage collected.
>>
>> Any ideas why this is happening?
>>
>> Cheers,
>> Brian
>
> I can't promise a definitive answer but looking at the doc.s:-
>
> isAlive()
> Return whether the thread is alive.
>
> Roughly, a thread is alive from the moment the start() method
> returns until its run() method terminates. The module function
> enumerate() returns a list of all alive threads.
>
> I guess that the word 'roughly' indicates that returning from the
> start
> () call does not mean that all the threads have actually started, and
> so calling join is illegal. Try calling isAlive on all the threads
> before returning from _make_some_processes.
>
> Regards, Paul C.
> --
> http://mail.python.org/mailman/listinfo/python-list
Hey Paul,
I guess I was unclear in my explanation - the deadlock only happens
when I *don't* call join.
Cheers,
Brian
More information about the Python-list
mailing list