<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 24 Oct 2009, at 21:37, larudwer wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><br>"Brian Quinlan" <<a href="mailto:brian@sweetapp.com">brian@sweetapp.com</a>> schrieb im Newsbeitrag <br><a href="news:mailman.1895.1256264717.2807.python-list@python.org">news:mailman.1895.1256264717.2807.python-list@python.org</a>...<br><blockquote type="cite"><br></blockquote><blockquote type="cite">Any ideas why this is happening?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Cheers,<br></blockquote><blockquote type="cite">Brian<br></blockquote><br>IMHO your code is buggy. You run in an typical race condition.<br><br>consider following part in your code:<br><br><blockquote type="cite">def _make_some_processes(q):<br></blockquote><blockquote type="cite"> processes = []<br></blockquote><blockquote type="cite"> for _ in range(10):<br></blockquote><blockquote type="cite"> p = multiprocessing.Process(target=_process_worker, args=(q,))<br></blockquote><blockquote type="cite"> p.start()<br></blockquote><blockquote type="cite"> processes.append(p)<br></blockquote><blockquote type="cite"> return processes<br></blockquote><br><br>p.start() may start an process right now, in 5 seconds or an week later, <br>depending on how the scheduler of your OS works.<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote><div><br></div>Agreed.</div><div><br><blockquote type="cite"><div>Since all your processes are working on the same queue it is -- very -- <br>likely that the first process got started, processed all the input and <br>finished, while all the others haven't even got started.</div></blockquote><div><br></div><div>Agreed.</div><br><blockquote type="cite"><div> Though your first <br>process exits, and your main process also exits, because the queue is empty <br>now ;).<br><font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote><div><br></div><div>The main process shouldn't (and doesn't exit) - the _do function exits (with some processes possibly still running) and the next iteration in </div><div><br></div><div><div>for i in range(100):</div><div> _do(i)</div><div><br></div><div>is evaluated.</div></div><div><br></div><br><blockquote type="cite"><div><blockquote type="cite"> while not q.empty():<br></blockquote><blockquote type="cite"> pass<br></blockquote><br>If you where using p.join() your main process wourd terminate when the last <br>process terminates !<br>That's an different exit condition!<br><br></div></blockquote><div><br></div><div>When you say "your main process would terminate", you mean that the _do function would exit, right? Because process.join() has nothing to do with terminating the calling process - it just blocks until process terminates.</div><br><blockquote type="cite"><div>When the main process terminates all the garbage collection fun happens. I <br>hope you don't wonder that your Queue and the underlaying pipe got closed <br>and collected!<font class="Apple-style-span" color="#000000"><font class="Apple-style-span" color="#144FAE"><br></font></font></div></blockquote><br><div>I expected the queue and underlying queue and pipe to get collected.</div><div><br></div><blockquote type="cite"><div>Well now that all the work has been done, your OS may remember that someone <br>sometimes in the past told him to start an process.<br></div></blockquote><div><br></div><div>Sure, that could happen at this stage. Are you saying that it is the user of the multiprocessing module's responsibility to ensure that the queue is not collected in the parent process until all the child processes using it have exited? Actually, causing the queues to never be collected fixes the deadlock:</div><div><br></div><div>+ p = []</div><div><div>def _do(i):</div><div> print('Run:', i)</div><div> q = multiprocessing.Queue()</div><div>+ p.append(q)</div><div> print('Created queue')</div><div> for j in range(30):</div><div> q.put(i*30+j)</div><div> processes = _make_some_processes(q)</div><div> print('Created processes')</div><div><br></div><div> while not q.empty():</div><div> pass</div><div> print('Q is empty')</div><div><br></div><div>This behavior is counter-intuitive and, as far as I can tell, not documented anywhere. So it feels like a bug.</div></div><div><br></div>Cheers,</div><div>Brian</div><div><br><blockquote type="cite"><div><blockquote type="cite">def _process_worker(q):<br></blockquote><blockquote type="cite"> while True:<br></blockquote><blockquote type="cite"> try:<br></blockquote><blockquote type="cite"> something = q.get(block=True, timeout=0.1)<br></blockquote><blockquote type="cite"> except queue.Empty:<br></blockquote><blockquote type="cite"> return<br></blockquote><blockquote type="cite"> else:<br></blockquote><blockquote type="cite"> print('Grabbed item from queue:', something)<br></blockquote><br>The line<br><br>something = q.get(block=True, timeout=0.1)<br><br>should cause some kind of runtime error because q is already collected at <br>that time.<br>Depending on your luck and the OS this bug may be handled or not. Obviously <br>you are not lucky on OSX ;)<br><br>That's what i think happens.<br><br><br><br><br><br><br><br>-- <br><a href="http://mail.python.org/mailman/listinfo/python-list">http://mail.python.org/mailman/listinfo/python-list</a><br></div></blockquote></div><br></body></html>