<div dir="ltr">I actually know very little about multiprocessing (have never used it) but I imagine the way you normally interact with multiprocessing is using a synchronous calls that talk to the subprocesses and their work queues and so on, right?<br>
<br>In the asyncio world you would put that work in a thread and then use run_in_executor() with a thread executor -- the thread would then be managing the subprocesses and talking to them. While you are waiting for that thread to complete your other coroutines will still work.<br>
<br>Unless you want to rewrite the communication and process management as coroutines, but that sounds like a lot of work.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jul 26, 2014 at 1:59 PM, Dan O'Reilly <span dir="ltr"><<a href="mailto:oreilldf@gmail.com" target="_blank">oreilldf@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I think it would be helpful for folks using the asyncio module to be able to make non-blocking calls to objects in the multiprocessing module more easily. While some use-cases for using multiprocessing can be replaced with ProcessPoolExecutor/run_in_executor, there are others that cannot; more advanced usages of multiprocessing.Pool aren't supported by ProcessPoolExecutor (initializer/initargs, contexts, etc.), and other multiprocessing classes like Lock and Queue have blocking methods that could be made into coroutines.<div>
<br></div><div>Consider this (extremely contrived, but use your imagination) example of a asyncio-friendly Queue:</div><div><br></div><div><font face="courier new, monospace">import asyncio</font></div><div><font face="courier new, monospace">import time</font></div>
<div><br></div><div><div><div><font face="courier new, monospace">def do_proc_work(q, val, val2):</font></div><div><font face="courier new, monospace"> time.sleep(3) # Imagine this is some expensive CPU work.</font></div>
<div><font face="courier new, monospace"> ok = val + val2</font></div><div><font face="courier new, monospace"> print("Passing {} to parent".format(ok))</font></div><div><font face="courier new, monospace"> q.put(ok) </font><span style="font-family:'courier new',monospace"># The Queue can be used with the normal blocking API, too.</span></div>
<div><font face="courier new, monospace"> item = q.get() </font></div><div><font face="courier new, monospace"> print("got {} back from parent".format(item))</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">def do_some_async_io_task():</font></div><div><font face="courier new, monospace"> # Imagine there's some kind of asynchronous I/O </font></div><div><font face="courier new, monospace"> # going on here that utilizes asyncio.</font></div>
<div><font face="courier new, monospace"> asyncio.sleep(5)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">@asyncio.coroutine</font></div><div><font face="courier new, monospace">def do_work(q):</font></div>
<div><font face="courier new, monospace"> loop.run_in_executor(ProcessPoolExecutor(),</font></div><div><font face="courier new, monospace"> do_proc_work, q, 1, 2)</font></div><div><font face="courier new, monospace"> do_some_async_io_task()</font></div>
<div><font face="courier new, monospace"> item = yield from q.coro_get() # Non-blocking get that won't affect our io_task</font></div><div><font face="courier new, monospace"> print("Got {} from worker".format(item))</font></div>
<div><font face="courier new, monospace"> item = item + 25</font></div><div><font face="courier new, monospace"> yield from q.coro_put(item)</font></div></div><div><br></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">if __name__ == "__main__":</font></div><div><font face="courier new, monospace"> q = AsyncProcessQueue() # This is our new asyncio-friendly version of multiprocessing.Queue </font></div>
<div><font face="courier new, monospace"> loop = asyncio.get_event_loop()</font></div><div><font face="courier new, monospace"> loop.run_until_complete(do_work(q))</font></div></div><div><br></div><div>I have seen some rumblings about a desire to do this kind of integration on the bug tracker (<a href="http://bugs.python.org/issue10037#msg162497" target="_blank">http://bugs.python.org/issue10037#msg162497</a> and <a href="http://bugs.python.org/issue9248#msg221963" target="_blank">http://bugs.python.org/issue9248#msg221963</a>) though that discussion is specifically tied to merging the enhancements from the Billiard library into multiprocessing.Pool. Are there still plans to do that? If so, should asyncio integration with multiprocessing be rolled into those plans, or does it make sense to pursue it separately?</div>
<div><br></div><div>Even more generally, do people think this kind of integration is a good idea to begin with? I know using asyncio is primarily about *avoiding* the headaches of concurrent threads/processes, but there are always going to be cases where CPU-intensive work is going to be required in a primarily I/O-bound application. The easier it is to for developers to handle those use-cases, the better, IMO.</div>
<div><br></div><div>Note that the same sort of integration could be done with the threading module, though I think there's a fairly limited use-case for that; most times you'd want to use threads over processes, you could probably just use non-blocking I/O instead.</div>
<div><br></div><div>Thanks,</div><div>Dan</div><div><br></div></div>
<br>_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" target="_blank">http://python.org/psf/codeofconduct/</a><br></blockquote></div><br><br clear="all"><br>-- <br>--Guido van Rossum (<a href="http://python.org/~guido">python.org/~guido</a>)
</div>