On Jun 6, 2016, at 5:38 AM, Christoph Groth <christoph@grothesque.org> wrote:
As far as I can see the original concurrent.futures.ProcessPoolExecutor has no advantage when used in asyncio-based programs except when some coroutine blocks for long enough for the call queue to become empty. (But hey, async programming is about non-blocking coroutines!)
That’s incorrect. The ProcessPoolExecutor is a way for you to run CPU-intensive tasks without affecting the responsiveness of the main event loop in your program. As long as you keep your arguments small (pickling/unpickling big data structures is pretty detrimental to performance here) and the duration of the task is non-trivial, the child processes enable you to use more CPU cores, and since they have their own GILs, this doesn't affect your loop at all, unlike using a thread pool. I agree that the process pool is useless for a torrent of very fine-grained tasks where locking, futures and IPC are a significant fraction of the work to be done. But that doesn’t mean the process pool executor has no advantage. We’re using the PPE just fine at Facebook in cases where using TPE was being affected by the GIL. -- Lukasz Langa | Facebook Production Engineer | The Ministry of Silly Walks (+1) 650-681-7811