From: Andrew Barnert <abarnert@yahoo.com>
Date: January 21, 2014, 0:42:11 PST
To: Ram Rachum <ram.rachum@gmail.com>
Cc: "python-ideas@googlegroups.com" <python-ideas@googlegroups.com>
Subject: Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`
On Jan 17, 2014, at 5:00, Ram Rachum <ram.rachum@gmail.com> wrote:Hi,I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process.
What for?
Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either.
There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor.