[Python-ideas] Fwd: Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`

Google apparently ate this message, and the next one, so... Forwarding them. Apologies for the mess. Apparently you can't just reply to messages that arrive on the list via Google Groups? Sent from a random iPhone Begin forwarded message:

From: Andrew Barnert Date: January 21, 2014, 0:42:11 PST To: Ram Rachum Cc: "python-ideas@googlegroups.com" Subject: Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`

On Jan 17, 2014, at 5:00, Ram Rachum wrote:

...

Hi,

I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process.

What for?

Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either.

There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor.