[Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`

[Andrew Barnert]

I slapped together a fork of concurrent/futures/process.py. It's named "procthreadex.py", and it just uses a ThreadPoolExecutor in the _process_worker function. You can get it at http://pastebin.com/Ba2KPYy3, and a test program skeleton at http://pastebin.com/ifwX6NaB.

Maybe you can find a use case where ProcessThreadPoolExecutor(4, 4) outperforms ProcessPoolExecutor(16). (I haven't been able to.)



>________________________________
> From: Ram Rachum <ram.rachum at gmail.com>
>To: Andrew Barnert <abarnert at yahoo.com> 
>Cc: "python-ideas at googlegroups.com" <python-ideas at googlegroups.com> 
>Sent: Tuesday, January 21, 2014 2:17 AM
>Subject: Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`
> 
>
>
>If you're writing code that needs to use both a lot of IO and a lot of CPU. For example, you're downloading many items from the internet and then doing post-processing on them.
>
>
>
>On Tue, Jan 21, 2014 at 10:42 AM, Andrew Barnert <abarnert at yahoo.com> wrote:
>
>On Jan 17, 2014, at 5:00, Ram Rachum <ram.rachum at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process.
>>
>>What for?
>>
>>Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either.
>>
>>There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140121/3065f84a/attachment-0001.html>