[Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`

[Ram Rachum]

Thanks for writing this Andrew!

I think you're right, it doesn't really offer a performance advantage over
using multiple processes, so I guess I should stick to ProcessPoolExecutor.

Thanks for taking the time to write this!


Ram.


On Tue, Jan 21, 2014 at 9:30 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> I slapped together a fork of concurrent/futures/process.py. It's named
> "procthreadex.py", and it just uses a ThreadPoolExecutor in the
> _process_worker function. You can get it at http://pastebin.com/Ba2KPYy3,
> and a test program skeleton at http://pastebin.com/ifwX6NaB.
>
> Maybe you can find a use case where ProcessThreadPoolExecutor(4, 4)
> outperforms ProcessPoolExecutor(16). (I haven't been able to.)
>
>   ------------------------------
>  *From:* Ram Rachum <ram.rachum at gmail.com>
> *To:* Andrew Barnert <abarnert at yahoo.com>
> *Cc:* "python-ideas at googlegroups.com" <python-ideas at googlegroups.com>
> *Sent:* Tuesday, January 21, 2014 2:17 AM
>
> *Subject:* Re: [Python-ideas] Add `n_threads` argument to
> `concurrent.futures.ProcessPoolExecutor`
>
> If you're writing code that needs to use both a lot of IO and a lot of
> CPU. For example, you're downloading many items from the internet and then
> doing post-processing on them.
>
>
> On Tue, Jan 21, 2014 at 10:42 AM, Andrew Barnert <abarnert at yahoo.com>wrote:
>
> On Jan 17, 2014, at 5:00, Ram Rachum <ram.rachum at gmail.com> wrote:
>
> > Hi,
> >
> > I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each
> process contain multiple worker threads. We could have an `n_threads`
> argument to the constructor, defaulting to 1 to maintain backward
> compatibility, and setting a value higher than 1 would cause multiple
> threads to be spawned in each process.
>
> What for?
>
> Generally you use processes because you can't use threads. Whether this is
> because you're running CPU-bound code that needs to get around the GIL,
> because you want complete isolation between tasks, because your platform
> doesn't support threads, or any other reason I can think of, you wouldn't
> want threads per process either.
>
> There are use cases for multiple processes of multiple threads, like
> running four independent IOCP-based servers (let them all try to use all
> your cores and let the kernel load balance among them), or isolated tasks
> with sharing-based subtasks... But those kinds of uses don't make sense in
> a single executor.
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140121/53fa9af1/attachment.html>