<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>I slapped together a fork of concurrent/futures/process.py. It's named "procthreadex.py", and it just uses a ThreadPoolExecutor in the _process_worker function. You can get it at http://pastebin.com/Ba2KPYy3, and a test program skeleton at http://pastebin.com/ifwX6NaB.</span></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: 'times new roman', 'new york', times, serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: 'times new roman', 'new york', times, serif; background-color: transparent; font-style: normal;"><span>Maybe you can find a use case where ProcessThreadPoolExecutor(4, 4) outperforms ProcessPoolExecutor(16). (I haven't been able to.)</span></div><div><br><blockquote style="border-left: 2px
solid rgb(16, 16, 255); margin-left: 5px; margin-top: 5px; padding-left: 5px;"> <div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"> <div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"> <div dir="ltr"> <hr size="1"> <font size="2" face="Arial"> <b><span style="font-weight:bold;">From:</span></b> Ram Rachum <ram.rachum@gmail.com><br> <b><span style="font-weight: bold;">To:</span></b> Andrew Barnert <abarnert@yahoo.com> <br><b><span style="font-weight: bold;">Cc:</span></b> "python-ideas@googlegroups.com" <python-ideas@googlegroups.com> <br> <b><span style="font-weight: bold;">Sent:</span></b> Tuesday, January 21, 2014 2:17 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`<br> </font> </div> <div class="y_msg_container"><br>
<meta http-equiv="x-dns-prefetch-control" content="off"><div id="yiv1179095114"><div dir="ltr">If you're writing code that needs to use both a lot of IO and a lot of CPU. For example, you're downloading many items from the internet and then doing post-processing on them.</div><div class="yiv1179095114gmail_extra">
<br><br><div class="yiv1179095114gmail_quote">On Tue, Jan 21, 2014 at 10:42 AM, Andrew Barnert <span dir="ltr"><<a rel="nofollow" ymailto="mailto:abarnert@yahoo.com" target="_blank" href="mailto:abarnert@yahoo.com">abarnert@yahoo.com</a>></span> wrote:<br><blockquote class="yiv1179095114gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="yiv1179095114im">On Jan 17, 2014, at 5:00, Ram Rachum <<a rel="nofollow" ymailto="mailto:ram.rachum@gmail.com" target="_blank" href="mailto:ram.rachum@gmail.com">ram.rachum@gmail.com</a>> wrote:<br>
<br>
> Hi,<br>
><br>
> I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process.<br>
<br>
</div>What for?<br>
<br>
Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either.<br>
<br>
There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor.</blockquote>
</div><br></div>
</div><meta http-equiv="x-dns-prefetch-control" content="on"><br><br></div> </div> </div> </blockquote></div> </div></body></html>