<div dir="ltr">Thanks for writing this Andrew!<div><br></div><div>I think you're right, it doesn't really offer a performance advantage over using multiple processes, so I guess I should stick to ProcessPoolExecutor.</div>
<div><br></div><div>Thanks for taking the time to write this! </div><div><br></div><div><br></div><div>Ram.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 21, 2014 at 9:30 PM, Andrew Barnert <span dir="ltr"><<a href="mailto:abarnert@yahoo.com" target="_blank">abarnert@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div><span>I slapped together a fork of concurrent/futures/process.py. It's named "procthreadex.py", and it just uses a ThreadPoolExecutor in the _process_worker function. You can get it at <a href="http://pastebin.com/Ba2KPYy3" target="_blank">http://pastebin.com/Ba2KPYy3</a>, and a test program skeleton at <a href="http://pastebin.com/ifwX6NaB" target="_blank">http://pastebin.com/ifwX6NaB</a>.</span></div>
<div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif"><span><br></span></div><div style="font-style:normal;font-size:16px;background-color:transparent;font-family:'times new roman','new york',times,serif">
<span>Maybe you can find a use case where ProcessThreadPoolExecutor(4, 4) outperforms ProcessPoolExecutor(16). (I haven't been able to.)</span></div><div><br><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;margin-top:5px;padding-left:5px">
<div style="font-family:'times new roman','new york',times,serif;font-size:12pt"> <div style="font-family:'times new roman','new york',times,serif;font-size:12pt"> <div dir="ltr"> <hr size="1">
<font face="Arial"> <b><span style="font-weight:bold">From:</span></b> Ram Rachum <<a href="mailto:ram.rachum@gmail.com" target="_blank">ram.rachum@gmail.com</a>><br> <b><span style="font-weight:bold">To:</span></b> Andrew Barnert <<a href="mailto:abarnert@yahoo.com" target="_blank">abarnert@yahoo.com</a>> <br>
<b><span style="font-weight:bold">Cc:</span></b> "<a href="mailto:python-ideas@googlegroups.com" target="_blank">python-ideas@googlegroups.com</a>" <<a href="mailto:python-ideas@googlegroups.com" target="_blank">python-ideas@googlegroups.com</a>> <br>
<b><span style="font-weight:bold">Sent:</span></b> Tuesday, January 21, 2014 2:17 AM<div class="im"><br> <b><span style="font-weight:bold">Subject:</span></b> Re: [Python-ideas] Add `n_threads` argument to `concurrent.futures.ProcessPoolExecutor`<br>
</div></font> </div> <div><br><div><div class="h5">
<div><div dir="ltr">If you're writing code that needs to use both a lot of IO and a lot of CPU. For example, you're downloading many items from the internet and then doing post-processing on them.</div><div>
<br><br><div>On Tue, Jan 21, 2014 at 10:42 AM, Andrew Barnert <span dir="ltr"><<a rel="nofollow" href="mailto:abarnert@yahoo.com" target="_blank">abarnert@yahoo.com</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On Jan 17, 2014, at 5:00, Ram Rachum <<a rel="nofollow" href="mailto:ram.rachum@gmail.com" target="_blank">ram.rachum@gmail.com</a>> wrote:<br>
<br>
> Hi,<br>
><br>
> I'd like to use `concurrent.futures.ProcessPoolExecutor` but have each process contain multiple worker threads. We could have an `n_threads` argument to the constructor, defaulting to 1 to maintain backward compatibility, and setting a value higher than 1 would cause multiple threads to be spawned in each process.<br>
<br>
</div>What for?<br>
<br>
Generally you use processes because you can't use threads. Whether this is because you're running CPU-bound code that needs to get around the GIL, because you want complete isolation between tasks, because your platform doesn't support threads, or any other reason I can think of, you wouldn't want threads per process either.<br>
<br>
There are use cases for multiple processes of multiple threads, like running four independent IOCP-based servers (let them all try to use all your cores and let the kernel load balance among them), or isolated tasks with sharing-based subtasks... But those kinds of uses don't make sense in a single executor.</blockquote>
</div><br></div>
</div><br><br></div></div></div> </div> </div> </blockquote></div> </div></div></blockquote></div><br></div>