[Python-ideas] Composability and concurrent.futures

[Matt Joiner]

>
> To be clear, I meant to refer to processes *or* threads when discussing
> the problem originally. The ProcessPoolExecutor is pretty useful (in my
> experience) for easily getting speedup even on pure-Python CPU-bound
> workloads.
>

FWIW that wasn't the default "use processes" spike. In my experience toying
with concurrency in Python, trying to manage the load threads put on the
system always ends badly. The 2 best supported concurrency mechanisms,
threads and processes are constantly tête-à-tête, neither are adequate when
you start to consider extreme concurrency scenarios. I suggest this because
if you're considering composing executors, you're already trying to reduce
the overhead (wastage) that processes and threads are incurring on your
system for these purposes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120524/e28075c8/attachment.html>