[Python-ideas] Composability and concurrent.futures

[Adrian Sampson]

On May 21, 2012, at 9:17 AM, Matt Joiner wrote:

> Personally if I implemented a composable thread pool I'd have it
> global, creation and submission of tasks would be proxied to it via
> some composable executor class.

I agree completely. Maybe the implementation I described was overly
hacky for the sake of transparent compatibility with the existing
(non-composable) executors in concurrent.futures. Ideally, the system
would have one global pool which many concurrency APIs -- not just
concurrent.futures -- could potentially share.

(In a *really* ideal world, the OS would provide thread pool management
-- like GCD, which you mentioned, or scheduler activations. But a
cross-platform library currently requires a less ambitious solution.)

> In short, I think composable thread pools are a hack at best and won't
> gain you anything except a slightly reduced threading overhead. If you
> want optimal utilization, threading isn't the right place to be
> looking.

To be clear, I meant to refer to processes *or* threads when discussing
the problem originally. The ProcessPoolExecutor is pretty useful (in my
experience) for easily getting speedup even on pure-Python CPU-bound
workloads.

Adrian