On May 21, 2012, at 9:17 AM, Matt Joiner wrote:
Personally if I implemented a composable thread pool I'd have it global, creation and submission of tasks would be proxied to it via some composable executor class.
I agree completely. Maybe the implementation I described was overly hacky for the sake of transparent compatibility with the existing (non-composable) executors in concurrent.futures. Ideally, the system would have one global pool which many concurrency APIs -- not just concurrent.futures -- could potentially share. (In a *really* ideal world, the OS would provide thread pool management -- like GCD, which you mentioned, or scheduler activations. But a cross-platform library currently requires a less ambitious solution.)
In short, I think composable thread pools are a hack at best and won't gain you anything except a slightly reduced threading overhead. If you want optimal utilization, threading isn't the right place to be looking.
To be clear, I meant to refer to processes *or* threads when discussing the problem originally. The ProcessPoolExecutor is pretty useful (in my experience) for easily getting speedup even on pure-Python CPU-bound workloads. Adrian