[Python-Dev] [PEP 3148] futures - execute computations asynchronously

[Nick Coghlan]

Brian Quinlan wrote:
>> IOW, as far as I can tell from the PEP, it doesn't look like you can
>> compose futures without *global* knowledge of the application...  and
>> in and of itself, this seems to negate the PEP's own motivation to
>> prevent duplication of parallel execution handling!
>>
>> That is, if I use code from module A and module B that both want to
>> invoke tasks asynchronously, and I want to invoke A and B
>> asynchronously, what happens?  Based on the design of the API, it
>> appears there is nothing you can do except refactor A and B to take an
>> executor in a parameter, instead of creating their own.
> 
> A and B could both use their own executor instances. You would need to
> refactor A and B if you wanted to manage thread and process counts
> globally.

You may want to consider providing global thread and process executors
in the futures module itself. Code which just wants to say "do this in
the background" without having to manage the lifecycle of its own
executor instance is then free to do so. I've had a lot of experience
with a framework that provides this and it is *very* convenient (it's
also a good way to avoid deadlocks due to synchronous notification APIs).


On PJE's broader point, async event loops with non-blocking I/O and
messages passed back to the event loop to indicate completion of
operations and relying on threads and processes to farm out tasks (which
you will later block on in order to retrieve the results) are completely
different programming models. This PEP doesn't change that - it just
makes certain aspects of the latter approach easier to handle.

Trying to design an API that can cope with either model strikes me as a
fool's errand. They differ at such a fundamental level that I don't see
how a hybrid API could be particularly optimal for either approach.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------