[stdlib-sig] futures - a new package for asynchronous execution

[Brian Quinlan]

On Nov 7, 2009, at 10:59 AM, Antoine Pitrou wrote:

>
>> I'm +1 adding something which provides this functionality to the
>> standard library, in fact it's been on my todo list since adding
>> multiprocessing. I actually suggested something like this in private
>> discussions around possible additions to jython for additional
>> concurrency constructs.
>>
>> That being said, I'll try to carve time off soon to dig into the pep
>> and the implementation.
>
> Have you thought at looking at Twisted and Deferred objects?
> What's the point of starting from scratch when an existing project  
> has a
> well-known and well-established API?

Hey Antoine,

I'm not an expert in Twisted but I think that Deferreds are looking to  
solve a slightly different problem.

I'm trying to solve the problem of easily parallelizing *synchronous*  
tasks (using threads of processes). My original use case looked  
something like this:

for source_path in source_paths:
     for destination_path in destination_paths:
         DoBlockingNetworkCopy(source_path, destination_path)

Which, in my proposed package, could be written as:

copies = []
for source_path in source_paths:
     for destination_path in destination_paths:
         copies.append(partial(DoBlockingNetworkCopy, source_path,  
destination_path))

with ThreadPoolExecutor(20) as e:
     e.run_to_futures(copies, return_when=FIRST_EXCEPTION)

Twisted Deferreds represent a fundamentally *asynchronous* operation  
(e.g. non-blocking I/O).

Cheers,
Brian