[stdlib-sig] futures - a new package for asynchronous execution

[Brian Quinlan]

On 7 Nov 2009, at 12:40, Antoine Pitrou wrote:

>
>> To Antoine's Twisted comment, I don't see a direct comparison. From  
>> my
>> understanding Twisted's Deferred objects are ways to have callbacks
>> executed once an async event occurs, not to help execute code
>> concurrently.
>
> Well, waiting for concurrently executing code to terminate *is* a case
> of waiting for an async event to happen.
>
> Deferred objects are a generic mechanism to chain reactions to
> termination or failure of code. Whether the event your Deferred reacts
> to is "async" or not is really a matter of how you use it (and of how
> you define "async" -- perhaps you meant "I/O" but Deferreds are not
> specialized for I/O).

They do seem specialized for continuation-passing style programming  
though. As far as I can tell from the docs (http://python.net/crew/mwh/apidocs/twisted.internet.defer.Deferred.html 
), the only way to process the results of a Deferred is my installing  
a callback.

Maybe you could outline (at a super-high-level) how you would  
implement my URL-downloading example using a Deferred-based API? Maybe  
something like:

def print_success(result, url):
    print('%r page is %d bytes' % (url, len(result)))

def print_failure(exception, url):
   print('%r generated an exception: %s' % (url, exception))

with ThreadedDeferredMaker(max_threads=5) as dm
   deferreds = []
   for url in URLS:
     deferred = dm.defer(load_url, url)
     deferred. addCallbacks(print_success, print_failure, url=url)
     deferred.unpause()
     deferreds.append(deferred)
   dm.wait_for_all_to_complete(deferreds)

The semantics aren't quite the same because the order of the output  
would be non-deterministic in this case. OTOH, you are going to get  
intermediate results as they become available, which is cool.


Cheers,
Brian