[stdlib-sig] futures - a new package for asynchronous execution

Jeffrey Yasskin jyasskin at gmail.com
Fri Nov 13 06:19:05 CET 2009

I am very happy with those changes.

I think deadlock should be addressed before the first release as it
changes the detailed semantics of some of the operations, but you've
promised to do that, so cool. :)

I think it's fine to leave the embedding of Deferred-like things into
futures and the embedding of futures into Deferred-like things until a
later release. I expect it to be requested, but I don't think it'll be
hard to add later.

On Thu, Nov 12, 2009 at 6:38 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
> I compiled a summary of people's feedback (about technical issues - I agree
> that the docs could be better but agreeing on the API seems like the first
> step) and have some API change proposals.
> Here is a summary of the feedback:
> - Use Twisted Deferreds rather than Futures
> - The API too complex
> - Make Future a callable and drop the .result()/.exception() methods
> - Remove .wait() from Executor
> - Make it easy to process results in the order of completion rather than in
> the order that the futures were generated
> - Executor context managers should wait until their workers complete before
> exiting
> - Extract Executor.map, etc. into separate functions/modules
> - FutureList has too many methods or is not necessary
> - Executor should have an easy way to produce a single future
> - Should be able to wait on an arbitrary list of futures
> - Should have a way of avoiding deadlock (will follow-up on this separately)
> Here is what I suggest as far as API changes (the docs suck, I'll polish
> them when we reach consensus):
> FutureList is eliminated completely.
> Future remains unchanged - I disagree that Deferreds would be better, that
> .exception() is not useful, and that .result() should be renamed .get() or
> .__call__(). But I am easily persuadable :-)
> The Executor ABC is simplified to only contain a single method:
> def Executor.submit(self, fn, *args, **kwargs) :
> Submits a call for execution and returns a Future representing the pending
> results of fn(*args, **kwargs)
> map becomes a utility function:
> def map(executor, *iterables, timeout=None)
> Equivalent to map(func, *iterables) but executed asynchronously and possibly
> out-of-order. The returned iterator raises a TimeoutError if __next__() is
> called and the result isn’t available after timeout seconds from the
> original call to run_to_results(). If timeout is not specified or None then
> there is no limit to the wait time. If a call raises an exception then that
> exception will be raised when its value is retrieved from the iterator.
> wait becomes a utility function that can wait on any iterable of Futures:
> def wait(futures, return_when=ALL_COMPLETED)
> Wait until the given condition is met for the given futures. This method
> should always be called using keyword arguments, which are:
> timeout can be used to control the maximum number of seconds to wait before
> returning. If timeout is not specified or None then there is no limit to the
> wait time.
> return_when indicates when the method should return. It must be one of the
> following constants:
> a new utility function is added that iterates over the given Futures and
> returns the as they are completed:
> def itercompleted(futures, timeout=None):
> Returns an iterator that returns a completed Future from the given list when
> __next__() is called. If no Futures are completed then __next__() is called
> then __next__() waits until one does complete. Raises a TimeoutError if
> __next__() is called and no completed future is available after timeout
> seconds from the original call.
> The URL loading example becomes:
> import functools
> import urllib.request
> import futures
> URLS = ['http://www.foxnews.com/',
>        'http://www.cnn.com/',
>        'http://europe.wsj.com/',
>        'http://www.bbc.co.uk/',
>        'http://some-made-up-domain.com/']
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
> with futures.ThreadPoolExecutor(50) as executor:
>  fs = [executor.submit(load_url, url, timeout=30) for url in URLS]
> for future in futures.itercompleted(fs):
>    if future.exception() is not None:
>        print('%r generated an exception: %s' % (url, future.exception()))
>    else:
>        print('%r page is %d bytes' % (url, len(future.result())))
> What do you think? Are we moving in the right direction?
> Cheers,
> Brian
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig

Jeffrey Yasskin

More information about the stdlib-sig mailing list