[stdlib-sig] futures - a new package for asynchronous execution

Frank Wierzbicki fwierzbicki at gmail.com
Sat Nov 7 01:09:53 CET 2009

On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.
> I recently wrote a solution for the use case of parallelizing network copies
> and RPC using threads without forcing the user to explicitly creating thread
> pools, work queues, etc.
> I have a concrete implementation that I'll describe below but I'd be happy
> to hear about other strategies!
> The basic idea is to implement an asynchronous execution method patterned
> heavily on java.util.concurrent (but less lame because Python has functions
> as first-class objects).  Here is a fairly advanced example:
> import futures
> import functools
> import urllib.request
> URLS = [
>    'http://www.foxnews.com/',
>    'http://www.cnn.com/',
>    'http://europe.wsj.com/',
>    'http://www.bbc.co.uk/',
>    'http://some-made-up-domain.com/']
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> #   with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:
>    future_list = executor.run_to_futures(
>        [functools.partial(load_url, url, 30) for url in URLS])
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
>    if future.exception() is not None:
>        print('%r generated an exception: %s' % (url, future.exception()))
>    else:
>        print('%r page is %d bytes' % (url, len(future.result())))
> In this example, executor.run_to_futures() returns only when every url has
> been retrieved but it is possible to return immediately, on the first
> completion or on the first failure depending on the desired work pattern.
> The complete docs are here:
> http://sweetapp.com/futures/
> A draft PEP is here:
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
> And the code is here:
> http://pypi.python.org/pypi/futures3/
Since this is modeled on java.util.concurrent Futures, I bet it would
be pretty straightforward to implement in Jython... I'm going to be a
little swamped with job hunt related stuff for a bit, but after that I
love to have a talk about this.


More information about the stdlib-sig mailing list