Java-style futures in Python - only better

[Brian Quinlan]

Hey all,

I've been working on an Java-style futures implementation in Python. 
Futures are a way of representing asynchronous operations e.g. 
operations that are run in another thread or process. The are are a easy 
but powerful way of parallelizing sequential operations. The also 
provide a consistent interface across implementations e.g. they can 
provide the same interface to threading and to multiprocessing.

For example:

def is_prime(n):
     "Return True iff n is prime"
     ...

def check_primes(numbers):
     return map(is_prime, numbers)

Could be parallelized as:

def check_primes(numbers):
     # ProcessPoolExecutor will create one worker process
     # per CPU if called without arguments. Using threads
     # is valueless because of the GIL.
     with futures.ProcessPoolExecutor() as executor:
         return executor.map(is_prime, numbers)

A more complex example:

def load_url(url, timeout):
     return urllib.request.urlopen(url, timeout=timeout).read()

### Download the content of some URLs - ignore failures.
def download_urls(urls, timeout=60):
     url_to_content = {}
     for url in urls:
         try:
             url_to_content[url] = load_url(url, timeout=timeout)
         except:
             pass
     return url_to_content

Could be parallelized as:

# Making this a global variable used any many functions ensures that
# the global thread count is kept under control.
executor = futures.ThreadPoolExecutor(50)
def download_urls(urls, timeout=60):
     url_to_content = {}
     # Run load_url for every url and get the result as a FuturesList.
     fs = executor.run_to_futures(
             (functools.partial(load_url, url, timeout) for url in urls),
             timeout=timeout)
     for url, future in zip(urls, fs.successful_futures()):
         url_to_content[url] = future.result()
     return url_to_content


The Python 3.0+ code is here:
http://code.google.com/p/pythonfutures/source

Any feedback on the API would be very much appreciated!

Cheers,
Brian