Java-style futures in Python - only better

[Brian Quinlan]

Colin J. Williams wrote:
> Brian,
> 
> Since the word "future" is part of the Python lingo:
> 
> A future statement is a directive to the compiler that a particular 
> module should be compiled using syntax or semantics that will be 
> available in a specified future release of Python. The future statement 
> is intended to ease migration to future versions of Python that 
> introduce incompatible changes to the language. It allows use of the new 
> features on a per-module basis before the release in which the feature 
> becomes standard.
> 
> Have you given thought to the use of another word?

I named the module "futures" (plural) to try to reduce the potential 
confusion with the "__futures__" module.

The concept of a future is fairly well known in CS 
[http://en.wikipedia.org/wiki/Future_(programming)] so giving it a 
completely different name would be a bit annoying.

Cheers,
Brian

> Colin W.
> 
> Brian Quinlan wrote:
>> Hey all,
>>
>> I've been working on an Java-style futures implementation in Python. 
>> Futures are a way of representing asynchronous operations e.g. 
>> operations that are run in another thread or process. The are are a 
>> easy but powerful way of parallelizing sequential operations. The also 
>> provide a consistent interface across implementations e.g. they can 
>> provide the same interface to threading and to multiprocessing.
>>
>> For example:
>>
>> def is_prime(n):
>>     "Return True iff n is prime"
>>     ...
>>
>> def check_primes(numbers):
>>     return map(is_prime, numbers)
>>
>> Could be parallelized as:
>>
>> def check_primes(numbers):
>>     # ProcessPoolExecutor will create one worker process
>>     # per CPU if called without arguments. Using threads
>>     # is valueless because of the GIL.
>>     with futures.ProcessPoolExecutor() as executor:
>>         return executor.map(is_prime, numbers)
>>
>> A more complex example:
>>
>> def load_url(url, timeout):
>>     return urllib.request.urlopen(url, timeout=timeout).read()
>>
>> ### Download the content of some URLs - ignore failures.
>> def download_urls(urls, timeout=60):
>>     url_to_content = {}
>>     for url in urls:
>>         try:
>>             url_to_content[url] = load_url(url, timeout=timeout)
>>         except:
>>             pass
>>     return url_to_content
>>
>> Could be parallelized as:
>>
>> # Making this a global variable used any many functions ensures that
>> # the global thread count is kept under control.
>> executor = futures.ThreadPoolExecutor(50)
>> def download_urls(urls, timeout=60):
>>     url_to_content = {}
>>     # Run load_url for every url and get the result as a FuturesList.
>>     fs = executor.run_to_futures(
>>             (functools.partial(load_url, url, timeout) for url in urls),
>>             timeout=timeout)
>>     for url, future in zip(urls, fs.successful_futures()):
>>         url_to_content[url] = future.result()
>>     return url_to_content
>>
>>
>> The Python 3.0+ code is here:
>> http://code.google.com/p/pythonfutures/source
>>
>> Any feedback on the API would be very much appreciated!
>>
>> Cheers,
>> Brian
>>
>>