[stdlib-sig] futures - a new package for asynchronous execution

[Brian Quinlan]

On Nov 7, 2009, at 11:44 AM, Jesse Noller wrote:

> On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> Hey all,
>>
>> I'd like to propose adding a module/package to Python that makes it  
>> easy to
>> parallelize arbitrary function calls.
>> I recently wrote a solution for the use case of parallelizing  
>> network copies
>> and RPC using threads without forcing the user to explicitly  
>> creating thread
>> pools, work queues, etc.
>> I have a concrete implementation that I'll describe below but I'd  
>> be happy
>> to hear about other strategies!
>> The basic idea is to implement an asynchronous execution method  
>> patterned
>> heavily on java.util.concurrent (but less lame because Python has  
>> functions
>> as first-class objects).  Here is a fairly advanced example:
>> import futures
>> import functools
>> import urllib.request
>>
>> URLS = [
>>    'http://www.foxnews.com/',
>>    'http://www.cnn.com/',
>>    'http://europe.wsj.com/',
>>    'http://www.bbc.co.uk/',
>>    'http://some-made-up-domain.com/']
>>
>> def load_url(url, timeout):
>>    return urllib.request.urlopen(url, timeout=timeout).read()
>>
>> # Use a thread pool with 5 threads to download the URLs. Using a pool
>> # of processes would involve changing the initialization to:
>> #   with futures.ProcessPoolExecutor(max_processes=5) as executor
>> with futures.ThreadPoolExecutor(max_threads=5) as executor:
>>    future_list = executor.run_to_futures(
>>        [functools.partial(load_url, url, 30) for url in URLS])
>>
>> # Check the results of each future.
>> for url, future in zip(URLS, future_list):
>>    if future.exception() is not None:
>>        print('%r generated an exception: %s' % (url,  
>> future.exception()))
>>    else:
>>        print('%r page is %d bytes' % (url, len(future.result())))
>>
>> In this example, executor.run_to_futures() returns only when every  
>> url has
>> been retrieved but it is possible to return immediately, on the first
>> completion or on the first failure depending on the desired work  
>> pattern.
>>
>> The complete docs are here:
>> http://sweetapp.com/futures/
>>
>> A draft PEP is here:
>> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>>
>> And the code is here:
>> http://pypi.python.org/pypi/futures3/
>>
>> All feedback appreciated!
>>
>> Cheers,
>> Brian
>> _______________________________________________
>> stdlib-sig mailing list
>> stdlib-sig at python.org
>> http://mail.python.org/mailman/listinfo/stdlib-sig
>>
>>
>
> Hey brian; a few things - I think the code looks good, and your docs
> are really good so far;

Cool, thanks.

> but I'd personally like to see tests

The tests live here:
http://code.google.com/p/pythonfutures/source/browse/trunk/python3/test_futures.py

Last time I measured, there was 100% test coverage (it's a crappy  
metric but an easy one) but I'm not completely happy with them because  
they use time.sleep() in several places to try to provoke deadlock.

> and more
> examples within the docs. I obviously like the concept/idea, but tests
> are a must, and more examples in the docs would make it a lot better.


More examples in the docs sound good, I'll work on that.

Cheers,
Brian