[stdlib-sig] futures - a new package for asynchronous execution

[Jesse Noller]

On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.
> I recently wrote a solution for the use case of parallelizing network copies
> and RPC using threads without forcing the user to explicitly creating thread
> pools, work queues, etc.
> I have a concrete implementation that I'll describe below but I'd be happy
> to hear about other strategies!
> The basic idea is to implement an asynchronous execution method patterned
> heavily on java.util.concurrent (but less lame because Python has functions
> as first-class objects).  Here is a fairly advanced example:
> import futures
> import functools
> import urllib.request
>
> URLS = [
>    'http://www.foxnews.com/',
>    'http://www.cnn.com/',
>    'http://europe.wsj.com/',
>    'http://www.bbc.co.uk/',
>    'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
>
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> #   with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:
>    future_list = executor.run_to_futures(
>        [functools.partial(load_url, url, 30) for url in URLS])
>
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
>    if future.exception() is not None:
>        print('%r generated an exception: %s' % (url, future.exception()))
>    else:
>        print('%r page is %d bytes' % (url, len(future.result())))
>
> In this example, executor.run_to_futures() returns only when every url has
> been retrieved but it is possible to return immediately, on the first
> completion or on the first failure depending on the desired work pattern.
>
> The complete docs are here:
> http://sweetapp.com/futures/
>
> A draft PEP is here:
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>
> And the code is here:
> http://pypi.python.org/pypi/futures3/
>
> All feedback appreciated!
>
> Cheers,
> Brian
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>
>

Hey brian; a few things - I think the code looks good, and your docs
are really good so far; but I'd personally like to see tests and more
examples within the docs. I obviously like the concept/idea, but tests
are a must, and more examples in the docs would make it a lot better.

jesse