[stdlib-sig] futures - a new package for asynchronous execution
jnoller at gmail.com
Sat Nov 7 01:44:46 CET 2009
On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.
> I recently wrote a solution for the use case of parallelizing network copies
> and RPC using threads without forcing the user to explicitly creating thread
> pools, work queues, etc.
> I have a concrete implementation that I'll describe below but I'd be happy
> to hear about other strategies!
> The basic idea is to implement an asynchronous execution method patterned
> heavily on java.util.concurrent (but less lame because Python has functions
> as first-class objects). Here is a fairly advanced example:
> import futures
> import functools
> import urllib.request
> URLS = [
> def load_url(url, timeout):
> return urllib.request.urlopen(url, timeout=timeout).read()
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> # with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:
> future_list = executor.run_to_futures(
> [functools.partial(load_url, url, 30) for url in URLS])
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
> if future.exception() is not None:
> print('%r generated an exception: %s' % (url, future.exception()))
> print('%r page is %d bytes' % (url, len(future.result())))
> In this example, executor.run_to_futures() returns only when every url has
> been retrieved but it is possible to return immediately, on the first
> completion or on the first failure depending on the desired work pattern.
> The complete docs are here:
> A draft PEP is here:
> And the code is here:
> All feedback appreciated!
> stdlib-sig mailing list
> stdlib-sig at python.org
Hey brian; a few things - I think the code looks good, and your docs
are really good so far; but I'd personally like to see tests and more
examples within the docs. I obviously like the concept/idea, but tests
are a must, and more examples in the docs would make it a lot better.
More information about the stdlib-sig