[stdlib-sig] futures - a new package for asynchronous execution
brian at sweetapp.com
Sat Nov 7 06:11:27 CET 2009
On Nov 7, 2009, at 11:44 AM, Jesse Noller wrote:
> On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com>
>> Hey all,
>> I'd like to propose adding a module/package to Python that makes it
>> easy to
>> parallelize arbitrary function calls.
>> I recently wrote a solution for the use case of parallelizing
>> network copies
>> and RPC using threads without forcing the user to explicitly
>> creating thread
>> pools, work queues, etc.
>> I have a concrete implementation that I'll describe below but I'd
>> be happy
>> to hear about other strategies!
>> The basic idea is to implement an asynchronous execution method
>> heavily on java.util.concurrent (but less lame because Python has
>> as first-class objects). Here is a fairly advanced example:
>> import futures
>> import functools
>> import urllib.request
>> URLS = [
>> def load_url(url, timeout):
>> return urllib.request.urlopen(url, timeout=timeout).read()
>> # Use a thread pool with 5 threads to download the URLs. Using a pool
>> # of processes would involve changing the initialization to:
>> # with futures.ProcessPoolExecutor(max_processes=5) as executor
>> with futures.ThreadPoolExecutor(max_threads=5) as executor:
>> future_list = executor.run_to_futures(
>> [functools.partial(load_url, url, 30) for url in URLS])
>> # Check the results of each future.
>> for url, future in zip(URLS, future_list):
>> if future.exception() is not None:
>> print('%r generated an exception: %s' % (url,
>> print('%r page is %d bytes' % (url, len(future.result())))
>> In this example, executor.run_to_futures() returns only when every
>> url has
>> been retrieved but it is possible to return immediately, on the first
>> completion or on the first failure depending on the desired work
>> The complete docs are here:
>> A draft PEP is here:
>> And the code is here:
>> All feedback appreciated!
>> stdlib-sig mailing list
>> stdlib-sig at python.org
> Hey brian; a few things - I think the code looks good, and your docs
> are really good so far;
> but I'd personally like to see tests
The tests live here:
Last time I measured, there was 100% test coverage (it's a crappy
metric but an easy one) but I'm not completely happy with them because
they use time.sleep() in several places to try to provoke deadlock.
> and more
> examples within the docs. I obviously like the concept/idea, but tests
> are a must, and more examples in the docs would make it a lot better.
More examples in the docs sound good, I'll work on that.
More information about the stdlib-sig