<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hey all,<br><br><div>I'd like to propose adding a module/package to Python that makes it easy to parallelize arbitrary function calls.</div><div><br></div><div>I recently wrote a solution for the use case of parallelizing network copies and RPC using threads without forcing the user to explicitly creating thread pools, work queues, etc.</div><div><br></div><div>I have a concrete implementation that I'll describe below but I'd be happy to hear about other strategies!</div><div><br></div><div>The basic idea is to implement an asynchronous execution method patterned heavily on java.util.concurrent (but less lame because Python has functions as first-class objects). Here is a fairly advanced example:</div><div><br>import futures<br>import functools<br>import urllib.request<br><br>URLS = [<br> '<a href="http://www.foxnews.com/'">http://www.foxnews.com/'</a>,<br> '<a href="http://www.cnn.com/'">http://www.cnn.com/'</a>,<br> '<a href="http://europe.wsj.com/'">http://europe.wsj.com/'</a>,<br> '<a href="http://www.bbc.co.uk/'">http://www.bbc.co.uk/'</a>,<br> '<a href="http://some-made-up-domain.com/'">http://some-made-up-domain.com/'</a>]<br><br>def load_url(url, timeout):<br> return urllib.request.urlopen(url, timeout=timeout).read()<br><br># Use a thread pool with 5 threads to download the URLs. Using a pool<br># of processes would involve changing the initialization to:<br># with futures.ProcessPoolExecutor(max_processes=5) as executor<br>with futures.ThreadPoolExecutor(max_threads=5) as executor:<br> future_list = executor.run_to_futures(<br> [functools.partial(load_url, url, 30) for url in URLS])<br><br># Check the results of each future.<br>for url, future in zip(URLS, future_list):<br> if future.exception() is not None:<br> print('%r generated an exception: %s' % (url, future.exception()))<br> else:<br> print('%r page is %d bytes' % (url, len(future.result())))<br><br>In this example, executor.run_to_futures() returns only when every url has been retrieved but it is possible to return immediately, on the first completion or on the first failure depending on the desired work pattern.<br><br>The complete docs are here:<br><a href="http://sweetapp.com/futures/">http://sweetapp.com/futures/</a><br><br>A draft PEP is here:<br><a href="http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt">http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt</a><br><br>And the code is here:<br><a href="http://pypi.python.org/pypi/futures3/">http://pypi.python.org/pypi/futures3/</a><br><br>All feedback appreciated!<br><br>Cheers,<br>Brian</div></body></html>