I am very happy with those changes.
I think deadlock should be addressed before the first release as it
changes the detailed semantics of some of the operations, but you've
promised to do that, so cool. :)
I think it's fine to leave the embedding of Deferred-like things into
futures and the embedding of futures into Deferred-like things until a
later release. I expect it to be requested, but I don't think it'll be
hard to add later.
On Thu, Nov 12, 2009 at 6:38 PM, Brian Quinlan
Hey all,
I compiled a summary of people's feedback (about technical issues - I agree that the docs could be better but agreeing on the API seems like the first step) and have some API change proposals.
Here is a summary of the feedback: - Use Twisted Deferreds rather than Futures - The API too complex - Make Future a callable and drop the .result()/.exception() methods - Remove .wait() from Executor - Make it easy to process results in the order of completion rather than in the order that the futures were generated - Executor context managers should wait until their workers complete before exiting - Extract Executor.map, etc. into separate functions/modules - FutureList has too many methods or is not necessary - Executor should have an easy way to produce a single future - Should be able to wait on an arbitrary list of futures - Should have a way of avoiding deadlock (will follow-up on this separately)
Here is what I suggest as far as API changes (the docs suck, I'll polish them when we reach consensus):
FutureList is eliminated completely.
Future remains unchanged - I disagree that Deferreds would be better, that .exception() is not useful, and that .result() should be renamed .get() or .__call__(). But I am easily persuadable :-)
The Executor ABC is simplified to only contain a single method:
def Executor.submit(self, fn, *args, **kwargs) :
Submits a call for execution and returns a Future representing the pending results of fn(*args, **kwargs)
map becomes a utility function:
def map(executor, *iterables, timeout=None)
Equivalent to map(func, *iterables) but executed asynchronously and possibly out-of-order. The returned iterator raises a TimeoutError if __next__() is called and the result isn’t available after timeout seconds from the original call to run_to_results(). If timeout is not specified or None then there is no limit to the wait time. If a call raises an exception then that exception will be raised when its value is retrieved from the iterator.
wait becomes a utility function that can wait on any iterable of Futures:
def wait(futures, return_when=ALL_COMPLETED)
Wait until the given condition is met for the given futures. This method should always be called using keyword arguments, which are:
timeout can be used to control the maximum number of seconds to wait before returning. If timeout is not specified or None then there is no limit to the wait time.
return_when indicates when the method should return. It must be one of the following constants:
NEXT_COMPLETED NEXT_EXCEPTION ALL_COMPLETED
a new utility function is added that iterates over the given Futures and returns the as they are completed:
def itercompleted(futures, timeout=None):
Returns an iterator that returns a completed Future from the given list when __next__() is called. If no Futures are completed then __next__() is called then __next__() waits until one does complete. Raises a TimeoutError if __next__() is called and no completed future is available after timeout seconds from the original call.
The URL loading example becomes:
import functools import urllib.request import futures
URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/']
def load_url(url, timeout): return urllib.request.urlopen(url, timeout=timeout).read()
with futures.ThreadPoolExecutor(50) as executor: fs = [executor.submit(load_url, url, timeout=30) for url in URLS]
for future in futures.itercompleted(fs): if future.exception() is not None: print('%r generated an exception: %s' % (url, future.exception())) else: print('%r page is %d bytes' % (url, len(future.result())))
What do you think? Are we moving in the right direction?
Cheers, Brian
_______________________________________________ stdlib-sig mailing list stdlib-sig@python.org http://mail.python.org/mailman/listinfo/stdlib-sig
-- Namasté, Jeffrey Yasskin http://jeffrey.yasskin.info/