Ignoring the API overlap with Futures/Promises for a moment, let me throw out this straw man approach to the event loop that seems to my naive eye like it pull together a lot of these ideas… Rather than passing in your callbacks, factories, etc., asynchronous APIs return a lightweight object you register your callback with. Unlike promises, deferrers, etc., this is a one-time thing: only one callback can register with it. However, it can be chained. The registered callback is invoked with the output of the operation when it completes. Timer.wait(20).then(callme, *args, **kw) # I could do Timer.wait(20).then(callme, *args, **kw).then(piped_from_callme) #I could not do handler = Timer.wait(20) handler.then(callme) handler.then(callme2) # this would throw an exception. # I/O example… sock.accept().then(handle_connection) # invokes handle_connection(conn, addr) # Read some data conn.read(1024).then(handle_incoming) # handle_incoming invoked with up to 1024 bytes, read asynchronously. # Write some data conn.write("data").then(handle_written) # handle_written invoked with up number 5, giving number of bytes written async. # Connect HTTP channel and add it to HTTP dispatcher. channel.connect((hostname,80)).then(dispatcher.add_channel) # Listen to FD's for I/O events descriptors.select(r, w, e).then(handle) # handle(readable, writables, oobs) It seems like only supporting a single callback per returned handle lets us circumvent a lot of the weight associated with normal promise/future/deferred pattern type implementations, but the chaining could come in handy as it may cover some of the use-cases being considered when multiple events per fd came up, plus chaining is pretty powerful, especially when it comes at little cost. The API would be much more extensive than "then()", of course, with things like "every", etc. we'd have to pull examples from everything already discussed. Just wanted to throw out there to get beat up about ;-) Shane Green www.umbrellacode.com 805-452-9666 | shane@umbrellacode.com
A lot of things become trivially easy if they assumption is that they can never fail. Deferreds/Promises/Tasks/Futures are about sane error handling, not sane success handling. (There's a few parts in the current proposal where this falls short, like par, but that's another post) On Tue, Dec 18, 2012 at 10:44 PM, Shane Green <shane@umbrellacode.com>wrote:
-- Jasper
Oh, I forgot error-backs. True, though, error-handling is a bit more difficult. I'm not sure I see it as being much more challenging that asynchronous callback error handling/reporting in general, though: it can still be executed in the exception context, etc. And the chaining can even be used to attach extended error logging to all your callback chains, without losing or swallowing anything that wouldn't have been lost or swallowed by other approaches, unless I'm overlooking something. Shane Green www.umbrellacode.com 805-452-9666 | shane@umbrellacode.com On Dec 18, 2012, at 7:51 PM, "Jasper St. Pierre" <jstpierre@mecheye.net> wrote:
The point of PEP 3156 is not to make using callbacks easy. It is to make callbacks mostly disappear in favor of coroutines, but keeping them around in order to provide interoperability with callback-based frameworks such as Twisted or Tornado. Your handlers appear to be an attempt at reinventing Twisted's Deferred. But Deferred already exists, and it works perfectly fine with the current callback-based event loop spec in the PEP. It's not clear how your handlers will enable a coroutine to wait for the result (or exception) however. --Guido On Tue, Dec 18, 2012 at 7:44 PM, Shane Green <shane@umbrellacode.com> wrote:
-- --Guido van Rossum (python.org/~guido)
I guess this is a good place as any to bring this up, but we really need to address issues with error handling and things like par(). par() has one way to handle errors: if one task (using it as a general term to encompass futures and coroutines) fails, all tasks fail. This is nowhere near acceptable. As a simple example, par(grab_page(" http://google.com"), grab_page("http://yahoo.com")) should not fail if one of the two sites returns a 500; the results of another may still be useful to us. I can think of an approach that doesn't require passing more arguments to par(), but may be absurdly silly: the results generated by par() are not directly results returned by the task, but instead an intermediate wrapper value that allows us to hoist the error handling into the caller. for intermediate in par(*tasks): try: result = intermediate.result() except ValueError as e: print("bad") else: print("good") But this makes the trade-off that you can't immediately cancel all the other tasks when one task fails. The only truly way to be notified when a task has finished, either with success, with error, is a callback, which I think we should flesh out entirely in our Futures model. And, of course, we should make sure that we can handle the four situations mentioned in [0] , even if we don't solve them with callbacks. [0] https://gist.github.com/3889970 On Tue, Dec 18, 2012 at 11:28 PM, Guido van Rossum <guido@python.org> wrote:
-- Jasper
On Tue, Dec 18, 2012 at 8:45 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Yes, there need to be a few variants. If you want all the results, regardless of errors, we can provide a variant of par() whose result is a list of futures instead of a list of results (or a single exception). This could also add a timeout. There also needs to be a way to take a set of tasks and wait for the first one to complete. (In fact, put a timeout on this and you can build any other variant easily.) PEP 3148 probably shows the way here, it has as_completed() and wait(), although we cannot emulate these APIs exactly (since they block -- we need something you can use in a yield from, e.g. fs = {set of Futures} while fs: f = yield from wait_one(fs) # Optionally with a timeout fs.remove(f) <use f> (We could possibly do the remove() call ih wait_one(), although that may limit the argument type to a set.)
Yeah, that's the par() variant that returns futures instead of results.
Proposal?
That's longwinded and written in a confrontational style. Can you summarize? -- --Guido van Rossum (python.org/~guido)
On Wed, Dec 19, 2012 at 12:36 AM, Guido van Rossum <guido@python.org> wrote:
I'm not sure if this will work out, but I think the par() could have some sort of "immediate result" callback which fires when one of the sub-tasks fire. If we then take out the part where we fail and abort automatically, we might have a close enough approximation: def fail_silently(par_task, subtask): try: return subtask.result() except Exception as e: print("grabbing failed", e) return None pages = list(yield par(grab_page("http://google.com"), grab_page(" http://yahoo.com"), subtask_completed=fail_silently)) Where par returns a list of values instead of a list of tasks. But maybe the ability to manipulate the return value from the subtask completion callback hands it a bit too much power. I like the initial approach, but the details need fleshing out. I think it would be neat if we could have several standard behaviors in the stdlib: subtask_completed=fail_silently, subtask_completed=abort_task, etc.
Yeah, this was more at a lament at libraries like jQuery that implement the CommonJS Promise/A specification wrong. It's really only relevant if we choose to add errbacks, as it's about the composition and sematics between callbacks/errbacks, and chaining the two. --
--Guido van Rossum (python.org/~guido)
-- Jasper
On Tuesday, December 18, 2012, Jasper St. Pierre wrote:
That looks reasonable too, although the signature may need to be adjusted. (How does it cancel the remaining tasks if it wants to? Or does par() do that if this callback raises?) maybe call it filter? But what did you think of my wait_one() proposal? It may work beter in a coroutine, where callbacks are considered a nuisance.
No, no, no! Please. No errbacks. No chaining. Coroutines have a different way to spell those already: errbacks -> except clauses, chaining -> multiple yield-froms in one coroutine, or call another coroutine. Please. --Guido -- --Guido van Rossum (on iPad)
On Wed, Dec 19, 2012 at 1:24 AM, Guido van Rossum <guido@python.org> wrote: ... snip ... That looks reasonable too, although the signature may need to be adjusted.
(How does it cancel the remaining tasks if it wants to? Or does par() do that if this callback raises?) maybe call it filter?
The subtask completion callback can call abort() on the overall par_task, which could cancel the rest of the unfinished tasks. def abort_task(par_task, subtask): try: return subtask.result() except ValueError: par_task.abort() The issue with this approach is that since the par() would return values again, not tasks, we'd can't handle errors locally. Futures are also immutable, so we can't modify the values after they resolve. Maybe we'd have something like: def fail_silently(par_task, subtask): try: subtask.result() except ValueError as e: return Future.completed(None) # an already completed future that has a value of None, sorry, don't remember the exact spelling else: return subtask which allows us: for task in par(*tasks, subtask_completion=fail_silently): # ... Which allows us both local error handling, as well as batch error handling. But it's very verbose from the side of the callback. Hm.
But what did you think of my wait_one() proposal? It may work beter in a coroutine, where callbacks are considered a nuisance.
To be honest, I didn't quite understand it. I'd have to go back and re-read PEP 3148.
-- Jasper
On Tue, Dec 18, 2012 at 10:41 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Tasks don't have abort(), I suppose you meant cancel().
Hm indeed. Unless you can get your thoughts straight I think I'd rather go with the wait_one() API, which can be used to build anything else you like, but doesn't require one to be quite so clever with callbacks. (Did I say I hate callbacks?) -- --Guido van Rossum (python.org/~guido)
I read over the wait_one() proposal again, and I still don't understand it, so it would need more explanation to me. But I don't see the point of avoiding callbacks. In this case, we have two or more in-flight requests that can be finished at any time. This does not have a synchronous code equivalent -- callbacks are pretty much the only mechanism we can use to be notified when something is done. On Wed, Dec 19, 2012 at 12:26 PM, Guido van Rossum <guido@python.org> wrote:
-- Jasper
On Fri, Dec 21, 2012 at 3:38 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Perhaps you haven't quite gotten used to coroutines? There are callbacks underneath making it all work, but the user code rarely sees those. Let's start with the following *synchronous* code as an example. def indexer(urls): # urls is a set of strings done = {} # dict mapping url to (data, links) while urls: data = urlfetch(url.pop()) links = parse(data) done[url] = (data, links) for link in link: if link not in urls and link not in done: urls.add(link) return done (Let's hope this is indexing a small static site and not the entire internet. :-) Now suppose we make urlfetch() a coroutine and we want to run all the urlfetches in parallel. The toplevel index() function becomes a coroutine too. We use the convention that coroutines' names end in _async, to remind us that they return Futures. The phrase "x = yield from foo_async()" is equivalent to the synchronous call "x = foo()". @coroutine def indexer_async(urls): done = {} # A dict mapping tasks to urls: running = {Task(urlfetch_async(url)), url for url in urls} while running: # The yield from will return a Future tsk = *yield from* wait_one_async(running) url = running.pop(tsk) data = tsk.result() # May raise links = parse(data) done[url] = (data, links) for link in links: if link not in urls and link not in done: urls.add(link) tsk = Task(urlfetch_async(link) running[tsk] = link return done This creates len(urls) initial tasks to parse the urls, and creates new urls as new links are parsed. The assumption here is that the only blocking I/O is done in the urlfetch_async() task. The indexer blocks at the *yield from* in the marked line, at which point any or all of the urlfetch tasks get to run some, and once one of them completes, wait_one_async() returns that task. (A task is a Future that wraps a coroutine, by the way. wait_one_async() works with Futures too.) We then inspect the completed task with .result(), which gives us the data, which we parse as usual. The data structures are a little more elaborate because we have to keep track of the mapping from task to url. We add new tasks to the running dict as soon as we have parsed their links, so they can all get started. Note that in PEP 3156, I don't use the _async convention, but everything in this example will work there once wait_one() is added. Also note that the trick is that wait_one_async() must return a Future whose result is another Future. The first Future is used (and thrown away) by *yield from*; that Future's result is one of the original Futures representing a completed task. I hope this is clearer. I'm not saying this is the best or only way of writing an async indexer using yield from (and I left out error handling) but hopefully it is an illustrative example. -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 10:45 AM, Guido van Rossum <guido@python.org> wrote: ... snip ... (gmail messed up parsing this, apparently) Aha, that cleared it up, thanks. wait_one_async() takes an iterable of tasks, and returns a Future that will fire when a Future completes, containing that Future. I can't think of anything *wrong* with that, except that if anything, 1) it feels like a bit of an abuse to use Futures this way, 2) it feels a bit low-level. -- Jasper
On Fri, Dec 21, 2012 at 7:57 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
But not more low-level than callbacks. Once you're used to coroutines and Futures, you don't want things that use callbacks. Fortunately there's an easy way to turn a callback into a Future: f = Future() old_style_async(callback=f.set_result) result = yield from f Assuming old_style_async() calls its callback with one arg, a useful result, that result will now end up in the variable 'result'. If this happens a lot it's easy to wrap it in a helper function, so you can write: result = yield from wrap_in_future(old_style_async) -- --Guido van Rossum (python.org/~guido)
Ah, I see. I did not read though the PEP like I should have. Given that I didn't do my homework, it would be an awesome coincidence if they enabled a coroutine to wait for the result (or exception) ;-) Shane Green www.umbrellacode.com 805-452-9666 | shane@umbrellacode.com On Dec 18, 2012, at 8:28 PM, Guido van Rossum <guido@python.org> wrote:
A lot of things become trivially easy if they assumption is that they can never fail. Deferreds/Promises/Tasks/Futures are about sane error handling, not sane success handling. (There's a few parts in the current proposal where this falls short, like par, but that's another post) On Tue, Dec 18, 2012 at 10:44 PM, Shane Green <shane@umbrellacode.com>wrote:
-- Jasper
Oh, I forgot error-backs. True, though, error-handling is a bit more difficult. I'm not sure I see it as being much more challenging that asynchronous callback error handling/reporting in general, though: it can still be executed in the exception context, etc. And the chaining can even be used to attach extended error logging to all your callback chains, without losing or swallowing anything that wouldn't have been lost or swallowed by other approaches, unless I'm overlooking something. Shane Green www.umbrellacode.com 805-452-9666 | shane@umbrellacode.com On Dec 18, 2012, at 7:51 PM, "Jasper St. Pierre" <jstpierre@mecheye.net> wrote:
The point of PEP 3156 is not to make using callbacks easy. It is to make callbacks mostly disappear in favor of coroutines, but keeping them around in order to provide interoperability with callback-based frameworks such as Twisted or Tornado. Your handlers appear to be an attempt at reinventing Twisted's Deferred. But Deferred already exists, and it works perfectly fine with the current callback-based event loop spec in the PEP. It's not clear how your handlers will enable a coroutine to wait for the result (or exception) however. --Guido On Tue, Dec 18, 2012 at 7:44 PM, Shane Green <shane@umbrellacode.com> wrote:
-- --Guido van Rossum (python.org/~guido)
I guess this is a good place as any to bring this up, but we really need to address issues with error handling and things like par(). par() has one way to handle errors: if one task (using it as a general term to encompass futures and coroutines) fails, all tasks fail. This is nowhere near acceptable. As a simple example, par(grab_page(" http://google.com"), grab_page("http://yahoo.com")) should not fail if one of the two sites returns a 500; the results of another may still be useful to us. I can think of an approach that doesn't require passing more arguments to par(), but may be absurdly silly: the results generated by par() are not directly results returned by the task, but instead an intermediate wrapper value that allows us to hoist the error handling into the caller. for intermediate in par(*tasks): try: result = intermediate.result() except ValueError as e: print("bad") else: print("good") But this makes the trade-off that you can't immediately cancel all the other tasks when one task fails. The only truly way to be notified when a task has finished, either with success, with error, is a callback, which I think we should flesh out entirely in our Futures model. And, of course, we should make sure that we can handle the four situations mentioned in [0] , even if we don't solve them with callbacks. [0] https://gist.github.com/3889970 On Tue, Dec 18, 2012 at 11:28 PM, Guido van Rossum <guido@python.org> wrote:
-- Jasper
On Tue, Dec 18, 2012 at 8:45 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Yes, there need to be a few variants. If you want all the results, regardless of errors, we can provide a variant of par() whose result is a list of futures instead of a list of results (or a single exception). This could also add a timeout. There also needs to be a way to take a set of tasks and wait for the first one to complete. (In fact, put a timeout on this and you can build any other variant easily.) PEP 3148 probably shows the way here, it has as_completed() and wait(), although we cannot emulate these APIs exactly (since they block -- we need something you can use in a yield from, e.g. fs = {set of Futures} while fs: f = yield from wait_one(fs) # Optionally with a timeout fs.remove(f) <use f> (We could possibly do the remove() call ih wait_one(), although that may limit the argument type to a set.)
Yeah, that's the par() variant that returns futures instead of results.
Proposal?
That's longwinded and written in a confrontational style. Can you summarize? -- --Guido van Rossum (python.org/~guido)
On Wed, Dec 19, 2012 at 12:36 AM, Guido van Rossum <guido@python.org> wrote:
I'm not sure if this will work out, but I think the par() could have some sort of "immediate result" callback which fires when one of the sub-tasks fire. If we then take out the part where we fail and abort automatically, we might have a close enough approximation: def fail_silently(par_task, subtask): try: return subtask.result() except Exception as e: print("grabbing failed", e) return None pages = list(yield par(grab_page("http://google.com"), grab_page(" http://yahoo.com"), subtask_completed=fail_silently)) Where par returns a list of values instead of a list of tasks. But maybe the ability to manipulate the return value from the subtask completion callback hands it a bit too much power. I like the initial approach, but the details need fleshing out. I think it would be neat if we could have several standard behaviors in the stdlib: subtask_completed=fail_silently, subtask_completed=abort_task, etc.
Yeah, this was more at a lament at libraries like jQuery that implement the CommonJS Promise/A specification wrong. It's really only relevant if we choose to add errbacks, as it's about the composition and sematics between callbacks/errbacks, and chaining the two. --
--Guido van Rossum (python.org/~guido)
-- Jasper
On Tuesday, December 18, 2012, Jasper St. Pierre wrote:
That looks reasonable too, although the signature may need to be adjusted. (How does it cancel the remaining tasks if it wants to? Or does par() do that if this callback raises?) maybe call it filter? But what did you think of my wait_one() proposal? It may work beter in a coroutine, where callbacks are considered a nuisance.
No, no, no! Please. No errbacks. No chaining. Coroutines have a different way to spell those already: errbacks -> except clauses, chaining -> multiple yield-froms in one coroutine, or call another coroutine. Please. --Guido -- --Guido van Rossum (on iPad)
On Wed, Dec 19, 2012 at 1:24 AM, Guido van Rossum <guido@python.org> wrote: ... snip ... That looks reasonable too, although the signature may need to be adjusted.
(How does it cancel the remaining tasks if it wants to? Or does par() do that if this callback raises?) maybe call it filter?
The subtask completion callback can call abort() on the overall par_task, which could cancel the rest of the unfinished tasks. def abort_task(par_task, subtask): try: return subtask.result() except ValueError: par_task.abort() The issue with this approach is that since the par() would return values again, not tasks, we'd can't handle errors locally. Futures are also immutable, so we can't modify the values after they resolve. Maybe we'd have something like: def fail_silently(par_task, subtask): try: subtask.result() except ValueError as e: return Future.completed(None) # an already completed future that has a value of None, sorry, don't remember the exact spelling else: return subtask which allows us: for task in par(*tasks, subtask_completion=fail_silently): # ... Which allows us both local error handling, as well as batch error handling. But it's very verbose from the side of the callback. Hm.
But what did you think of my wait_one() proposal? It may work beter in a coroutine, where callbacks are considered a nuisance.
To be honest, I didn't quite understand it. I'd have to go back and re-read PEP 3148.
-- Jasper
On Tue, Dec 18, 2012 at 10:41 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Tasks don't have abort(), I suppose you meant cancel().
Hm indeed. Unless you can get your thoughts straight I think I'd rather go with the wait_one() API, which can be used to build anything else you like, but doesn't require one to be quite so clever with callbacks. (Did I say I hate callbacks?) -- --Guido van Rossum (python.org/~guido)
I read over the wait_one() proposal again, and I still don't understand it, so it would need more explanation to me. But I don't see the point of avoiding callbacks. In this case, we have two or more in-flight requests that can be finished at any time. This does not have a synchronous code equivalent -- callbacks are pretty much the only mechanism we can use to be notified when something is done. On Wed, Dec 19, 2012 at 12:26 PM, Guido van Rossum <guido@python.org> wrote:
-- Jasper
On Fri, Dec 21, 2012 at 3:38 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Perhaps you haven't quite gotten used to coroutines? There are callbacks underneath making it all work, but the user code rarely sees those. Let's start with the following *synchronous* code as an example. def indexer(urls): # urls is a set of strings done = {} # dict mapping url to (data, links) while urls: data = urlfetch(url.pop()) links = parse(data) done[url] = (data, links) for link in link: if link not in urls and link not in done: urls.add(link) return done (Let's hope this is indexing a small static site and not the entire internet. :-) Now suppose we make urlfetch() a coroutine and we want to run all the urlfetches in parallel. The toplevel index() function becomes a coroutine too. We use the convention that coroutines' names end in _async, to remind us that they return Futures. The phrase "x = yield from foo_async()" is equivalent to the synchronous call "x = foo()". @coroutine def indexer_async(urls): done = {} # A dict mapping tasks to urls: running = {Task(urlfetch_async(url)), url for url in urls} while running: # The yield from will return a Future tsk = *yield from* wait_one_async(running) url = running.pop(tsk) data = tsk.result() # May raise links = parse(data) done[url] = (data, links) for link in links: if link not in urls and link not in done: urls.add(link) tsk = Task(urlfetch_async(link) running[tsk] = link return done This creates len(urls) initial tasks to parse the urls, and creates new urls as new links are parsed. The assumption here is that the only blocking I/O is done in the urlfetch_async() task. The indexer blocks at the *yield from* in the marked line, at which point any or all of the urlfetch tasks get to run some, and once one of them completes, wait_one_async() returns that task. (A task is a Future that wraps a coroutine, by the way. wait_one_async() works with Futures too.) We then inspect the completed task with .result(), which gives us the data, which we parse as usual. The data structures are a little more elaborate because we have to keep track of the mapping from task to url. We add new tasks to the running dict as soon as we have parsed their links, so they can all get started. Note that in PEP 3156, I don't use the _async convention, but everything in this example will work there once wait_one() is added. Also note that the trick is that wait_one_async() must return a Future whose result is another Future. The first Future is used (and thrown away) by *yield from*; that Future's result is one of the original Futures representing a completed task. I hope this is clearer. I'm not saying this is the best or only way of writing an async indexer using yield from (and I left out error handling) but hopefully it is an illustrative example. -- --Guido van Rossum (python.org/~guido)
On Fri, Dec 21, 2012 at 10:45 AM, Guido van Rossum <guido@python.org> wrote: ... snip ... (gmail messed up parsing this, apparently) Aha, that cleared it up, thanks. wait_one_async() takes an iterable of tasks, and returns a Future that will fire when a Future completes, containing that Future. I can't think of anything *wrong* with that, except that if anything, 1) it feels like a bit of an abuse to use Futures this way, 2) it feels a bit low-level. -- Jasper
On Fri, Dec 21, 2012 at 7:57 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
But not more low-level than callbacks. Once you're used to coroutines and Futures, you don't want things that use callbacks. Fortunately there's an easy way to turn a callback into a Future: f = Future() old_style_async(callback=f.set_result) result = yield from f Assuming old_style_async() calls its callback with one arg, a useful result, that result will now end up in the variable 'result'. If this happens a lot it's easy to wrap it in a helper function, so you can write: result = yield from wrap_in_future(old_style_async) -- --Guido van Rossum (python.org/~guido)
Ah, I see. I did not read though the PEP like I should have. Given that I didn't do my homework, it would be an awesome coincidence if they enabled a coroutine to wait for the result (or exception) ;-) Shane Green www.umbrellacode.com 805-452-9666 | shane@umbrellacode.com On Dec 18, 2012, at 8:28 PM, Guido van Rossum <guido@python.org> wrote:
participants (3)
-
Guido van Rossum -
Jasper St. Pierre -
Shane Green