[Python-ideas] async: feedback on EventLoop API

[Guido van Rossum]

On Tue, Dec 18, 2012 at 8:45 PM, Jasper St. Pierre
<jstpierre at mecheye.net> wrote:
> I guess this is a good place as any to bring this up, but we really need to
> address issues with error handling and things like par().
>
> par() has one way to handle errors: if one task (using it as a general term
> to encompass futures and coroutines) fails, all tasks fail.
>
> This is nowhere near acceptable. As a simple example,
> par(grab_page("http://google.com"), grab_page("http://yahoo.com")) should
> not fail if one of the two sites returns a 500; the results of another may
> still be useful to us.

Yes, there need to be a few variants. If you want all the results,
regardless of errors, we can provide a variant of par() whose result
is a list of futures instead of a list of results (or a single
exception). This could also add a timeout. There also needs to be a
way to take a set of tasks and wait for the first one to complete. (In
fact, put a timeout on this and you can build any other variant
easily.)

PEP 3148 probably shows the way here, it has as_completed() and
wait(), although we cannot emulate these APIs exactly (since they
block -- we need something you can use in a yield from, e.g.

fs = {set of Futures}
while fs:
    f = yield from wait_one(fs)  # Optionally with a timeout
    fs.remove(f)
    <use f>

(We could possibly do the remove() call ih wait_one(), although that
may limit the argument type to a set.)

> I can think of an approach that doesn't require passing more arguments to
> par(), but may be absurdly silly: the results generated by par() are not
> directly results returned by the task, but instead an intermediate wrapper
> value that allows us to hoist the error handling into the caller.
>
>     for intermediate in par(*tasks):
>         try:
>             result = intermediate.result()
>         except ValueError as e:
>             print("bad")
>         else:
>             print("good")
>
> But this makes the trade-off that you can't immediately cancel all the other
> tasks when one task fails.

Yeah, that's the par() variant that returns futures instead of results.

> The only truly way to be notified when a task has finished, either with
> success, with error, is a callback, which I think we should flesh out
> entirely in our Futures model.

Proposal?

> And, of course, we should make sure that we can handle the four situations
> mentioned in [0] , even if we don't solve them with callbacks.
>
> [0] https://gist.github.com/3889970

That's longwinded and written in a confrontational style. Can you summarize?

-- 
--Guido van Rossum (python.org/~guido)