[Python-ideas] async: feedback on EventLoop API

[Jasper St. Pierre]

On Wed, Dec 19, 2012 at 12:36 AM, Guido van Rossum <guido at python.org> wrote:

> On Tue, Dec 18, 2012 at 8:45 PM, Jasper St. Pierre
> <jstpierre at mecheye.net> wrote:
> > I guess this is a good place as any to bring this up, but we really need
> to
> > address issues with error handling and things like par().
> >
> > par() has one way to handle errors: if one task (using it as a general
> term
> > to encompass futures and coroutines) fails, all tasks fail.
> >
> > This is nowhere near acceptable. As a simple example,
> > par(grab_page("http://google.com"), grab_page("http://yahoo.com"))
> should
> > not fail if one of the two sites returns a 500; the results of another
> may
> > still be useful to us.
>
> Yes, there need to be a few variants. If you want all the results,
> regardless of errors, we can provide a variant of par() whose result
> is a list of futures instead of a list of results (or a single
> exception). This could also add a timeout. There also needs to be a
> way to take a set of tasks and wait for the first one to complete. (In
> fact, put a timeout on this and you can build any other variant
> easily.)
>
> PEP 3148 probably shows the way here, it has as_completed() and
> wait(), although we cannot emulate these APIs exactly (since they
> block -- we need something you can use in a yield from, e.g.
>
> fs = {set of Futures}
> while fs:
>     f = yield from wait_one(fs)  # Optionally with a timeout
>     fs.remove(f)
>     <use f>
>
> (We could possibly do the remove() call ih wait_one(), although that
> may limit the argument type to a set.)
>
> > I can think of an approach that doesn't require passing more arguments to
> > par(), but may be absurdly silly: the results generated by par() are not
> > directly results returned by the task, but instead an intermediate
> wrapper
> > value that allows us to hoist the error handling into the caller.
> >
> >     for intermediate in par(*tasks):
> >         try:
> >             result = intermediate.result()
> >         except ValueError as e:
> >             print("bad")
> >         else:
> >             print("good")
> >
> > But this makes the trade-off that you can't immediately cancel all the
> other
> > tasks when one task fails.
>
> Yeah, that's the par() variant that returns futures instead of results.
>
> > The only truly way to be notified when a task has finished, either with
> > success, with error, is a callback, which I think we should flesh out
> > entirely in our Futures model.
>
> Proposal?
>

I'm not sure if this will work out, but I think the par() could have some
sort of "immediate result" callback which fires when one of the sub-tasks
fire. If we then take out the part where we fail and abort automatically,
we might have a close enough approximation:

    def fail_silently(par_task, subtask):
        try:
            return subtask.result()
        except Exception as e:
            print("grabbing failed", e)
            return None

    pages = list(yield par(grab_page("http://google.com"), grab_page("
http://yahoo.com"), subtask_completed=fail_silently))

Where par returns a list of values instead of a list of tasks. But maybe
the ability to manipulate the return value from the subtask completion
callback hands it a bit too much power.

I like the initial approach, but the details need fleshing out. I think it
would be neat if we could have several standard behaviors in the stdlib:
subtask_completed=fail_silently, subtask_completed=abort_task, etc.

> And, of course, we should make sure that we can handle the four situations
> > mentioned in [0] , even if we don't solve them with callbacks.
> >
> > [0] https://gist.github.com/3889970
>
> That's longwinded and written in a confrontational style. Can you
> summarize?
>

Yeah, this was more at a lament at libraries like jQuery that implement the
CommonJS Promise/A specification wrong. It's really only relevant if we
choose to add errbacks, as it's about the composition and sematics between
callbacks/errbacks, and chaining the two.

--
> --Guido van Rossum (python.org/~guido)
>



-- 
  Jasper
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121219/894ebc6c/attachment.html>