[Python-ideas] async: feedback on EventLoop API

[Guido van Rossum]

On Tuesday, December 18, 2012, Jasper St. Pierre wrote:

> On Wed, Dec 19, 2012 at 12:36 AM, Guido van Rossum <guido at python.org<javascript:_e({}, 'cvml', 'guido at python.org');>
> > wrote:
>
>> On Tue, Dec 18, 2012 at 8:45 PM, Jasper St. Pierre
>> <jstpierre at mecheye.net <javascript:_e({}, 'cvml',
>> 'jstpierre at mecheye.net');>> wrote:
>> > I guess this is a good place as any to bring this up, but we really
>> need to
>> > address issues with error handling and things like par().
>> >
>> > par() has one way to handle errors: if one task (using it as a general
>> term
>> > to encompass futures and coroutines) fails, all tasks fail.
>> >
>> > This is nowhere near acceptable. As a simple example,
>> > par(grab_page("http://google.com"), grab_page("http://yahoo.com"))
>> should
>> > not fail if one of the two sites returns a 500; the results of another
>> may
>> > still be useful to us.
>>
>> Yes, there need to be a few variants. If you want all the results,
>> regardless of errors, we can provide a variant of par() whose result
>> is a list of futures instead of a list of results (or a single
>> exception). This could also add a timeout. There also needs to be a
>> way to take a set of tasks and wait for the first one to complete. (In
>> fact, put a timeout on this and you can build any other variant
>> easily.)
>>
>> PEP 3148 probably shows the way here, it has as_completed() and
>> wait(), although we cannot emulate these APIs exactly (since they
>> block -- we need something you can use in a yield from, e.g.
>>
>> fs = {set of Futures}
>> while fs:
>>     f = yield from wait_one(fs)  # Optionally with a timeout
>>     fs.remove(f)
>>     <use f>
>>
>> (We could possibly do the remove() call ih wait_one(), although that
>> may limit the argument type to a set.)
>>
>> > I can think of an approach that doesn't require passing more arguments
>> to
>> > par(), but may be absurdly silly: the results generated by par() are not
>> > directly results returned by the task, but instead an intermediate
>> wrapper
>> > value that allows us to hoist the error handling into the caller.
>> >
>> >     for intermediate in par(*tasks):
>> >         try:
>> >             result = intermediate.result()
>> >         except ValueError as e:
>> >             print("bad")
>> >         else:
>> >             print("good")
>> >
>> > But this makes the trade-off that you can't immediately cancel all the
>> other
>> > tasks when one task fails.
>>
>> Yeah, that's the par() variant that returns futures instead of results.
>>
>> > The only truly way to be notified when a task has finished, either with
>> > success, with error, is a callback, which I think we should flesh out
>> > entirely in our Futures model.
>>
>> Proposal?
>>
>
> I'm not sure if this will work out, but I think the par() could have some
> sort of "immediate result" callback which fires when one of the sub-tasks
> fire. If we then take out the part where we fail and abort automatically,
> we might have a close enough approximation:
>
>     def fail_silently(par_task, subtask):
>         try:
>             return subtask.result()
>         except Exception as e:
>             print("grabbing failed", e)
>             return None
>
>     pages = list(yield par(grab_page("http://google.com"), grab_page("
> http://yahoo.com"), subtask_completed=fail_silently))
>
> Where par returns a list of values instead of a list of tasks. But maybe
> the ability to manipulate the return value from the subtask completion
> callback hands it a bit too much power.
>

That looks reasonable too, although the signature may need to be adjusted.
(How does it cancel the remaining tasks if it wants to? Or does par() do
that if this callback raises?) maybe call it filter?

But what did you think of my wait_one() proposal? It may work beter in a
coroutine, where callbacks are considered a nuisance.



> I like the initial approach, but the details need fleshing out. I think it
> would be neat if we could have several standard behaviors in the stdlib:
> subtask_completed=fail_silently, subtask_completed=abort_task, etc.
>
> > And, of course, we should make sure that we can handle the four
>> situations
>> > mentioned in [0] , even if we don't solve them with callbacks.
>> >
>> > [0] https://gist.github.com/3889970
>>
>> That's longwinded and written in a confrontational style. Can you
>> summarize?
>>
>
> Yeah, this was more at a lament at libraries like jQuery that implement
> the CommonJS Promise/A specification wrong. It's really only relevant if we
> choose to add errbacks, as it's about the composition and sematics between
> callbacks/errbacks, and chaining the two.
>

No, no, no! Please. No errbacks. No chaining. Coroutines have a different
way to spell those already: errbacks -> except clauses, chaining ->
multiple yield-froms in one coroutine, or call another coroutine. Please.

--Guido


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121218/b523a336/attachment.html>