[Python-ideas] [Python-Dev] Python needs a standard asynchronous return object

[Guido van Rossum]

On Sun, Sep 12, 2010 at 4:03 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 11 Sep 2010 19:26:50 -0700
> Guido van Rossum <guido at python.org> wrote:
>>
>> But thinking about this more I don't know that it will be easy to mix
>> PEP 3148, which is solidly thread-based, with a PEP 342 style
>> scheduler (whether or not the PEP 380 enhancements are applied, or
>> even PEP 3152).
>
> I'm not sure why. The implementation is certainly thread-based, but
> functions such as `wait(fs, timeout=None, return_when=ALL_COMPLETED)`
> could be implemented in termes of a single-threaded event loop / job
> scheduler.

Sure, but the tricky thing is to make it pluggable so that PEP 3148
and Twisted and other frameworks can use it all together, and a single
call will accept a mixture of Futures.

I also worry that "impure" code will have a hard time -- e.g. when
mixing generator-based coroutines and thread-based futures, it would
be quite bad if a coroutine called .result() on a Future or the
.wait() function instead of yielding to the scheduler.

> Actually, Twisted has a similar primitive in DeferredList, although
> more powerful since the DeferredList itself is a Deferred, and can
> therefore be further combined, etc.:
>
> http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.defer.DeferredList.html

This sounds similar to the way you can create derived futures in Java.

>> And comparing the
>> blog's examples to PEP 3148, I find Twisted's terminology rather
>> confusing compared to the PEP's clean Futures API (where IMO you can
>> ignore almost everything except result()).
>
> Well, apart from the API which may be considered a taste issue (I have
> used Deferreds long before I heard about Futures, so perhaps I'm a bit
> biased),

I heard of Deferred long before PEP 3148 was even conceived, but I
find Twisted's terminology terribly confusing while I find the PEP's
names easy to understand.

> the following API doc in PEP 3148 shows that the Future model
> of callbacks is less rich than Twisted's:
>
> “add_done_callback(fn)
>
>    Attaches a callable fn to the future that will be called when the
>    future is cancelled or finishes running. fn will be called with the
>    future as its only argument.
>
>    Added callables are called in the order that they were added and
>    are always called in a thread belonging to the process that added
>    them. If the callable raises an Exception then it will be logged
>    and ignored. If the callable raises another BaseException then
>    behavior is not defined.”
>
> With Twisted Deferreds, when a callback or errback raises an error, its
> exception isn't “logged and ignored”, it is passed to the remaining
> errback chain attached to the Deferred. This is part of what makes
> Deferreds more complicated to understand, but it also makes them more
> powerful.

Yeah, please do explain why Twisted has so much machinery to handle exceptions?

ISTM that the main difference is that add_done_callback() isn't meant
for callbacks that return a value. So then the exceptions that might
be raised are kind of "out of band". For any API that returns a value
I agree that raising an exception should be handled -- but in the PEP
342 world we can do that by passing exceptions back into coroutine
using throw(), so no separate "success" and "failure" callbacks are
needed.

> Another key point is that a callback can itself return another Deferred
> object, in which case the next callback (or errback, in case of error)
> will be called only once the other Deferred produces a result. This is
> all handled transparently and you can freely mix callbacks that
> immediately return a value, and callbacks that return a Deferred whose
> final value will be available later. And the other Deferred can have
> its own callback/errback chain, etc.

Yeah, that is part of what makes it so utterly confusing. PEP 380
supports a similar thing but much cleaner, without ever using
callbacks.

> (just for the record, the “final value” of a Deferred is the value
> returned by the last callback in the chain)
>
>
> I think the main reason, though, that people find Deferreds
> inconvenient is that they force you to think in terms of
> asynchronicity (well, almost: you can of course hack yourself
> some code which blocks until a Deferred has a value, but it's
> extremely discouraged). They would like to have officially
> supported methods like `result(timeout=None)` which make simple things
> (like quick scripts to fetch a bunch of URLs) simpler. Twisted is
> generally used for server applications where such code is out of
> question (in an async model, that is).

Actually I think the main reason is historic: Twisted introduced
callback-based asynchronous (thread-less) programming when there was
no alternative in Python, and they invented both the mechanisms and
the terminology as they were figuring it all out. That is no mean
feat. But with PEP 342 (generator-based coroutines) and especially PEP
380 (yield from) there *is* an alternative, and while Twisted has
added APIs to support generators, it hasn't started to deprecate its
other APIs, and its terminology becomes hard to follow for people
(like me, frankly) who first learned this stuff through PEP 342.

-- 
--Guido van Rossum (python.org/~guido)