[Python-ideas] [Python-Dev] Python needs a standard asynchronous return object

[Antoine Pitrou]

On Sat, 11 Sep 2010 19:26:50 -0700
Guido van Rossum <guido at python.org> wrote:
> 
> But thinking about this more I don't know that it will be easy to mix
> PEP 3148, which is solidly thread-based, with a PEP 342 style
> scheduler (whether or not the PEP 380 enhancements are applied, or
> even PEP 3152).

I'm not sure why. The implementation is certainly thread-based, but
functions such as `wait(fs, timeout=None, return_when=ALL_COMPLETED)`
could be implemented in termes of a single-threaded event loop / job
scheduler.

Actually, Twisted has a similar primitive in DeferredList, although
more powerful since the DeferredList itself is a Deferred, and can
therefore be further combined, etc.:

http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.defer.DeferredList.html

> And comparing the
> blog's examples to PEP 3148, I find Twisted's terminology rather
> confusing compared to the PEP's clean Futures API (where IMO you can
> ignore almost everything except result()).

Well, apart from the API which may be considered a taste issue (I have
used Deferreds long before I heard about Futures, so perhaps I'm a bit
biased), the following API doc in PEP 3148 shows that the Future model
of callbacks is less rich than Twisted's:

“add_done_callback(fn)

    Attaches a callable fn to the future that will be called when the
    future is cancelled or finishes running. fn will be called with the
    future as its only argument.

    Added callables are called in the order that they were added and
    are always called in a thread belonging to the process that added
    them. If the callable raises an Exception then it will be logged
    and ignored. If the callable raises another BaseException then
    behavior is not defined.”

With Twisted Deferreds, when a callback or errback raises an error, its
exception isn't “logged and ignored”, it is passed to the remaining
errback chain attached to the Deferred. This is part of what makes
Deferreds more complicated to understand, but it also makes them more
powerful.

Another key point is that a callback can itself return another Deferred
object, in which case the next callback (or errback, in case of error)
will be called only once the other Deferred produces a result. This is
all handled transparently and you can freely mix callbacks that
immediately return a value, and callbacks that return a Deferred whose
final value will be available later. And the other Deferred can have
its own callback/errback chain, etc.

(just for the record, the “final value” of a Deferred is the value
returned by the last callback in the chain)


I think the main reason, though, that people find Deferreds
inconvenient is that they force you to think in terms of
asynchronicity (well, almost: you can of course hack yourself
some code which blocks until a Deferred has a value, but it's
extremely discouraged). They would like to have officially
supported methods like `result(timeout=None)` which make simple things
(like quick scripts to fetch a bunch of URLs) simpler. Twisted is
generally used for server applications where such code is out of
question (in an async model, that is).

Regards

Antoine.