[Python-Dev] Python needs a standard asynchronous return object

[James Yonan]

I'd like to propose that the Python community standardize on a 
"deferred" object for asynchronous return values, modeled after the 
well-thought-out Twisted Deferred class.

With more and more Python libraries implementing asynchronicity (for 
example Futures -- PEP 3148), it's crucial to have a standard deferred 
object in place so that code using a single asynchronous reactor can 
interoperate with different asynchronous libraries.

I think a lot of people don't realize how much cooler and more elegant 
it is to return a deferred object from an asynchronous function rather 
than using a generic callback approach (where you pass a function 
argument to the asynchronous function telling it where to call when the 
asynchronous operation completes).

While asynchronous systems have been shown to have excellent scalability 
properties, the callback-based programming style often used in 
asynchronous programming has been criticized for breaking up the 
sequential readability of program logic.

This problem is elegantly addressed by using Deferred Generators.  Since 
Python 2.5 added enhanced generators (i.e. the capability for "yield" to 
return a value), the infrastructure is now in place to allow an 
asynchronous function to be written in a sequential style, without the 
use of explicit callbacks.

See the following blog article for a nice write-up on the capability:

http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html

Mekk's Twisted Deferred example:

@defer.inlineCallbacks
def someFunction():
     a = 1
     b = yield deferredReturningFunction(a)
     c = yield anotherDeferredReturningFunction(a, b)
     defer.returnValue(c)

What's cool about this is that between the two yield statements, the 
Twisted reactor is in control meaning that other pending asynchronous 
tasks can be attended to or the thread's remaining time slice can be 
yielded to the kernel, yet this is all accomplished without the use of 
multi-threading.  Another interesting aspect of this approach is that 
since it leverages on Python's enhanced generators, an exception thrown 
inside either of the deferred-returning functions will be propagated 
through to someFunction() where it can be handled with try/except.

Think about what this means -- this sort of emulates the "stackless" 
design pattern you would expect in Erlang or Stackless Python without 
leaving standard Python.  And it's made possible under the hood by 
Python Enhanced Generators.

Needless to say, it would be great to see this coolness be part of the 
standard Python library, instead of having every Python asynchronous 
library implement its own ad-hoc callback system.

James Yonan