[Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted

[Dustin Mitchell]

As the maintainer of a pretty large, complex app written in Twisted, I 
think this is great.  I look forward to a future of being able to select 
from a broad library of async tools, and being able to write tools that can 
be used outside of Twisted.

Buildbot began, lo these many years ago, doing a lot of things in memory on 
on local disk, neither of which require asynchronous IO.  So a lot of API 
methods did not originally return Deferreds.  Those methods are then used 
by other methods, many of which also do not return Deferreds.  Now, we want 
to use a database backend, and parallelize some of the operations, meaning 
that the methods need to return a Deferred.  Unfortunately, that requires a 
complete tree traversal of all of the methods and methods that call them, 
rewriting them to take and return Deferreds.  There's no "halfway" 
solution.  This is a little easier with generators (@inlineCallbacks), 
since the syntax doesn't change much, but it's a significant change to the 
API (in fact, this is a large part of the reason for the big rewrite for 
Buildbot-0.9.x).

I bring all this up to say, this PEP will introduce a new "kind" of method 
signature into standard Python, one which the caller must know, and the use 
of which changes the signature of the caller.  That can cause sweeping 
changes, and debugging those changes can be tricky.  Two things can help:

First, `yield from somemeth()` should work fine even if `somemeth` is not a 
coroutine function, and authors of async tools should be encouraged to use 
this form to assist future-compatibility.  Second, `somemeth()` without a 
yield should fail loudly if `somemeth` is a coroutine function.  Otherwise, 
the effects can be pretty confusing.

In http://code.google.com/p/uthreads, I accomplished the latter by taking 
advantage of garbage collection: if the generator is garbage collected 
before it's begun, then it's probably not been yielded.  This is a bit 
gross, but good enough as a debugging technique.

On the topic of debugging, I also took pains to make sure that tracebacks 
looked reasonable, filtering out scheduler code[1].  I haven't looked 
closely at Tulip to see if that's a problem.  Most of the "noise" in the 
tracebacks came from the lack of 'yield from', so it may not be an issue at 
all.

Dustin

[1] 
http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130104/755fc363/attachment.html>