[Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted

[Dustin J. Mitchell]

As the maintainer of a pretty large, complex app written in Twisted, I
think this is great.  I look forward to a future of being able to
select from a broad library of async tools, and being able to write
tools that can be used outside of Twisted.

Buildbot began, lo these many years ago, doing a lot of things in
memory on on local disk, neither of which require asynchronous IO.  So
a lot of API methods did not originally return Deferreds.  Those
methods are then used by other methods, many of which also do not
return Deferreds.  Now, we want to use a database backend, and
parallelize some of the operations, meaning that the methods need to
return a Deferred.  Unfortunately, that requires a complete tree
traversal of all of the methods and methods that call them, rewriting
them to take and return Deferreds.  There's no "halfway" solution.
This is a little easier with generators (@inlineCallbacks), since the
syntax doesn't change much, but it's a significant change to the API
(in fact, this is a large part of the reason for the big rewrite for
Buildbot-0.9.x).

I bring all this up to say, this PEP will introduce a new "kind" of
method signature into standard Python, one which the caller must know,
and the use of which changes the signature of the caller.  That can
cause sweeping changes, and debugging those changes can be tricky.
Two things can help:

First, `yield from somemeth()` should work fine even if `somemeth` is
not a coroutine function, and authors of async tools should be
encouraged to use this form to assist future-compatibility.  Second,
`somemeth()` without a yield should fail loudly if `somemeth` is a
coroutine function.  Otherwise, the effects can be pretty confusing.

In http://code.google.com/p/uthreads, I accomplished the latter by
taking advantage of garbage collection: if the generator is garbage
collected before it's begun, then it's probably not been yielded.
This is a bit gross, but good enough as a debugging technique.

On the topic of debugging, I also took pains to make sure that
tracebacks looked reasonable, filtering out scheduler code[1].  I
haven't looked closely at Tulip to see if that's a problem.  Most of
the "noise" in the tracebacks came from the lack of 'yield from', so
it may not be an issue at all.

Dustin

P.S. Apologies for the bad threading - I wasn't on the list when this
was last posted.

[1] http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253