[Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted

[Guido van Rossum]

On Fri, Jan 4, 2013 at 2:38 PM, Dustin Mitchell <djmitche at gmail.com> wrote:
> As the maintainer of a pretty large, complex app written in Twisted, I think
> this is great.  I look forward to a future of being able to select from a
> broad library of async tools, and being able to write tools that can be used
> outside of Twisted.

Thanks. Me too. :-)

> Buildbot began, lo these many years ago, doing a lot of things in memory on
> on local disk, neither of which require asynchronous IO.  So a lot of API
> methods did not originally return Deferreds.  Those methods are then used by
> other methods, many of which also do not return Deferreds.  Now, we want to
> use a database backend, and parallelize some of the operations, meaning that
> the methods need to return a Deferred.  Unfortunately, that requires a
> complete tree traversal of all of the methods and methods that call them,
> rewriting them to take and return Deferreds.  There's no "halfway" solution.
> This is a little easier with generators (@inlineCallbacks), since the syntax
> doesn't change much, but it's a significant change to the API (in fact, this
> is a large part of the reason for the big rewrite for Buildbot-0.9.x).
>
> I bring all this up to say, this PEP will introduce a new "kind" of method
> signature into standard Python, one which the caller must know, and the use
> of which changes the signature of the caller.  That can cause sweeping
> changes, and debugging those changes can be tricky.

Yes, and this is the biggest unproven point of the PEP. (The rest is
all backed by a decade or more of experience.)

> Two things can help:
>
> First, `yield from somemeth()` should work fine even if `somemeth` is not a
> coroutine function, and authors of async tools should be encouraged to use
> this form to assist future-compatibility.  Second, `somemeth()` without a
> yield should fail loudly if `somemeth` is a coroutine function.  Otherwise,
> the effects can be pretty confusing.

That would be nice. But the way yield from and generators work, that's
hard to accomplish without further changes to the language -- and I
don't want to have to change the language again (at least not
immediately -- maybe in a few releases, after we've learned what the
real issues are). The best I can do for the first requirement is to
define @coroutine in a way that if the decorated function isn't a
generator, it is wrapped in one. For the second requirement, if you
call somemeth() and ignore the result, nothing happens at all -- this
is indeed infuriating but I see no way to change this.(*) If you use
the result, well, Futures have different attributes than most other
objects so hopefully you'll get a loud AttributeError or TypeError
soon, but of course if you pass it into something else which uses it,
it may still be difficult to track. Hopefully these error messages
provide a hint:

>>> f.foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Future' object has no attribute 'foo'
>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'Future' object is not callable
>>>

(*) There's a heavy gun we might use, but I would make this optional,
as a heavy duty debugging mode only. @coroutine could wrap generators
in a lightweight object with a __del__ method and an __iter__ method.
If __del__ is called before __iter__ is ever called, it could raise an
exception or log a warning. But this probably adds too much overhead
to have it always enabled.

> In http://code.google.com/p/uthreads, I accomplished the latter by taking
> advantage of garbage collection: if the generator is garbage collected
> before it's begun, then it's probably not been yielded.  This is a bit
> gross, but good enough as a debugging technique.

Eh, yeah, what I said. :-)

> On the topic of debugging, I also took pains to make sure that tracebacks
> looked reasonable, filtering out scheduler code[1].  I haven't looked
> closely at Tulip to see if that's a problem.  Most of the "noise" in the
> tracebacks came from the lack of 'yield from', so it may not be an issue at
> all.

One of the great advantages of using yield from is that the tracebacks
automatically look nice.

> Dustin
>
> [1]
> http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253



-- 
--Guido van Rossum (python.org/~guido)