[Python-ideas] Documenting asyncio methods as returning awaitables

[Guido van Rossum]

I agree there's been considerable confusion. For example, quoting from
the conversation you linked, "While coroutines are the focus of the
library, they're based on futures". That's actually incorrect.

Until PEP 492 (async/await), there were two separate concepts: Future
and coroutine. A Future is an object with certain methods (e.g.
get_result(), cancel(), add_done_callback()). A coroutine is a
generator object -- it sports no such methods (though it has some of
its own, e.g. send() and throw()). But not every generator object is a
coroutine -- coroutine is expected to have certain "behavior" that
makes it interact correctly with a scheduler. (Details of this
behavior don't matter for this explanation, but it involves yielding
zero or more Futures. Also, the @asyncio.coroutine decorator must be
used to mark the generator as supporting that "behavior".)

Coroutines are more efficient, because when a coroutine calls and
waits for another coroutine (using yield from, or in 3.5 also await)
no trip to the scheduler is required -- it's all taken care of by the
Python interpreter.

Now, the confusion typically occurs because when you use yield from,
it accepts either a coroutine or a Future. And in most cases you're
not really aware (and you don't care) whether a particular thing
you're waiting on is a coroutine or a Future -- you just want to wait
for it, letting the event loop do other things, until it has a result
for you, and either type supports that.

However sometimes you *do* care about the type -- and that's typically
because you want a Future, so you can call some of its methods like
cancel() or add_done_callback(). The correct way to do this, when
you're not sure whether something is a Future or a coroutine, is to
call ensure_future(). If what you've got is already a Future it will
just return that unchanged; if you've got a coroutine it wraps it in a
Task.

Many asyncio operations take either a Future or a coroutine -- they
all just call ensure_future() on that argument.

So how do things change in Python 3.5 with PEP 492? Not much -- the
same story applies, except there's a third type of object, confusingly
called a coroutine object (as opposed to the coroutine I was talking
about above, which is called a generator object). A coroutine object
is almost the same as a generator object, and supports mostly the same
interface (e.g. send(), throw()).

We can treat generator objects with coroutine "behavior" and proper
(PEP 492) coroutine objects as essentially interchangeable, because
that's how PEP 492 was designed. (Differences come out only when
you're making a mistake, such as trying to iterate over one. Iterating
over a pre-PEP-492 coroutine is invalid, but (because it's implemented
as a generator object) you can still call its iter() method. Calling
iter() on a PEP 492 coroutine object fails with a TypeError.

So what should the docs do?

IMO they should be very clear about the distinction between functions
that return Futures and functions that return coroutines (of either
kind). I think it's fine if they are fuzzy about whether the latter
return a PEP 492 style coroutine (i.e. defined with async def) or a
pre-PEP-492 coroutine (marked with @asyncio.coroutine), since those
are almost entirely interchangeable, and the plan is to eventually
make everything a PEP 492 coroutine.

Finally, what should you do if you have a Future but you need a
coroutine? This has come up a few times but it's probably an
indication that there's something you haven't understood yet. The only
API that requires a coroutine (and rejects a Future) is the Task()
constructor, but you should only call that with a coroutine you
defined yourself -- if it's something you received, you should be
using ensure_future(), which will do the right thing (wrapping a
coroutine in a Task).

Good luck!

--Guido



On Mon, Jan 25, 2016 at 7:56 AM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> The official asyncio documentation includes this note:
>
> """
> Note: In this documentation, some methods are documented as
> coroutines, even if they are plain Python functions returning a
> Future. This is intentional to have a freedom of tweaking the
> implementation of these functions in the future. If such a function is
> needed to be used in a callback-style code, wrap its result with
> ensure_future().
> """
>
> Despite the note, this still causes confusion. See for example
> https://mail.python.org/pipermail/python-list/2016-January/702342.html
>
> As of Python 3.5, "awaitable" is a thing, and as of Python 3.5.1,
> ensure_future is supposed to accept any awaitable. Would it be better
> then to document these methods as returning awaitables rather than as
> coroutines?
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido)