Documenting asyncio methods as returning awaitables
The official asyncio documentation includes this note: """ Note: In this documentation, some methods are documented as coroutines, even if they are plain Python functions returning a Future. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be used in a callback-style code, wrap its result with ensure_future(). """ Despite the note, this still causes confusion. See for example https://mail.python.org/pipermail/python-list/2016-January/702342.html As of Python 3.5, "awaitable" is a thing, and as of Python 3.5.1, ensure_future is supposed to accept any awaitable. Would it be better then to document these methods as returning awaitables rather than as coroutines?
I agree there's been considerable confusion. For example, quoting from the conversation you linked, "While coroutines are the focus of the library, they're based on futures". That's actually incorrect. Until PEP 492 (async/await), there were two separate concepts: Future and coroutine. A Future is an object with certain methods (e.g. get_result(), cancel(), add_done_callback()). A coroutine is a generator object -- it sports no such methods (though it has some of its own, e.g. send() and throw()). But not every generator object is a coroutine -- coroutine is expected to have certain "behavior" that makes it interact correctly with a scheduler. (Details of this behavior don't matter for this explanation, but it involves yielding zero or more Futures. Also, the @asyncio.coroutine decorator must be used to mark the generator as supporting that "behavior".) Coroutines are more efficient, because when a coroutine calls and waits for another coroutine (using yield from, or in 3.5 also await) no trip to the scheduler is required -- it's all taken care of by the Python interpreter. Now, the confusion typically occurs because when you use yield from, it accepts either a coroutine or a Future. And in most cases you're not really aware (and you don't care) whether a particular thing you're waiting on is a coroutine or a Future -- you just want to wait for it, letting the event loop do other things, until it has a result for you, and either type supports that. However sometimes you *do* care about the type -- and that's typically because you want a Future, so you can call some of its methods like cancel() or add_done_callback(). The correct way to do this, when you're not sure whether something is a Future or a coroutine, is to call ensure_future(). If what you've got is already a Future it will just return that unchanged; if you've got a coroutine it wraps it in a Task. Many asyncio operations take either a Future or a coroutine -- they all just call ensure_future() on that argument. So how do things change in Python 3.5 with PEP 492? Not much -- the same story applies, except there's a third type of object, confusingly called a coroutine object (as opposed to the coroutine I was talking about above, which is called a generator object). A coroutine object is almost the same as a generator object, and supports mostly the same interface (e.g. send(), throw()). We can treat generator objects with coroutine "behavior" and proper (PEP 492) coroutine objects as essentially interchangeable, because that's how PEP 492 was designed. (Differences come out only when you're making a mistake, such as trying to iterate over one. Iterating over a pre-PEP-492 coroutine is invalid, but (because it's implemented as a generator object) you can still call its iter() method. Calling iter() on a PEP 492 coroutine object fails with a TypeError. So what should the docs do? IMO they should be very clear about the distinction between functions that return Futures and functions that return coroutines (of either kind). I think it's fine if they are fuzzy about whether the latter return a PEP 492 style coroutine (i.e. defined with async def) or a pre-PEP-492 coroutine (marked with @asyncio.coroutine), since those are almost entirely interchangeable, and the plan is to eventually make everything a PEP 492 coroutine. Finally, what should you do if you have a Future but you need a coroutine? This has come up a few times but it's probably an indication that there's something you haven't understood yet. The only API that requires a coroutine (and rejects a Future) is the Task() constructor, but you should only call that with a coroutine you defined yourself -- if it's something you received, you should be using ensure_future(), which will do the right thing (wrapping a coroutine in a Task). Good luck! --Guido On Mon, Jan 25, 2016 at 7:56 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
I agree there's been considerable confusion. For example, quoting from the conversation you linked, "While coroutines are the focus of the library, they're based on futures". That's actually incorrect. Until PEP 492 (async/await), there were two separate concepts: Future and coroutine. A Future is an object with certain methods (e.g. get_result(), cancel(), add_done_callback()). A coroutine is a generator object -- it sports no such methods (though it has some of its own, e.g. send() and throw()). But not every generator object is a coroutine -- coroutine is expected to have certain "behavior" that makes it interact correctly with a scheduler. (Details of this behavior don't matter for this explanation, but it involves yielding zero or more Futures. Also, the @asyncio.coroutine decorator must be used to mark the generator as supporting that "behavior".) Coroutines are more efficient, because when a coroutine calls and waits for another coroutine (using yield from, or in 3.5 also await) no trip to the scheduler is required -- it's all taken care of by the Python interpreter. Now, the confusion typically occurs because when you use yield from, it accepts either a coroutine or a Future. And in most cases you're not really aware (and you don't care) whether a particular thing you're waiting on is a coroutine or a Future -- you just want to wait for it, letting the event loop do other things, until it has a result for you, and either type supports that. However sometimes you *do* care about the type -- and that's typically because you want a Future, so you can call some of its methods like cancel() or add_done_callback(). The correct way to do this, when you're not sure whether something is a Future or a coroutine, is to call ensure_future(). If what you've got is already a Future it will just return that unchanged; if you've got a coroutine it wraps it in a Task. Many asyncio operations take either a Future or a coroutine -- they all just call ensure_future() on that argument. So how do things change in Python 3.5 with PEP 492? Not much -- the same story applies, except there's a third type of object, confusingly called a coroutine object (as opposed to the coroutine I was talking about above, which is called a generator object). A coroutine object is almost the same as a generator object, and supports mostly the same interface (e.g. send(), throw()). We can treat generator objects with coroutine "behavior" and proper (PEP 492) coroutine objects as essentially interchangeable, because that's how PEP 492 was designed. (Differences come out only when you're making a mistake, such as trying to iterate over one. Iterating over a pre-PEP-492 coroutine is invalid, but (because it's implemented as a generator object) you can still call its iter() method. Calling iter() on a PEP 492 coroutine object fails with a TypeError. So what should the docs do? IMO they should be very clear about the distinction between functions that return Futures and functions that return coroutines (of either kind). I think it's fine if they are fuzzy about whether the latter return a PEP 492 style coroutine (i.e. defined with async def) or a pre-PEP-492 coroutine (marked with @asyncio.coroutine), since those are almost entirely interchangeable, and the plan is to eventually make everything a PEP 492 coroutine. Finally, what should you do if you have a Future but you need a coroutine? This has come up a few times but it's probably an indication that there's something you haven't understood yet. The only API that requires a coroutine (and rejects a Future) is the Task() constructor, but you should only call that with a coroutine you defined yourself -- if it's something you received, you should be using ensure_future(), which will do the right thing (wrapping a coroutine in a Task). Good luck! --Guido On Mon, Jan 25, 2016 at 7:56 AM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
participants (2)
-
Guido van Rossum -
Ian Kelly