On Tue, May 5, 2015 at 2:29 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 May 2015 at 22:12, Guido van Rossum <guido@python.org> wrote:
> I apologize for the confusing documentation. We need more help from
> qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
> after that I was exhausted and did not want to take on writing the end user
> documentation as well, so it was left unfinished. :-(

Fair enough. When I properly document one of my projects, *then* I'll
think about complaining :-) These things happen.

> In PEP 3156 (asyncio package) there are really three separate concepts:
>
> - Future, which is a specific class (of which Task is a subclass);
>
> - coroutine, by which in this context is meant a generator object obtained
> by calling a generator function decorated with @asyncio.coroutine and
> written to conform to the asyncio protocol for coroutines (i.e. don't use
> bare yield, only use yield from, and the latter always with either a Future
> or a coroutine as argument);
>
> - either of the above, which is actually the most common requirement -- most
> asyncio functions that support one also support the other, and either is
> allowable as the argument to `yield from`.
>
> In the implementation we so often flipped between Future and coroutine that
> I imagine sometimes the implementation and docs differ; also, we don't have
> a good short name for "either of the above" so we end up using one or the
> other as a shorthand.

OK, that makes a lot of sense.

> *Unless* you want to attach callbacks, inspect the result or exception, or
> cancel it (all of which require a Future), your code shouldn't be concerned
> about the difference -- you should just use `res = yield from func(args)`
> and use try/except to catch exceptions if you care. And if you do need a
> Future, you can call the function asyncio.async() on it (which in PEP 492 is
> renamed to ensure_future()).

Again, makes sense. Although there are some bits of example code in
the docs that call asyncio.async() on a coroutine and throw away the
result (for example,
https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete).
That confuses me. Are you saying that async() modifies its (coroutine)
argument to make it a Future? Rather than wrapping a coroutine in a
Future, which gets returned?

No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future alone. I'm stumped why that example calls async() and then throws the result away. I suspect it won't work without it (or else Victor wouldn't have added the call) but the reason seems, um, deep. I think wrapping it in a Task enters the generator in the event loop's queue of runnables -- otherwise the generator may well be garbage-collected without ever running.

Such complexity doesn't belong in such a simple example though.
 
> In the PEP 492 world, these concepts map as follows:
>
> - Future translates to "something with an __await__ method" (and asyncio
> Futures are trivially made compliant by defining Future.__await__ as an
> alias for Future.__iter__);
>
> - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
> with `async def` or a generator decorated with @types.coroutine -- note that
> @asyncio.coroutine incorporates the latter);
>
> - "either of the above" maps to "awaitable".

OK. Although "future" is a nicer term than "something with an
__await__ method" and the plethora of flavours of coroutine is not
great. But given that the only term we'll need in common cases is
"awaitable", it's still a net improvement.

So in the PEP 492 world, there's no such thing as a Task outside of
asyncio? Or, to put it another way, a Task is only relevant in an IO
context (unless an alternative event loop library implemented a
similar concept), and we should only be talking in terms of awaitables
and futures (given concurrent.futures and asyncio, I doubt you're
going to be able to stop people using "Future" for the generic term
for "something with an __await__ method" at best, and quite possibly
as equivalent to "awaitable", unfortunately).

I'm not sure. But it's true that Futures and Tasks in asyncio serve the purpose of linking the event loop (whose basic functioning is callback-based) to coroutines (implemented by generators). The basic idea is that when some I/O completes the event loop will call a callback function registered for that particular I/O operation; the callback then is actually a bound method of a Future or Task that causes the latter to be marked as "complete" (i.e. having a result) which in turn will call other callbacks (registered with the Future using add_done_callback()); in the case of a Task (i.e. a special kind of Future that wraps a generator/coroutine) this will resume the coroutine. (Actually it may resume an entire stack of coroutines that are blocked waiting for each other at yield-from; in my spare time I'm working on an explanation of the machinery underlying yield, yield from and await that will explain this.)

It's likely that you could write a much simpler event loop by assuming just coroutines (either the kind implemented by generators using yield from or the PEP 492 kind). The reason asyncio uses callbacks at the lower levels is the hope of fostering interoperability with Twisted and Tornado (and even gevent, which also has an event loop at the bottom of everything).

--
--Guido van Rossum (python.org/~guido)