[Python-Dev] PEP 492: async/await in Python; version 4

[Guido van Rossum]

On Tue, May 5, 2015 at 2:29 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 5 May 2015 at 22:12, Guido van Rossum <guido at python.org> wrote:
> > I apologize for the confusing documentation. We need more help from
> > qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
> > after that I was exhausted and did not want to take on writing the end
> user
> > documentation as well, so it was left unfinished. :-(
>
> Fair enough. When I properly document one of my projects, *then* I'll
> think about complaining :-) These things happen.
>
> > In PEP 3156 (asyncio package) there are really three separate concepts:
> >
> > - Future, which is a specific class (of which Task is a subclass);
> >
> > - coroutine, by which in this context is meant a generator object
> obtained
> > by calling a generator function decorated with @asyncio.coroutine and
> > written to conform to the asyncio protocol for coroutines (i.e. don't use
> > bare yield, only use yield from, and the latter always with either a
> Future
> > or a coroutine as argument);
> >
> > - either of the above, which is actually the most common requirement --
> most
> > asyncio functions that support one also support the other, and either is
> > allowable as the argument to `yield from`.
> >
> > In the implementation we so often flipped between Future and coroutine
> that
> > I imagine sometimes the implementation and docs differ; also, we don't
> have
> > a good short name for "either of the above" so we end up using one or the
> > other as a shorthand.
>
> OK, that makes a lot of sense.
>
> > *Unless* you want to attach callbacks, inspect the result or exception,
> or
> > cancel it (all of which require a Future), your code shouldn't be
> concerned
> > about the difference -- you should just use `res = yield from func(args)`
> > and use try/except to catch exceptions if you care. And if you do need a
> > Future, you can call the function asyncio.async() on it (which in PEP
> 492 is
> > renamed to ensure_future()).
>
> Again, makes sense. Although there are some bits of example code in
> the docs that call asyncio.async() on a coroutine and throw away the
> result (for example,
>
> https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete
> ).
> That confuses me. Are you saying that async() modifies its (coroutine)
> argument to make it a Future? Rather than wrapping a coroutine in a
> Future, which gets returned?
>

No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future
alone. I'm stumped why that example calls async() and then throws the
result away. I suspect it won't work without it (or else Victor wouldn't
have added the call) but the reason seems, um, deep. I think wrapping it in
a Task enters the generator in the event loop's queue of runnables --
otherwise the generator may well be garbage-collected without ever running.

Such complexity doesn't belong in such a simple example though.


> > In the PEP 492 world, these concepts map as follows:
> >
> > - Future translates to "something with an __await__ method" (and asyncio
> > Futures are trivially made compliant by defining Future.__await__ as an
> > alias for Future.__iter__);
> >
> > - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
> > with `async def` or a generator decorated with @types.coroutine -- note
> that
> > @asyncio.coroutine incorporates the latter);
> >
> > - "either of the above" maps to "awaitable".
>
> OK. Although "future" is a nicer term than "something with an
> __await__ method" and the plethora of flavours of coroutine is not
> great. But given that the only term we'll need in common cases is
> "awaitable", it's still a net improvement.
>
> So in the PEP 492 world, there's no such thing as a Task outside of
> asyncio? Or, to put it another way, a Task is only relevant in an IO
> context (unless an alternative event loop library implemented a
> similar concept), and we should only be talking in terms of awaitables
> and futures (given concurrent.futures and asyncio, I doubt you're
> going to be able to stop people using "Future" for the generic term
> for "something with an __await__ method" at best, and quite possibly
> as equivalent to "awaitable", unfortunately).
>

I'm not sure. But it's true that Futures and Tasks in asyncio serve the
purpose of linking the event loop (whose basic functioning is
callback-based) to coroutines (implemented by generators). The basic idea
is that when some I/O completes the event loop will call a callback
function registered for that particular I/O operation; the callback then is
actually a bound method of a Future or Task that causes the latter to be
marked as "complete" (i.e. having a result) which in turn will call other
callbacks (registered with the Future using add_done_callback()); in the
case of a Task (i.e. a special kind of Future that wraps a
generator/coroutine) this will resume the coroutine. (Actually it may
resume an entire stack of coroutines that are blocked waiting for each
other at yield-from; in my spare time I'm working on an explanation of the
machinery underlying yield, yield from and await that will explain this.)

It's likely that you could write a much simpler event loop by assuming just
coroutines (either the kind implemented by generators using yield from or
the PEP 492 kind). The reason asyncio uses callbacks at the lower levels is
the hope of fostering interoperability with Twisted and Tornado (and even
gevent, which also has an event loop at the bottom of everything).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150505/9981fdaa/attachment.html>