[Python-Dev] Minimal async event loop and async utilities (Was: PEP 492: async/await in Python; version 4)

[Guido van Rossum]

On Mon, May 11, 2015 at 1:37 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 6 May 2015 at 16:46, Guido van Rossum <guido at python.org> wrote:
> > This is actually a great idea, and I encourage you to go forward with it.
> > The biggest piece missing from your inventory is probably Task, which is
> > needed to wrap a Future around a coroutine.
>
> OK, I've been doing some work on this. You're right, the asyncio
> framework makes Future a key component.
>
> But I'm not 100% sure why Future (and Task) have to be so fundamental.
> Ignoring cancellation (see below!) I can build pretty much all of a
> basic event loop, plus equivalents of the asyncio locks and queues
> modules, without needing the concept of a Future at all. The
> create_task function becomes simply a function to add a coroutine to
> the ready queue, in this context. I can't return a Task (because I
> haven't implemented the Task or Future classes) but I don't actually
> know what significant functionality is lost as a result - is there a
> reasonably accessible example of where using the return value from
> create_task is important anywhere?
>

In asyncio the Task object is used to wait for the result. Of course if all
you need is to wait for the result you don't need to call create_task() --
so in your situation it's uninteresting. But Task is needed for
cancellation and Future is needed so I/O completion can be implemented
using callback functions.


> A slightly more complicated issue is with the run_until_complete
> function, which takes a Future, and hence is fundamentally tied to the
> Future API. However, it seems to me that a "minimal" implementation
> could work by having a run_until_complete() that just took an
> awaitable (i.e., anything that you can yield from). Again, is there a
> specific reason that you ended up going with run_until_complete taking
> a Future rather than just a coroutine?


Actually it takes a Future *or* a coroutine. (The docs or the arg name may
be confusing.) In asyncio, pretty much everything that takes one takes the
other.


> I think (but haven't confirmed
> yet by implementing it) that it should be possible to create a
> coroutine that acts like a Future, in the sense that you can tell it
> from outside (via send()) that it's completed and set its return
> value. But this is all theory, and if you have any practical
> experience that shows I'm going down a dead end, I'd be glad to know.
>

I don't know -- I never explored that.


> I'm not sure how useful this line of attack will be - if the API isn't
> compatible with asyncio.BaseEventLoop, it's not very useful in
> practice. On the other hand, if I can build a loop without Future or
> Task classes, it may indicate that those classes aren't quite as
> fundamental as asyncio makes them (which may allow some
> simplifications or generalisations).
>

Have you tried to implement waiting for I/O yet?

OTOH you may look at micropython's uasyncio -- IIRC it doesn't have Futures
and it definitely has I/O waiting.


> > I expect you'll also want to build cancellation into your "base async
> > framework"; and the primitives to wait for multiple awaitables. The next
> > step would be some mechanism to implement call_later()/call_at() (but
> this
> > needs to be pluggable since for a "real" event loop it needs to be
> > implemented by the basic I/O selector).
>
> These are where I suspect I'll have the most trouble if I haven't got
> a solid understanding of the role of the Future and Task classes (or
> alternatively, how to avoid them :-)) So I'm holding off on worrying
> about them for now. But certainly they need to be covered. In
> particular, call_later/call_at are the only "generic" example of any
> form of wait that actually *waits*, rather than returning immediately.
> So as you say, implementing them will show how the basic mechanism can
> be extended with a "real" selector (whether for I/O, or GUI events, or
> whatever).
>

Right.


> > If you can get this working it would be great to include this in the
> stdlib
> > as a separate "asynclib" library. The original asyncio library would
> then be
> > a specific implementation (using a subclass of asynclib.EventLoop) that
> adds
> > I/O, subprocesses, and integrates with the selectors module (or with
> IOCP,
> > on Windows).
>
> One thing I've not really considered in the above, is how a
> refactoring like this would work. Ignoring the "let's try to remove
> the Future class" approach above, my "basic event loop" is mostly just
> an alternative implementation of an event loop (or maybe an
> alternative policy - I'm not sure I understand the need for policies
> yet).


A policy is mostly a wrapper around an event loop factory plus state that
records the current event loop.


> So it may simply be a case of ripping coroutines.py, futures.py,
> locks.py, log.py, queues.py, and tasks.py out of asyncio and adding a
> new equivalent of events.py with my "minimal" loop in it. (So far,
> when I've tried to do that I get hit with some form of circular import
> problem - I've not worked out why yet, or how asyncio avoids the same
> problem).
>

That sounds like a surface problem. Keep on debugging. :-)


> That in itself would probably be a useful refactoring, splitting out
> the IO aspects of asyncio from the event loop / async aspects.
>

Well, if you can.


> > I don't see any particular hurry to get this in before 3.5; the
> refactoring
> > of asyncio can be done later, in a backward compatible way. It would be a
> > good way to test the architecture of asyncio!
>
> Agreed. It's also not at all clear to me how the new async/await
> syntax would fit in with this, so that probably needs some time to
> settle down. For example, in Python 3.5 would run_until_complete take
> an awaitable rather than a Future?
>

It doesn't need to change -- it already calls async() on its argument
before doing anything (though with PEP 492 that function will be renamed to
ensure_future()).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150511/f255ebd4/attachment-0001.html>