[Python-ideas] How the heck does async/await work in Python 3.5

[Terry Reedy]

On 2/25/2016 11:47 AM, Guido van Rossum wrote:
> On Wed, Feb 24, 2016 at 10:49 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> My memory is that any replacement event loop is expected to be a full
>> implmentation of the network io loop, so one cannot just make a gui loop
>> that works with coroutines.
>
> That's not correct. If an I/O loop does not implement the network I/O
> APIs it just cannot be used for network I/O, but it can still
> implement all the other functionality (like call_later).

The tk loop already has the run and call functionality (and gui event 
stuff).  The reason I want to integrate tkinter and asyncio is to be 
able to use the I/O APIs.

In particular, I want to experiment with subprocess_exec.  Assuming that 
my reading of the PEP is correct, I want to try using it run code 
checkers on code (such as the contents of an IDLE editor file). A GSOC 
student worked out everything but how to do so without blocking and 
freezing the ui. Beyond that, we need to replace the unreliable (but 
non-blocking) socket connection between IDLE and the user execution 
process).

> There are
> other loops that don't implement everything (e.g. the subprocess and
> datagram APIs are not supported by Proactor,

According to the PEP and doc
https://docs.python.org/3/library/asyncio-eventloops.html#windows,
the Proactor is the loop that supports subprocesses on Windows, while 
Windows Selector does not.  I will find out for sure soon enough.

> and signal support is
> spotty, too). If you find some implementation barrier that prevents
> that from working it should be fixed.

After lots of reading and thinking, I decided to start with simplest 
this that might work, and came up with the following surprisingly simple 
subclass.  It works with GUI versions of the first two examples in the 
doc.  It will satisfy many simple cases of using tkinter and asyncio 
together.

I will continue with other examples  The docstring describes possible 
ways to improve the integration.

I am not sure whether root.update handles the events handled by 
update_idletasks (or where 'idletasks' come from). If not, 
update_idletask() calls may be needed.  Serhiy should know.

---
'''Proof of concept for integrating asyncio and tk loops.

Terry Jan Reedy
Run with 'python -i' or from IDLE editor to keep tk window alive.

This version simply calls tk root.update in run_forever.
Without this, the tk window only appears after loop.stop,
unless root.update() is put in callbacks instead.

If there are no pending asyncio callbacks, self._run_once
will block indefinitely in a .select(None) call, waiting
for an I/O event.  To ensure that all tk events and callbacks
get timely attention, either _run_once should be revised or
__init__ should start an asyncio callback loop so that there is always a 
pending callback.  The details will depend on whether the GUI or network 
IO is intended to get priority attention.
'''

import asyncio as ai
import threading
import tkinter as tk
import datetime  # for example

# On Windows, need proactor for subprocess functions
EventLoop = ai.ProactorEventLoop
# Following gets ai.SelectorEventLoop
# EventLoop = type(ai.get_event_loop())

class TkEventLoop(EventLoop):
     def __init__(self, root):
         self.root = root
         super().__init__()

     def run_forever(self):  # copy with 1 new line for proof of concept
         """Run until stop() is called."""
         self._check_closed()
         if self.is_running():
             raise RuntimeError('Event loop is running.')
         self._set_coroutine_wrapper(self._debug)
         self._thread_id = threading.get_ident()
         try:
             while True:
                 self._run_once()  # can block indefinitely
                 self.root.update()  # new line
                 if self._stopping:
                     break
         finally:
             self._stopping = False
             self._thread_id = None
             self._set_coroutine_wrapper(False)

root = tk.Tk()
loop = TkEventLoop(root)
ai.set_event_loop(loop)

# Combine 2 event loop examples from BaseEventLoop doc.

hello = tk.Label(root)
timel = tk.Label(root)
hello.pack()
timel.pack()

def hello_world(loop):
     hello['text'] = 'Hello World'


loop.call_soon(hello_world, loop)

def display_date(end_time, loop):
     timel['text'] = datetime.datetime.now()
     if (loop.time() + 1.0) < end_time:
         loop.call_later(1, display_date, end_time, loop)
     else:
         loop.stop()

end_time = loop.time() + 5.1
loop.call_soon(display_date, end_time, loop)

# Blocking call interrupted by loop.stop()
loop.run_forever()
---

> The await expression (or yield from) does not require an event loop.
> Heck, it doesn't require futures.

Without either, are async/await actually useful?  The binary/abinary 
examples just shows that there is not much penalty for using async/await 
when not needed.  On my machine, abinary is pretty consistently 2% slower.

> It uses coroutines and it is totally
> up to the manager of those how they are scheduled (read Greg Ewing's
> very early explanations of yield from -- some have no I/O at all).

What do you mean by 'manager of [coroutines]'?  For loop, event loop, 
task, the nebulous 'scheduler', or something else?

> But there's something else that you might be after.

After getting non-blocking gui and io events (with callbacks) working 
together, I an after asynchronous iteration.  The asyncio modules used 
cooroutines for suspension, but as far as I can tell, not for iteration.

Normal synchronous iteration is fine for compute-bound iterators and a 
single iterator whose delays are tolerable because the thread or process 
has nothing better to do.  (And the system will switch to something else 
when the thread blocks, so the cpu is not actually blocked.)  But it 
does not work for concurrent intermitant streams.

For example, consider the datetime display in the example above.  The 
code is scattered around in three places.  I would like to be able to 
replace the pieces with synchronous-like code, something like the following.

async def display_date(interval, end_time, loop=ai.get_event_loop()):
     label = tk.Label(root)
     label.pack()
     async for tick in timer_soon(interval, end_time, loop):
          label['text'] = datetime.datetime.now()

The asynchronous iterator timer_soon would incorporate call_soon, 
call_later, and stop calls and 'yield' True at the appointed times. 
Doing so allows the widget creation and repeated widget event handling 
to be combined in one function.

How do I write it?  The async iterator examples in PEP 492 (the second 
is Example 1) shows all the boiler plate, which I could write from the 
spec given, but omit the essential detail of how to write the inner 
cooroutine that gets awaited on.

     async def fetch_data(self):
         ...

     async def _prefetch(self):
         ...
???

> When you're implementing this API on top of tkinter, you'll probably
> find that you'll have to use tkinter's way of sleeping anyways, so the
> implementation of waiting in BaseEventLoop using a selector is not
> useful for this scenario.

Are you referring to the possibly indefinite blocking call to select in 
._run_once?  A possible rewrite is to block on a tk root.mainloop call 
and make periodic non-blocking select(0) calls in callbacks.  If this 
were done, it might make sense to re-write call_later and call_at to use 
tk's root.after and callback scheduler.  I don't know if the same can be 
done with call_soon while maintaining the detailed run_forever spec in 
the 3.5.1 doc.

> There are probably some possible refactorings in the asyncio package
> to help you reuse a little more code,

I decided to start by reusing all code and just add a tk call. 
TkEventLoop will pass the same test suite as its superclass.


> but all in all I still think it would be very useful to have an
> asyncio loop integrated with Tkinter.

Most definitely.  'An' example is trivial, as posted above.
I will see how far it goes in running other examples.

> (Of course Tkinter does support network I/O,

Tcl's createfilehandler function is only available on unix.  I read 
somewhere that is uses selector polling.

> or somehow figure out how to wait using a Selector
 > *or* tkinter events in the same loop.

I suspect that waiting on one and polling the other is the only 
cross-platform solution.

-- 
Terry Jan Reedy