Re: [Python-ideas] How the heck does async/await work in Python 3.5

On 20.02.2016 07:53, Christian Gollwitzer wrote:
Because of this thread, I finally finished an older post collecting valuable insights from last year discussions regarding concurrency modules available in Python: http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It appears to me that it would fit here well. @python-ideas Back then, the old thread ("Concurrency Modules") was like basically meant to result in something useful. I hope the post covers the essence of the discussion. Some even suggested putting the table into the Python docs. I am unaware of the formal procedure here but I would be glad if somebody could point be at the right direction if that the survey table is wanted in the docs. Best, Sven

Today I also stumbled on this helpful "essay" from Brett Cannon about the same subject http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 On 23 February 2016 at 18:05, Sven R. Kunze <srkunze@mail.de> wrote:

On 2/23/2016 4:25 PM, Joao S. O. Bueno wrote:
In this essay, Brett says that asyncio added an event loop to Python. It did, but it was the second. The tk event loop was added about 20 years ago with tkinter. He also said that this enabled asynchronous programming to Python. It actually added async i/o to go along with async key-mouse-screen user interface. To illustrate, Brett gives this example, which only uses screen output. import asyncio # Borrowed from http://curio.readthedocs.org/en/latest/tutorial.html. @asyncio.coroutine def countdown(number, n): while n > 0: print('T-minus', n, '({})'.format(number)) yield from asyncio.sleep(1) n -= 1 loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(countdown("A", 2)), asyncio.ensure_future(countdown("B", 3))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() It outputs 2 or 1 lines at 1 second intervals. T-minus 2 (A) T-minus 3 (B) T-minus 1 (A) T-minus 2 (B) T-minus 1 (B) and stops after 2 seconds. The tkinter equivalent import tkinter root = tkinter.Tk() #root.withdraw() # optionally hide the window def countdown(number, n): if n > 0: print('T-minus', n, '({})'.format(number)) n -= 1 root.after(1000, countdown, number, n) root.after(1, countdown, 'A', 2) root.after(1, countdown, 'B', 3) root.mainloop() has the same output, but needs either the window close button [X] or other means to stop. (Tk mainloop() is equivalent to asyncio loop.run_forever()) Of course, in normal use, the output would go to a widget in the window, so having the window is not a problem. It would be nice to be able to use a repeatedly resumed generator instead of a repeatedly called callback with tkinter .after also. I am thinking about how that might be done. Maybe a generic callback that calls next(generator) when called. I may also look as Brett's example of interfacing with async and await keywords. -- Terry Jan Reedy

On 24 February 2016 at 02:37, Terry Reedy <tjreedy@udel.edu> wrote:
One of the things I would love to see (but don't have the time to work on) is a GUI event loop based around async/await. It would be a very useful example to make it clear to people that async/await isn't just about network protocols. Paul

On 2/24/2016 4:59 AM, Paul Moore wrote:
Aiming at this was part of the point of my closing comment about adapting generators to the tk mainloop callback interface. The next step would be to do the same for awaitables. However, I need more study of the details of asyncio and the async/await protocol. -- Terry Jan Reedy

On 25 February 2016 at 06:57, Terry Reedy <tjreedy@udel.edu> wrote:
Letting GUI event loops take over running awaitables is part of the rationale for the event loop management features in asyncio: https://docs.python.org/3/library/asyncio-eventloops.html Having an event loop interface adapter in the stdlib tk modules that can be used with asyncio.set_event_loop() would provide a much clearer demonstration of the full power of that approach than the current "*nix selectors vs Windows IOCP" capability (the latter is useful, but doesn't cover the "integration with an existing 3rd party event loop" use case). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2/24/2016 8:05 PM, Nick Coghlan wrote:
One issue I have with asyncio.base_events.BaseEventLoop is that it is not a base event loop. It is a derived network IO event loop which could have been called NetworkIOEventLoop. It 'subclasses' the true base event loop by containment. The latter, call it MinimalEventLoop, comprises the run and call methods listed in https://docs.python.org/3/library/asyncio-eventloop.html ('loop', not 'loops', as above) and perhaps the task methods. In any case, it is only about 1/4 the derived class. Integrating tkinter's event loop only involves this subset. I might have found it easier to think about this if there were a separate base class.
Having an event loop interface adapter in the stdlib tk modules that can be used with asyncio.set_event_loop()
My memory is that any replacement event loop is expected to be a full implmentation of the network io loop, so one cannot just make a gui loop that works with coroutines.
I believe the selector and proactor subclass only the network part of BaseEventLoop. A cross-platform gui might to modify the event part of either. I guess the following should work --- EventLoop = select_event_loop(OS, needs) class TkEventLoop: # modify loop methods as needed set_event_loop(TkEventLoop) --- The heart of BaseEventLoop is .run_forever, and the heart of this is while True: self._run_once() if self._stopping: break The tk equivalent of _run_once, call it tkupdate, is tkinter.Tk().update ("Enter event loop until all pending events have been processed by Tcl. [and return]). If 'while True' is kept as the master loop, it might be sufficient to add a call to tkupdate either before or after the _run_once_ call. I will have to try this and see what happens. This would be easiest to do is there were a self.updater attribute, initially None, which a user could set to an external update function to be called once in each loop. The opposite approach, far more difficult, would be to replace the Python while loop with the tk loop and change other methods as needed. Here is where having a giant class is awkward. There are 13 private attributes. Do the network methods depend on these? Another issue is that the the tk loop object is accessible from python only indirectly. At this point, I am wondering whether it would be easier to forget the asynio network io loop and write from scratch a tkinter-based loop class that has the minimum needed to work with async and await code. But what is that minimum? PEP 492 "assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop." However, AsstractEventLoop is not that, but an abstract network/socket io event loop and a majority of the methods cannot be needed for non-network, non-socket code. Let me try again. What is the actual minimal event loop api required by await? -- Terry Jan Reedy

On Wed, Feb 24, 2016 at 10:49 PM, Terry Reedy <tjreedy@udel.edu> wrote:
That's not correct. If an I/O loop does not implement the network I/O APIs it just cannot be used for network I/O, but it can still implement all the other functionality (like call_later). There are other loops that don't implement everything (e.g. the subprocess and datagram APIs are not supported by Proactor, and signal support is spotty, too). If you find some implementation barrier that prevents that from working it should be fixed. [...]
The await expression (or yield from) does not require an event loop. Heck, it doesn't require futures. It uses coroutines and it is totally up to the manager of those how they are scheduled (read Greg Ewing's very early explanations of yield from -- some have no I/O at all). But there's something else that you might be after. The APIs related to scheduling callbacks (timed or not) are the most fundamental. I find run_until_complete() also fundamental, and requires a Future class. So you need that (but you could have your own Future implementation if you start from scratch or from AbstractEventLoop, leaving the network I/O methods unimplemented. (After all they don't use @abstractmethod so as long as you don't call them you should be okay leaving them unimplemented.) You really should be able to pick and choose, and the comments with section names in AbstractEventLoop are meant to provide you with guidance as to what to implement and what to leave out. When you're implementing this API on top of tkinter, you'll probably find that you'll have to use tkinter's way of sleeping anyways, so the implementation of waiting in BaseEventLoop using a selector is not useful for this scenario. There are probably some possible refactorings in the asyncio package to help you reuse a little more code, but all in all I still think it would be very useful to have an asyncio loop integrated with Tkinter. (Of course Tkinter does support network I/O, so it would be possible to integrate with that, too. Or some hybrid where you somehow figure out how to wait using a Selector *or* tkinter events in the same loop.) -- --Guido van Rossum (python.org/~guido)

On Feb 25, 2016, at 08:47, Guido van Rossum <guido@python.org> wrote:
I remember that back in the late 90s, you often needed a loop like this pseudocode: next_ui = now() + 20ms while True: livesocks = poll(socks, timeout=next_ui-now()) for sock in livesocks: handle(sock) if now() >= next_ui: ui_loop_once() next_ui = now() + 20ms The idea is that GUI/audio/video/game code expects to only fire 20-60 times/second, and may do a lot of work each time; network code dealing with dozens of sockets expects to be fired a lot more often, and to do a lot less work each time. Is this still an issue? If I wanted to write, say, a BitTorrent client like Transmission or uTorrrent using Tkinter, and I fired the Tkinter loop after every selector poll, would it waste too much time checking for events from the OS and onidle handlers and so on thousands of times/second? (And if this is still an issue, is waking 50 times/sec still an acceptable way to solve it, or will you be keeping laptops from sleeping and so on?) I feel like I must have worked on something over the past decade that would tell me that... but everything I can think of (that didn't use something like Qt to abstract it out), the network stuff goes on a different thread or threads, and communicates with the UI by posting events in one direction and kicking a pipe in the other.

On 25 February 2016 at 15:42, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Is this still an issue? If I wanted to write, say, a BitTorrent client like Transmission or uTorrrent using Tkinter, and I fired the Tkinter loop after every selector poll, would it waste too much time checking for events from the OS and onidle handlers and so on thousands of times/second? (And if this is still an issue, is waking 50 times/sec still an acceptable way to solve it, or will you be keeping laptops from sleeping and so on?)
This sounds like we will need a "gear reduction" adapter function to be thrown in: a function that would count the maximum calls-per-second for the low-update part of the code - maybe it could go as a decorator in a helper project for AsyncIO. (disclaimer: I was on a bad day, and yesterday on the local hackerspace Python meeting I help coodinate, people asked for an example tic-tac-toe after I suggested exploring something with Pygame. We ended the night with possibly the first 30 frames/sec tic-tac-toe ever)

2016-02-25 17:47 GMT+01:00 Guido van Rossum <guido@python.org>:
It is actually quite easy to implement an asyncio loop over tkinter once you realise that tkapp.dooneevent() is very similar to poll(), and tkapp.createfilehandler() is very similar to register() (which is not that surprising since tcl use poll() internally if available). Thus it is possible to create a tk selector and reuse all the code in SelectorEventLoop. Unfortunately createfilehandler() is only available on UNIX, so some form of threading is probably innevitable on Windows. class TkSelector(selectors._BaseSelectorImpl): """Selector based on the Tk event loop.""" def __init__(self, app): super().__init__() self.app = app self.ready = [] self.is_timeout = False self.after_key = None def _file_cb(self, fileobj, mask): fd = self._fileobj_lookup(fileobj) self.ready.append((fd, mask)) def _timeout_cb(self): self.is_timeout = True def _reset_state(self): del self.ready[:] self.is_timeout = False if self.after_key: self.app.after_cancel(self.after_key) self.after_key = None def register(self, fileobj, events, data=None): key = super().register(fileobj, events, data) flags = 0 if events & EVENT_READ: flags |= tkinter.READABLE if events & EVENT_WRITE: flags |= tkinter.WRITABLE self.app.createfilehandler(fileobj, flags, self._file_cb) return key def unregister(self, fileobj): key = super().unregister(fileobj) self.app.deletefilehandler(fileobj) return key def select(self, timeout=None): if timeout is None: pass elif timeout <= 0: while self.app.dooneevent(_tkinter.DONT_WAIT): pass self.is_timeout = True else: self.after_key = self.app.after(math.ceil(timeout*1000), self._timeout_cb) while not (self.is_timeout or self.ready): self.app.dooneevent() ret = [] for fd, mask in self.ready: events = 0 if mask & tkinter.WRITABLE: events |= EVENT_WRITE if mask & tkinter.READABLE: events |= EVENT_READ key = self._key_from_fd(fd) if key: ret.append((key, events & key.events)) self._reset_state() return ret class TkEventLoop(asyncio.SelectorEventLoop): """Asyncio-compatible tkinter event loop.""" def __init__(self, app): selector = TkSelector(app) super().__init__(selector=selector)

On 2/26/2016 5:12 PM, Maxime S wrote:
It took me awhile to understand what you mean by 'tkapp'. Instances of tkinter.Tk get an undocumented .tk attribute that is an instance of the undocumented and hidden _tkinter class that is called 'tkapp' in printed representations. In other words, after import tkinter as tk root = tk.Tk() tkapp = root.tk tkapp has a dooneevent method. I found the tcl doc for it at https://www.tcl.tk/man/tcl/TclLib/DoOneEvent.htm Does calling it with DONT_WAIT "TCL_DONT_WAIT - Do not sleep: process only events that are ready at the time of the call." differ from calling root.update? def update(self): """Enter event loop until all pending events have been processed by Tcl.""" self.tk.call('update') If so, how? I interpret 'pending events' as 'events that are ready'. To add to the confusion, 2.7 aliased the tkapp methods (and attributes?) as Tkinter.tkinter functions, where Tkinter.tkinter was the C-coded _tkinter module. At least some people used the alieases. In 3.0, the aliases were removed, and when 'Tkinter' became 'tkinter', 'tkinter because '_tkinter'. (You may know all this, but I want it recorded and possibly added to the docs someday.)
tkapp.createfilehandler() is very similar to register() (which is not that surprising since tcl use poll() internally if available).
As you note below, tkapp.createfilehandler is unix-only and does not exist on Windows in 2.x or 3.x. In 2.7, however, the Tkinter.tkinter.createfilehandle alias did exist on Windows, but with with a value of None.
I am saving this for future reference. -- Terry Jan Reedy

2016-02-27 9:08 GMT+01:00 Terry Reedy <tjreedy@udel.edu>:
I actually meant an instance of Tk(), which also have dooneevent(). Sorry this wasn't clear.
Good point. The code under update() is essentially this (plus some error cheking): if (nargs == 1) { flags = TCL_DONT_WAIT; } else { flags = TCL_IDLE_EVENTS; } while (Tcl_DoOneEvent(flags) != 0) {} So, it is probably much more efficent to call update() than to do the same loop in python as I did, and it avoid messing around with _tkinter undocumented flags.

On 2/27/2016 4:42 AM, Maxime S wrote:
I was fooled because in 3.x dooneevent does not show up on the root completion list. 3.x Tk must have a __getattr__ that pulls attributes from its tk (tkapp) attribute. # 3.5.1
For 2.x, dooneevent does appear as a completion, though I don't know how or why, as I get the same False and False as above. (I will have to see if the completion code changed, or if something else changed to change its meaning.)
Thanks for checking this.
-- Terry Jan Reedy

On 2/25/2016 11:47 AM, Guido van Rossum wrote:
The tk loop already has the run and call functionality (and gui event stuff). The reason I want to integrate tkinter and asyncio is to be able to use the I/O APIs. In particular, I want to experiment with subprocess_exec. Assuming that my reading of the PEP is correct, I want to try using it run code checkers on code (such as the contents of an IDLE editor file). A GSOC student worked out everything but how to do so without blocking and freezing the ui. Beyond that, we need to replace the unreliable (but non-blocking) socket connection between IDLE and the user execution process).
According to the PEP and doc https://docs.python.org/3/library/asyncio-eventloops.html#windows, the Proactor is the loop that supports subprocesses on Windows, while Windows Selector does not. I will find out for sure soon enough.
After lots of reading and thinking, I decided to start with simplest this that might work, and came up with the following surprisingly simple subclass. It works with GUI versions of the first two examples in the doc. It will satisfy many simple cases of using tkinter and asyncio together. I will continue with other examples The docstring describes possible ways to improve the integration. I am not sure whether root.update handles the events handled by update_idletasks (or where 'idletasks' come from). If not, update_idletask() calls may be needed. Serhiy should know. --- '''Proof of concept for integrating asyncio and tk loops. Terry Jan Reedy Run with 'python -i' or from IDLE editor to keep tk window alive. This version simply calls tk root.update in run_forever. Without this, the tk window only appears after loop.stop, unless root.update() is put in callbacks instead. If there are no pending asyncio callbacks, self._run_once will block indefinitely in a .select(None) call, waiting for an I/O event. To ensure that all tk events and callbacks get timely attention, either _run_once should be revised or __init__ should start an asyncio callback loop so that there is always a pending callback. The details will depend on whether the GUI or network IO is intended to get priority attention. ''' import asyncio as ai import threading import tkinter as tk import datetime # for example # On Windows, need proactor for subprocess functions EventLoop = ai.ProactorEventLoop # Following gets ai.SelectorEventLoop # EventLoop = type(ai.get_event_loop()) class TkEventLoop(EventLoop): def __init__(self, root): self.root = root super().__init__() def run_forever(self): # copy with 1 new line for proof of concept """Run until stop() is called.""" self._check_closed() if self.is_running(): raise RuntimeError('Event loop is running.') self._set_coroutine_wrapper(self._debug) self._thread_id = threading.get_ident() try: while True: self._run_once() # can block indefinitely self.root.update() # new line if self._stopping: break finally: self._stopping = False self._thread_id = None self._set_coroutine_wrapper(False) root = tk.Tk() loop = TkEventLoop(root) ai.set_event_loop(loop) # Combine 2 event loop examples from BaseEventLoop doc. hello = tk.Label(root) timel = tk.Label(root) hello.pack() timel.pack() def hello_world(loop): hello['text'] = 'Hello World' loop.call_soon(hello_world, loop) def display_date(end_time, loop): timel['text'] = datetime.datetime.now() if (loop.time() + 1.0) < end_time: loop.call_later(1, display_date, end_time, loop) else: loop.stop() end_time = loop.time() + 5.1 loop.call_soon(display_date, end_time, loop) # Blocking call interrupted by loop.stop() loop.run_forever() ---
The await expression (or yield from) does not require an event loop. Heck, it doesn't require futures.
Without either, are async/await actually useful? The binary/abinary examples just shows that there is not much penalty for using async/await when not needed. On my machine, abinary is pretty consistently 2% slower.
What do you mean by 'manager of [coroutines]'? For loop, event loop, task, the nebulous 'scheduler', or something else?
But there's something else that you might be after.
After getting non-blocking gui and io events (with callbacks) working together, I an after asynchronous iteration. The asyncio modules used cooroutines for suspension, but as far as I can tell, not for iteration. Normal synchronous iteration is fine for compute-bound iterators and a single iterator whose delays are tolerable because the thread or process has nothing better to do. (And the system will switch to something else when the thread blocks, so the cpu is not actually blocked.) But it does not work for concurrent intermitant streams. For example, consider the datetime display in the example above. The code is scattered around in three places. I would like to be able to replace the pieces with synchronous-like code, something like the following. async def display_date(interval, end_time, loop=ai.get_event_loop()): label = tk.Label(root) label.pack() async for tick in timer_soon(interval, end_time, loop): label['text'] = datetime.datetime.now() The asynchronous iterator timer_soon would incorporate call_soon, call_later, and stop calls and 'yield' True at the appointed times. Doing so allows the widget creation and repeated widget event handling to be combined in one function. How do I write it? The async iterator examples in PEP 492 (the second is Example 1) shows all the boiler plate, which I could write from the spec given, but omit the essential detail of how to write the inner cooroutine that gets awaited on. async def fetch_data(self): ... async def _prefetch(self): ... ???
Are you referring to the possibly indefinite blocking call to select in ._run_once? A possible rewrite is to block on a tk root.mainloop call and make periodic non-blocking select(0) calls in callbacks. If this were done, it might make sense to re-write call_later and call_at to use tk's root.after and callback scheduler. I don't know if the same can be done with call_soon while maintaining the detailed run_forever spec in the 3.5.1 doc.
There are probably some possible refactorings in the asyncio package to help you reuse a little more code,
I decided to start by reusing all code and just add a tk call. TkEventLoop will pass the same test suite as its superclass.
but all in all I still think it would be very useful to have an asyncio loop integrated with Tkinter.
Most definitely. 'An' example is trivial, as posted above. I will see how far it goes in running other examples.
(Of course Tkinter does support network I/O,
Tcl's createfilehandler function is only available on unix. I read somewhere that is uses selector polling.
or somehow figure out how to wait using a Selector *or* tkinter events in the same loop.
I suspect that waiting on one and polling the other is the only cross-platform solution. -- Terry Jan Reedy

See also Doug Hellmann article on asyncio, from its serie of "Python 3 Module of the Week" articles: https://pymotw.com/3/asyncio/index.html Victor 2016-02-23 22:25 GMT+01:00 Joao S. O. Bueno <jsbueno@python.org.br>:
participants (9)
-
Andrew Barnert
-
Guido van Rossum
-
Joao S. O. Bueno
-
Maxime S
-
Nick Coghlan
-
Paul Moore
-
Sven R. Kunze
-
Terry Reedy
-
Victor Stinner