[Python-Dev] Event loops, PyOS_InputHook, and Tkinter

[Michiel Jan Laurens de Hoon]

Dear Pythoneers,

I use Python heavily for scientific computing, originally for 
computational physics and nowadays for computational biology. One of the 
extension modules I support is pygist, a module for scientific 
visualization. For this (and similar) packages to work, it is important 
to have an event loop in Python.

Currently, event loops are available in Python via PyOS_InputHook, a 
pointer to a user-defined function that is called when Python is idle 
(waiting for user input). However, an event loop using PyOS_InputHook 
has some inherent limitations, so I am thinking about how to improve 
event loop support in Python.

As an example, consider the current implementation of Tkinter. What's 
nice about it is that events as well as user-typed Python commands are 
handled without having to call mainloop() explicitly (except on some 
platforms):
"import Tkinter; Tkinter.Tk()" causes a Tk window to pop up that remains 
responsive throughout. It works as follows (using Tkinter as an example; 
pygist works essentially the same):

1) Importing Tkinter causes PyOS_InputHook to be set to the EventHook 
function in _tkinter.c.
2) Before Python calls fgets to read the next Python command typed by 
the user, it checks PyOS_InputHook and calls it if it is not NULL.
3) The EventHook function in _tkinter runs the following loop:
     - Check if user input is present; if so, exit the loop
     - Handle a Tcl/Tk event, if present
     - Sleep for 20 milliseconds
4) Once the EventHook function returns, Python continues to read the 
next user command. After executing the command, return to 2).

However, this implementation has the following problems:
1) Essentially, the event loop is a busy-wait loop with a 20 ms sleep in 
between. An event loop using select() (or equivalent on Windows) will 
give better performance.
2) Since this event loop runs inside Tkinter, there is no way for other 
extension modules to get their messages handled. Hence, we cannot have 
more than one extension module that needs an event loop. As an example, 
it would be nice to have a Tkinter GUI to steer a simulation and a 
(non-Tk) graphics output window to visualize the simulation.
3) Whereas PyOS_InputHook is called when Python is waiting for user 
input, it is not called when Python is waiting for anything else, for 
example one thread waiting for another. For example, IDLE uses two 
threads, one handling the GUI and one handling the user commands. When 
the second thread is waiting for the first thread (when waiting for user 
input to become available), PyOS_InputHook is not being called, and no 
Tkinter events are being handled. Hence, "import Tkinter; Tkinter.Tk()" 
does nothing when executed from an IDLE window. Which means that our 
scientific visualization software can only be run from Python started 
from the command line, whereas many users (especially on Windows) will 
want to use IDLE.

Now the problem I'm facing is that because of its integration with Tcl, 
this cannot be fixed easily with Tkinter as the GUI toolkit for Python. 
If the events to be handled were purely graphical events (move a window, 
repaint a window, etc.), there would be no harm in handling these events 
when waiting for e.g. another thread. With Tkinter, however, we cannot 
enter EventHook while waiting for another thread:
a) because EventHook only returns if user input is available (it doesn't 
wait for threads);
b) because EventHook also runs Tcl/Tk commands, and we wouldn't want to 
run some Tcl commands in some thread while waiting for another thread.

Therefore, as far as I can tell, there is no way to set up a true event 
loop in Python that works nicely with Tkinter, and there is no way to 
have an event loop function from IDLE.

So I'd like to ask the following questions:
1) Did I miss something? Is there some way to get an event loop with 
Tkinter?
2) Will Tkinter always be the standard toolkit for Python, or are there 
plans to replace it at some point?

I realize that Tkinter has been an important part of Python for some 
time now, and I don't expect it to be ditched just because of my event 
loop problems. At the same time, event loop support could use some 
improvement, so I'd like to call your attention to this issue. Tcl 
actually has event loops implemented quite nicely, and may serve as an 
example of how event loops may work in Python.

--Michiel.

-- 
Michiel de Hoon
Center for Computational Biology and Bioinformatics
Columbia University
1150 St Nicholas Avenue
New York, NY 10032