[IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook

Brian Granger ellisonbg.net at gmail.com
Sun Feb 8 15:15:53 EST 2009


Wow, thanks for taking the time to give us all the details of the
issues you ran into.  This is *super* helpful as we think about how to
move forward with refactoring the core to better support usage cases
like this.

At the same time, this type of thing will keep me up at night.  Our
core problem right now with the IPython core is that it is welded and
duct taped to the terminal and readline, making it difficult to use
outside of that setting.  The things you have done to get this working
(in my mind) are simply welding the Core to wx.  If we have to do
things like this for each GUI toolkit, we are back to having a core
that is non-reusable and difficult to maintain, test, extend, debug,
etc.  To make matters worse, these types of hacks are notoriously

The killer though is that, at the end of the day, if a user runs a
long computation (let's say a big PyMC computation for example), the
entire GUI will become unresponsive.  To me, the above hacks seems
like an awful lot of work to get a solution that is still

I think all of us have a vision that
IPython/numpy/scipy/matplotlib/sympy/ets/etc will allow us to build
scientific computing environments that blow the doors off of
Mathematica, Matlab, Maple, IDL, etc.  However, if our software always
has to be explained with "oh, that, yeh, when you run that type of
code, the GUI will freeze," we will never reach that goal.

So, let's try to figure out a real solution to these problems that
isn't crippled and let's us write robust, reusable and hackable code.



On Sun, Feb 8, 2009 at 2:22 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sat, Feb 07, 2009 at 10:09:05AM -0800, Brian Granger wrote:
>> > I am not using any thread in the wx GUI. This was a very important design
>> > choice, IMHO. I am fiddling a lot with the mainloop to keep as much a
>> > possible the terminal reactive when a computation is going on, but there
>> > are no threads (except when a subprocess is used, but I was very careful
>> > to make them play well with the mainloop).
>> Because we are likely moving in this direction, I want to understand
>> this better.  What do you mean by "fiddling with the mainloop?"  Is
>> this "fiddling" optional?  If it causes bugs in your GUI (is this what
>> you are saying?) why do you do it?  You imply that the terminal was
>> not responsive until you did this extra stuff.  Can you say a bit
>> more?
> Sorry, I was out.
> When I say fiddling with the event loop, I mean many things. The big deal
> is that you cannot get a refresh, or a key event without running the
> event loop. The user is executing code, and executing it in the event
> loop, bocking it, as we are going non-multithreaded, but it is
> imperative, sequential code, and we really need event-driven code that
> yields to the event-loop in order to get refreshes and events. So I
> overloaded a few operations, such as print and raw_input to yield back to
> the event loop. Now I couldn't simply use wx.Yield blindly, because it is
> fragile, and getting this right was a bit tricky.
> For screen refresh, you want to refresh ASAP, because you don't want
> debug statements of a long running calculation to pop up only after it
> finish. So I had to trigger controlled refresh events to do refreshes on
> prints. See method 'write', line 135 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/console_widget.py
> This method is injected in a sys.stdout and sys.stderr replacement. I had
> to be careful not to get a refresh called when we where already in a
> wx.Yield. This is why I had to add the refresh keyword, and use it
> cleverly. In addition, printing a lot of text to the terminal would slow
> down a lot, because a refresh is expensive. This is why I introduced the
> notion of a delayed refresh: there is a timer, and a queue. Trying to
> refresh triggers the timer, and the refresh happens delayed, so that the
> queue fills in. Obviously this means that if you are unlucky and the
> write method does not get called for a long time while you where waiting
> for the timer to empty, you don't get a refresh, because you don't enter
> the event loop again.
> I had to be careful with all other possible interactions, such as
> keyboard callbacks, not to trigger the Yield. Also, I had to write an
> OS-level output redirection that would weave the captured flux for
> compiled code with the Python flux, and call the refresh callback ASAP,
> this is done in
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/kernel/core/redirector_output_trap.py
> For the raw_input, I had to start a second event loop, to process
> keyboard event while the main event loop was blocked by the user code
> executing. See method 'raw_input', line 176 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
> There is readline-like logic that is used here to determine when this
> second event loop is terminated and the user code can resume.
> Finally, the trickiest part was running subprocesses. The reason being
> that I needed both synchronous prints, and key-event processing. The
> subprocess in run in a separate thread, see
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/_process/pipedprocess.py
> Its input stream is fed in by a raw_input-like mechanism, with a separate
> event-loop running, see method system_call, line 197 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
> . I have to use a separate event-loop because the call to a subprocess is
> blocking for the user code. The output stream is polled (yeah, I am not
> proud of the polling, I couldn't figure out something better), and fed to
> a 'buffered_write' method. No need to do a wx.Yield, here, as an
> event-loop is running in a separate thread. Which just need to reduce the
> number of write called per unit of time to avoid flooding the event loop,
> and to make sure we are thread-safe, see method 'buffered_write', line
> 264 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
> The termination of the subprocess is wired to a callback which flushes
> the buffer and terminates the inner event loop.
> I believe that this is mostly it. There is a lot more of little subtle
> things you have to be careful with: not calling your callbacks in
> infinite loops, making sure methods that can be called in other thread
> are thread safe, making sure you don't screw the order of events. This is
> quite tricky code, I believe. The good news is that all event loops are
> quite similar (the Tk one is different, in Python, as it runs in a
> separate OS thread, but we can forget that, and use it like the others).
> It would be possible to build an object that would abstract scintilla and
> the event-loop control out of different GUI toolkits, and expose a common
> interface. This would actually be fairly easy and would get us an
> interactive frontend for all toolkits.
> Gaël

More information about the IPython-dev mailing list