[IPython-dev] !commands in the zmq model

[Fernando Perez]

On Sat, Aug 28, 2010 at 5:51 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> Hum, I thought that I had it working. I can't test right now, but I seem
> to remember I had spend quite a lot of time on that.
>
> I am not claiming that my code was terribly good, ( :> "I write code, I
> look at it a year later, I vomit", Michael A., CACR) but if I remember
> correctly, there might be a few tricks to try and keep.
>
> * For pure Python code, as your example above, the trick was to use what
>  I have called the 'RedirectorOutputTrap' that registers a callback on
>  writing to the sys.stdout/sys.stderr. It is implemented in
>  IPython.kernel.core.redirector_output_trap.

For pure python in-process, we're OK because the zmq objects nicely
carry all stdout/err over the network, so we're in good shape.  But I
was using that same little python code as a test, by calling it in a
separate subprocess, to see how rapid async output and interrupts
could be hanlded.

> * For subprocesses, I had to resort to 2 threads, one executing the
>  process, the other polling its stdout (ugly). This logic can be found
>  in IPython.frontend.process.pipedprocess. It also adds a bit of logic
>  to be able to kill the process under Windows, which you might want to
>  keep in mind.

MMh, I've been testing your ipythonx implementation, and it definitely
does *not* show subprocess output asynchronously as it happens.  At
least not with the example above run in a subprocess, and it won't let
me interrupt the subprocess either (I see the KeyboardInterrupt happen
only at the end, and coming from the ipythonx code:


0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 done!
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)

/home/fperez/ipython/branches/0.10.1/IPython/frontend/wx/wx_frontend.pyc
in _on_key_down(self, event, skip)
    401                 if self.debug:
    402                     print >>sys.__stderr__, 'Raising KeyboardInterrupt'
--> 403                 raise KeyboardInterrupt
    404                 # XXX: We need to make really sure we

    405                 # get back to a prompt.


KeyboardInterrupt:


No worries, getting that kind of code to work is devilishly hard.

For now, the pexpect solution works very well, I'll report separately
on that.  And on windows, people will need to live with the suboptimal
piped behavior, until someone either ports pexpect to Windows or
writes something as good based on pipes (if it can be done).

> I am not sure if this code can be of some use to you or not, but just in
> case. I believe that it might be one of the few useful things that came
> out of my work. It actually even seems to have tests :$

Thanks for pointing it out though, I wasn't sure where it was.  We may
very well still use it to have at least as-robust-as-possible a
solution on Windows, so it's great that you brought it up.  On *nix I
think we'll go with the seemingly magical pexpect, which lets us have
our cake and eat it too.

Take care,

f