[IPython-dev] !commands in the zmq model

[Gael Varoquaux]

On Sat, Aug 28, 2010 at 12:29:53AM -0700, Fernando Perez wrote:
> We can interrupt subprocesses fine with our current
> (subprocess.Popen-based) implementation, but we can't get their output
> cleanly as it happens.  So in the usual:

>     for i in range(200):
>         print i,
>         sys.stdout.flush()
>         time.sleep(0.01)
>     print 'done!'

> We only see *all* the numbers at the very end. I haven't found any way
> around this with subprocess, and neither did our old ipythonx
> implementation done by Gael; as best as I understand popen it simply
> can't be done: in pipe mode, the C stdio library does block-buffered
> io on all process execution and there's no clean way I can find, to
> read from a pipe with a timeout.

Hum, I thought that I had it working. I can't test right now, but I seem
to remember I had spend quite a lot of time on that.

I am not claiming that my code was terribly good, ( :> "I write code, I
look at it a year later, I vomit", Michael A., CACR) but if I remember
correctly, there might be a few tricks to try and keep.

* For pure Python code, as your example above, the trick was to use what
  I have called the 'RedirectorOutputTrap' that registers a callback on
  writing to the sys.stdout/sys.stderr. It is implemented in
  IPython.kernel.core.redirector_output_trap.

* For subprocesses, I had to resort to 2 threads, one executing the
  process, the other polling its stdout (ugly). This logic can be found
  in IPython.frontend.process.pipedprocess. It also adds a bit of logic
  to be able to kill the process under Windows, which you might want to
  keep in mind.

I am not sure if this code can be of some use to you or not, but just in
case. I believe that it might be one of the few useful things that came
out of my work. It actually even seems to have tests :$

Cheers,

Gaël