[IPython-dev] !commands in the zmq model

[Brian Granger]

Fernando,

This is really great news.  The simple implementation that was there
covered the basic cases, but left a lot to be desired.

On Sat, Aug 28, 2010 at 12:29 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> Howdy,
>
> I've spent hours on getting a better implementation of
>
> !cmd
>
> in the multiprocess model, that:
>
> 1. shows output as it happens
> 2. is cleanly interruptible.

This is really wonderful.  Thanks for putting in the effort on this.

> We can interrupt subprocesses fine with our current
> (subprocess.Popen-based) implementation, but we can't get their output
> cleanly as it happens.  So in the usual:
>
>    for i in range(200):
>        print i,
>        sys.stdout.flush()
>        time.sleep(0.01)
>    print 'done!'
>
> We only see *all* the numbers at the very end. I haven't found any way
> around this with subprocess, and neither did our old ipythonx
> implementation done by Gael; as best as I understand popen it simply
> can't be done: in pipe mode, the C stdio library does block-buffered
> io on all process execution and there's no clean way I can find, to
> read from a pipe with a timeout.

Bummer, I thought it was smarter than that, but oh well.

> But the good news is that I think I have an implementation that will
> work on *nix (linux/mac), using pexpect.  On Windows we'll have to
> fall back to subprocess.popen(), with its limitations.

I think this is a very reasonable compromise.  Maybe it will motivate
some passionate Windows users to look further at a Windows pexpect ;-)

> In addition, the pexpect implementation gives us the benefit of
> correctly formatted 'ls' output, reverting the surprise Brian had
> initially from ls being only on one column.  Pexpect creates a proper
> pseudo-tty, and knows how to read from it very intelligently.

This is awesome!

> It's a shame it doesn't exist on windows, but it's apparently a very
> non-trivial task to port it (I remember hearing William Stein several
> times comment on how good it would be for Sage to have pexpect on
> Windows, and knowing them, if it was doable they would have already
> done it).
>
> I may not finish this today, I'm too exhausted, but if anyone knows
> this type of problem well, pitch in.  I'm sure I'll make good use of
> any help...

I am going to be focusing on the GUI stuff and ZMQ/PyZMQ stuff this
weekend.  I will be around on IRC and working if you want to chat.

Cheers,

Brian

> Cheers,
>
> f
>
> ps - for reference, the current implementation is along the lines of:
>
> from __future__ import print_function
>
> import sys
> import pexpect
>
> self = get_ipython()
>
> if 1:
>    def system( cmd):
>        cmd = self.var_expand(cmd, depth=2).strip()
>        sh = '/bin/bash'
>        timeout = 0.05 # seconds
>        pcmd = '%s -c %r' % (sh, cmd)
>        try:
>            child = pexpect.run(pcmd, logfile=sys.stdout)
>
>            ## child = pexpect.spawn(sh, ['-c', cmd])
>            ## while True:
>            ##     res = child.expect([pexpect.TIMEOUT, pexpect.EOF], timeout)
>            ##     if res==0:
>            ##         #pass
>            ##         print(child.before, end='')
>            ##     elif res==1:
>            ##         break
>
>        except KeyboardInterrupt:
>            print('\nInterrupted command: %r.' % cmd, file=sys.stderr)
>
>            #return child
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com