[Tutor] capture output from a subprocess while it is being produced

[Albert-Jan Roskam]

Hi,

I have a Tkinter program where I can fire up a commandline program which could run for quite a while (let's say 15 minutes or more). While the program is running, I would like to show the output that I normally see in the terminal (actually, the target platform is windows but now I am on Linux) in a text entry widget. The main purpose of this is to show that "something is happening". 

Anyway, below I tried to make a simplified version, without Tkinter. I fire up the commandline process in a separate thread, and I try to append lines of the output to a deque. The 'ls' command returns 2964 lines on my system. So I expect the deque to contain less items, because every time the while loop in process() sleeps, it would miss a few items. But it doesn't! It's like the ls output is returned as one chunk.

What am I doing wrong?

Thanks!

Albert-Jan


from subprocess import Popen, PIPE
from threading import Thread
from collections import deque
import sys
import time
import os.path


def process(dq):
    """Run a commandline program, with lots of output"""
    cwd = os.path.dirname(sys.executable)
    proc = Popen("ls -lF", cwd=cwd, shell=True, stdout=PIPE, stderr=PIPE)  
    #print len(proc.stdout.readlines())  # 2964
    output = errors = None
    while True:
        try:
            #output, errors = proc.communicate() # "Wait for process to terminate."  
            output = proc.stdout.read()  # docs: "Use communicate() rather than .stdin.write, .stdout.read or .stderr.read"
            errors = proc.stderr.read()
            proc.stdout.flush()
        except ValueError:
            break
        finally:
            if errors:
                raise RuntimeError(str(errors))

        if output:
            dq.append(output)
        time.sleep(0.0001)
        if not output:
            break

def update(dq):
    """This is supposed to return a new, not necessarily contiguous, chunk of output"""
    if dq:
        result = dq.pop()
        print time.asctime().center(79, "*")
        print result
        return result

if __name__ == "__main__": 
    
    dq = deque()
    output_thread = Thread(target=process, args=(dq,), name="output_thread")
    #output_thread.setDaemon(True)
    output_thread.start()
    
    
    lines = []
    while output_thread.is_alive():
        update(dq)
        time.sleep(0.1)
        if not dq: 
            break
        for item in dq:
            lines += item.splitlines()
    output_thread.join()
    print len(lines)