New line conversion with Popen attached to a pty

Jonathan Harden jonathan-h at
Thu Jun 20 12:20:20 CEST 2013


We have a class which executes external processes in a controlled environment and does "things" specified by the client program with each line of output. To do this we have been attaching stdout from the subprocess.Popen to a pseudo terminal (pty) made with pty.openempty and opened with os.fdopen. I noticed that we kept getting a bunch of extra new line characters. 

This is all using python 2.6.4 in a centos6 environment.

After some investigation I realised we needed to use universal_newline support so I enabled it for the Popen and specified the mode in the fdopen to be rU. Things still seemed to be coming out wrong so I wrote up a test program boiling it down to the simplest cases (which is at the end of this message). The output I was testing was this:

as seen through hexdump -C:

> hexdump -C output.txt 
00000000  46 61 6b 65 0d 0a 44 61  74 61 0d 0a              |Fake..Data..|

Now if I do a simple subprocess.Popen and set the stdout to subprocess.PIPE, then do I get the correct output of


When do the Popen attached to a pty I end up with


Does anyone know why the newline conversion would be incorrect, and what I could do to fix it? In fact if anyone even has any pointers to where this might be going wrong I'd be very helpful, I've done a lot of hours of fiddling with this and googling to no avail.



#!/usr/bin/env python2.6.4
import os
import pty
import subprocess
import select
import fcntl

class TestRead(object):

    def __init__(self):
        super(TestRead, self).__init__()

    def outputPipe(self):
        p1 = subprocess.Popen(
            ("/bin/cat", "output.txt"),
        print "1: %r" %

    def outputPty(self):
        outMaster, outSlave = pty.openpty()
        fcntl.fcntl(outMaster, fcntl.F_SETFL, os.O_NONBLOCK)

        p2 = subprocess.Popen(
            ("/bin/cat", "output.txt"),

        with os.fdopen(outMaster, 'rU') as pty_stdout:
            while True:
                    rfds, _, _ =[pty_stdout], [], [], 0.1)
                except select.error:

            for fd in rfds:
                buf =
                print "2: %r" % buf

if __name__ == "__main__":
    t = TestRead()

More information about the Python-list mailing list