Line-by-line processing when stdin is not a tty

[RG]

In article <mailman.1941.1281519759.1673.python-list at python.org>,
 Cameron Simpson <cs at zip.com.au> wrote:

> On 11Aug2010 00:11, RG <rNOSPAMon at flownet.com> wrote:
> | When stdin is not a tty, Python seems to buffer all the input through 
> | EOF before processing any of it:
> | 
> | [ron at mickey:~]$ cat | python
> | print 123
> | print 456 <hit ctrl-D here>
> | 123
> | 456
> | 
> | Is there a way to get Python to process input line-by-line the way it 
> | does when stdin is a TTY even when stdin is not a TTY?
> 
> What you're seeing here is not python's behaviour but cat's behaviour.
>
> Almost all programs do line buffering (flush buffer at newline) when the
> file is a terminal (character device) and block buffering (flush when a
> fixed size buffer, typically 8192 bytes or some larger power of 2) when
> the file is not a terminal. This is default behaviour for the stdio
> package.
> 
> So "cat" is simply not feeding any data to python until it has a lot of
> it;

I don't think that's right:

[ron at mickey:~]$ cat | cat
123
123
321
321

Cat seems to flush its buffer after every newline.  Also:

[ron at mickey:~]$ cat -u | python
print 123
print 456
123
456


> We would need to know
> more about your specific task to suggest workarounds.

I'm writing a system in a different language but want to use a Python 
library.  I know of lots of ways to do this (embed a Python interpreter, 
fire up a python server) but by far the easiest to implement is to have 
the main program spawn a Python interpreter and interact with it through 
its stdin/stdout.  In my code I explicitly force the output stream that 
is being piped to Python's stdin to be flushed so I know it's not a 
buffering problem on the input side.

rg