Nonblocking IO on popen pipe (isn't nonblocking)

[Noah]

Damn... The only problem I see with this explaination is
that this would imply that popen is almost totally useless.
Why bother opening a pipe with both in and out streams
if can't actually read and write from both streams?
I could have sworn that I have seen interactive programs
such as telnet and ftp scripted from Python.

I looked at the pty stuff, but I was going to implement that
AFTER I got pipes working. Pipes seem to be the most portable
and intuitive way to solve this problem.

Damnit! If I can do it in Perl I'm sure I can make it work in Python!

Noah


-----Original Message-----
From: python-list-admin at python.org
[mailto:python-list-admin at python.org]On Behalf Of Donn Cave
Sent: Thursday, April 11, 2002 9:22 AM
To: python-list at python.org
Subject: Re: Nonblocking IO on popen pipe (isn't nonblocking)


Quoth "Noah" <noah at noah.org>:
| I'm having headaches with nonblocking IO on a pipe.
| I opened a UNIX command through os.popen4.
| Then I set O_NONBLOCK on the output stream of the pipe.
| Finally I did a select() on the output pipe to wait for data.
| If select returns I would try to read the data.
| The problem is that select() is blocking. Select should block 
| until some data is available on the pipe, so I assume that
| select is never even seeing any data on the pipe.
| I added a timeout to select() and then tried to read the
| pipe, but it was empty (read() returned '').
|
| One program that I am attempting to talk to via a pipe is ftp.
| Any interactive program I open will block. 
| If I try open a program that terminates like "ls" or "uname -a" 
| then my script works as expected.
|
| What am I doing wrong that is causing interactive programs to block?

It isn't you.  I'd argue that your O_NONBLOCK is unnecessary, as long
as you're using select anyway, but it won't make any difference if
the command on the other end doesn't send the data.  It doesn't send
the data because its output goes through C library stdio buffers, which
treat a pipe like a disk file and buffer up whole blocks.  There could
be other problems, like some of the data is actually going to stderr,
but when you say the only programs that work are the ones that terminate,
that's a hallmark of this problem (buffers are flushed on exit.)

The only solution is to get away from popen2 and pipes, and use a
pseudotty instead.  2.1 and up have a usable posix.openpty() on some
UNIX platforms, and there's an older pty.py module that may or may
not be useful.  Ptys are sort of tricky and cross-platform support is
poor, but they'll work better in a couple of ways for applications
that are intended for interactive use.

	Donn Cave, donn at u.washington.edu