readline() blocks after select() says there's data??

[Donn Cave]

Quoth wealthychef at mac.com (wealthychef):
...
|> The complementary problem, of course, is that once you have invoked
|> readline(), it will read all the data available on the file descriptor,
|> or a lot of it anyway, and return only the first line.  The next time,
|> it will return the next line from its buffer, but you have no way to
|> know that it has another line waiting there.  select() can't see it.
|> There are solutions, I think the easiest thing in most cases is to
|> throw away the file object and just use the file descriptor (fileno())
|> (or sockets in this case.)
|
| This is even more frightening.  I don't want to lose any data!  How
| can I protect myself from this?  Why is using the file descriptor
| better?

Well, you're not exactly losing data, it's just buffered, though that
may be too fine a point depending on the circumstances.

The point with file descriptor is to use system I/O functions on the
device, and avoid buffered C I/O.  Basically because select is a
system I/O function.  If there were a C analogue to select, then you
could use it with C buffered file objects, but there is no such thing.

When select tells you "this thing is ready to read", it means the
device is ready for a system level read(), as in os.read(fd, bufsize).
So do that, and you'll get what select was telling you about.  It's
really simple.  The rules are the same, if you get an empty string
it's at "end of file" (the pipe closed.)

I have lost track of what kind of devices we're actually talking about -
I'm seeing the word "socket", but then what looks like popen2.Popen3.
Note that in Python, sockets are are normally socket objects, with recv()
methods etc., but these are unbuffered and recv() is like os.read().
Pipes (as created by Popen3) are either integer file descriptors or
file objects, there isn't any special system level pipe object.

As long as you're on UNIX, there's a certain purity of abstraction here
that you can exploit if you want it just to simplify matters - you can
get the file descriptor with s.fileno(), and you can use os.read() with
that - or you can use sock.recv() if you prefer.

	Donn Cave, donn at u.washington.edu