
I have been researching the question of how to ask a file descriptor how much data it has waiting for the next sequential read, with a view to discovering what cross-platform behavior we could count on for a hypothetical `waiting' method in Python's built-in file class. 1: Why bother? I have these main applications in mind: 1. Detecting EOF on a static plain file. 2. Non-blocking poll of a socket opened in non-blocking mode. 3. Non-blocking poll of a FIFO opened in non-blocking mode. 4. Non-blocking poll of a terminal device opened in non-blocking mode. These are all frequently requested capabilities on C newsgroups -- how often have *you* seen the "how do I detect an individual keypress" question from beginning programmers? I believe having these capabilities would substantially enhance Python's appeal. 2: What would be under the hood? Summary: We can do this portably, and we can do it with only one (1) new #ifdef. Our tools for this purpose will be the fstat(2) st_size field and the FIONREAD ioctl(2) call. They are complementary. In all supposedly POSIX-conformant environments I know of, the st_size field has a documented meaning for plain files (S_IFREG) and may or may not give a meaningful number for FIFOs, sockets, and tty devices. The Single Unix Specification is silent on the meaning of st_size for file types other than regular files (S_IFREG). I have filed a defect report about this with OpenGroup and am discussing appropriate language with them. (The last sentence of the Inferno operating system's language on stat(2) is interesting: "If the file resides on permanent storage and is not a directory, the length returned by stat is the number of bytes in the file. For directories, the length returned is zero. Some devices report a length that is the number of bytes that may be read from the device without blocking.") The FIONREAD ioctl(2) call, on the other hand, returns bytes waiting on character devices such as FIFOs, sockets, or ttys -- but does not return a useful value for files or directories or block devices. The FIONREAD ioctl was supported in both SVr4 and 4.2BSD. It's present in all the open-source Unixes, SunOS, Solaris, and AIX. Via Google search I have discovered that it's also supported in the Windows Sockets API and the GUSI POSIX libraries for the Macintosh. Thus, it can be considered portable for Python's purposes even though it's rather sparsely documented. I was able to obtain confirming information on Linux from Linus Torvalds himself. My information on Windows and the Mac is from Gavriel State, formerly a lead developer on Corel's WINE team and a programmer with extensive cross-platform experience. Gavriel reported on the MSCRT POSIX environment, on the Metrowerks Standard Library POSIX implementation for the Mac, and on the GUSI POSIX implementation for the Mac. 2.1: Plain files Torvalds and State confirm that for plain files (S_IFREG) the st_size field is reliable on all three platforms. On the Mac it gives the file's data fork size. One apparent difficulty with the plain-file case is that POSIX does not guarantee anything about seek_t quantities such as lseek(2) returns and the st_size field except that they can be compared for equality. Thus, under the strict letter of POSIX law, `waiting' can be used to detect EOF but not to get a reliable read-size return in any other file position. Fortunately, this is less an issue than it appears. The weakness of the POSIX language was a 1980s-era concession to a generation of mainframe operating systems with record-oriented file structures -- all of which are now either thoroughly obsolete or (in the case of IBM VM/CMS) have become Linux emulators :-). On modern operating systems under which files have character granularity, stat(2) emulations can be and are written to give the right result. 2.2: Block devices The directory case (S_IFDIR) is a complete loss. Under Unixes, including Linux, the fstat(2) size field gives the allocated size of the directory as if it were a plain file. Under MSCRT POSIX the meaning is undocumented and unclear. Metroworks returns garbage. GUSI POSIX returns the number of files in the directory! FIONREAD cannot be used on directories. Block devices (S_IFBLK) are a mess again. Linus points out that a system with removable or unmountable volumes *cannot* return a useful st_size field -- what happens when the device is dismounted? 2.3: Character devices Pipes and FIFOs (S_IFIFO) look better. On MSCRT the fstat(2) size field returns the number of bytes waiting to be read. This is also true under current Linuxes, though Torvalds says it is "an implementation detail" and recommends polling with the FIONREAD ioctl instead. Fortunately, FIONREAD is available under Unix, Windows, and the Mac. Sockets (S_IFSOCK) look better too. Under Linux, the fstat(2) size field gives number of bytes waiting. Torvalds again says this is "an implementation detail" and recommends polling with the FIONREAD ioctl. Neither MSCRT POSIX nor Metroworks has direct support for sockets. GUSI POSIX returns 1 (!) in the st_size field. But FIONREAD is available under Unix, Windows, and the GUSI POSIX libraries on the Mac. Character devices (S_IFCHR) can be polled with FIONREAD. This technique has a long history of use with tty devices under Unix. I don't know whether it will work with the equivalents of terminal devices for Windows and the Mac. Fortunately this is not a very important question, as those are GUI environments with the terminal devices are rarely if ever used. 3. How does this turn into Python? The upshot of our portability analysis is that by using FIONREAD and fstat(2), we can get useful results for plain files, pipes, and sockets on all three platforms. Directories and block devices are a complete loss. Character devices (in particular, ttys) we can poll reliably under Unix. What we'll get polling the equivalents of tty or character devices under Windows and the Mac is presently unknown, but also unimportant. My proposed semantics for a Python `waiting' method is that it reports the amount of data that would be returned by a read() call at the time of the waiting-method invocation. The interpreter throws OSError if such a report is impossible or forbidden. I have enclosed a patch against the current CVS sources, including documentation. This patch is tested and working against plain files, sockets, and FIFOs under Linux. I have also attached the Python test program I used under Linux. I would appreciate it if those of you on Windows and Macintosh machines would test the waiting method. The test program will take some porting, because it needs to write to a FIFO in background. Under Linux I do it this way: (echo -n '%s' >testfifo; echo 'Data written to FIFO.') & I don't know how to do the equivalent under Windows or Mac. When you run this program, it will try to mail me your test results. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Sometimes it is said that man cannot be trusted with the government of himself. Can he, then, be trusted with the government of others? -- Thomas Jefferson, in his 1801 inaugural address

This didn't compile under Windows. I have a patch (against CVS) that compiles, but doesnt appear to work (and will be forwarded to Eric under seperate cover) [news flash :-) Changing the open call to add "rb" as the mode makes it work - text v binary bites again] I didn't try any sort of fifo test. The sockets test failed with a socket error, but would certainly have failed had the socket connected, as my patch includes: #ifndef S_ISSOCK # define S_ISSOCK(mode) (0) #endif I have no idea if it managed to mail the results, but I guess not, so the output is below. The test file (after some small mods, including the "rb" param) is indeed 4252 bytes long. Hope this is useful! Mark. This program tests the `waiting' method of file objects. Good, you're running a patched Python with `waiting' available. First, plain files: There are 4252 bytes waiting to be read in this file. Please check this with your OS's directory tools. I'll now read a random number (3091) of bytes. The waiting method sees 1161 bytes left. 3091 + 1161 = 4252. That's consistent. Test passed. Now let's see if we can detect EOF reliably. I'll do a read()...the waiting method now returns 0 That looks like EOF. Now sockets: Connecting to imap.netaxs.com's IMAP server now... Traceback (most recent call last): File "c:\temp\waiting_test.py", line 57, in ? sock.connect(("imap.netaxs.com", 143)) File "<string>", line 1, in connect socket.error: (10060, 'Operation timed out')

Fredrik Lundh <fredrik@effbot.org>:
Fortunately, this is less an issue than it appears.
only if you ignore Windows...
I don't understand this. Explain? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Sometimes the law defends plunder and participates in it. Sometimes the law places the whole apparatus of judges, police, prisons and gendarmes at the service of the plunderers, and treats the victim -- when he defends himself -- as a criminal. -- Frederic Bastiat, "The Law"

I have a strong -1 on this. It violates the abstraction of Python file objects as a thin layer on top of C's stdio. I don't want to add any new features that can only be implemented by digging under the hood of stdio. There is no standard way to figure out how much data is buffered inside the FILE struct, so doing any kind of system call on the file descriptor is insufficient unless the file is opened in unbuffered mode -- not an attractive option in most applications. Apart from the stdio buffering issue, apps that really want to do this can already look under the hood, thereby clearly indicating that they make more assumptions about the platform than portable Python. For static files, an app can call os.fstat() itself. But I think it's a weakness of the app if it needs to resort to this -- Eric's example that motivated this desire in him didn't convince me at all. For sockets, and on Unix for pipes and FIFOs, an app can use the select module to find out whether data can be read right away. It doesn't tell how much data, but that's unnecessary -- at least for sockets (where this is a very common request), the recv() call will return short data rather than block for more if at least one byte can be read. (For pipes and FIFOs, you can use fstat() or FIONREAD if you really want -- but why bother?) --Guido van Rossum (home page: http://www.python.org/~guido/)

This didn't compile under Windows. I have a patch (against CVS) that compiles, but doesnt appear to work (and will be forwarded to Eric under seperate cover) [news flash :-) Changing the open call to add "rb" as the mode makes it work - text v binary bites again] I didn't try any sort of fifo test. The sockets test failed with a socket error, but would certainly have failed had the socket connected, as my patch includes: #ifndef S_ISSOCK # define S_ISSOCK(mode) (0) #endif I have no idea if it managed to mail the results, but I guess not, so the output is below. The test file (after some small mods, including the "rb" param) is indeed 4252 bytes long. Hope this is useful! Mark. This program tests the `waiting' method of file objects. Good, you're running a patched Python with `waiting' available. First, plain files: There are 4252 bytes waiting to be read in this file. Please check this with your OS's directory tools. I'll now read a random number (3091) of bytes. The waiting method sees 1161 bytes left. 3091 + 1161 = 4252. That's consistent. Test passed. Now let's see if we can detect EOF reliably. I'll do a read()...the waiting method now returns 0 That looks like EOF. Now sockets: Connecting to imap.netaxs.com's IMAP server now... Traceback (most recent call last): File "c:\temp\waiting_test.py", line 57, in ? sock.connect(("imap.netaxs.com", 143)) File "<string>", line 1, in connect socket.error: (10060, 'Operation timed out')

Fredrik Lundh <fredrik@effbot.org>:
Fortunately, this is less an issue than it appears.
only if you ignore Windows...
I don't understand this. Explain? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Sometimes the law defends plunder and participates in it. Sometimes the law places the whole apparatus of judges, police, prisons and gendarmes at the service of the plunderers, and treats the victim -- when he defends himself -- as a criminal. -- Frederic Bastiat, "The Law"

I have a strong -1 on this. It violates the abstraction of Python file objects as a thin layer on top of C's stdio. I don't want to add any new features that can only be implemented by digging under the hood of stdio. There is no standard way to figure out how much data is buffered inside the FILE struct, so doing any kind of system call on the file descriptor is insufficient unless the file is opened in unbuffered mode -- not an attractive option in most applications. Apart from the stdio buffering issue, apps that really want to do this can already look under the hood, thereby clearly indicating that they make more assumptions about the platform than portable Python. For static files, an app can call os.fstat() itself. But I think it's a weakness of the app if it needs to resort to this -- Eric's example that motivated this desire in him didn't convince me at all. For sockets, and on Unix for pipes and FIFOs, an app can use the select module to find out whether data can be read right away. It doesn't tell how much data, but that's unnecessary -- at least for sockets (where this is a very common request), the recv() call will return short data rather than block for more if at least one byte can be read. (For pipes and FIFOs, you can use fstat() or FIONREAD if you really want -- but why bother?) --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (4)
-
Eric S. Raymond
-
Fredrik Lundh
-
Guido van Rossum
-
Mark Hammond