Why do we flush before truncating?
http://www.python.org/sf/801631 gives a failing program on Windows, paraphrased: f = file('test.dat', 'wb') f.write('1234567890') # 10 bytes f.close() f = file('test.dat','rb+') f.read(5) print f.tell() # prints 5, as expected f.truncate() # leaves the file at 10 bytes print f.tell() # prints 10 The problem is that fileobject.c's file_truncate() calls fflush() before truncating. The C standard says that the effect of calling fflush() is undefined if the most recent operation on a stream opened for update was an input operation. The stream is indeed opened for update here, and the most recent operation performed by the *user* was indeed a read. It so happens that MS's fflush() changes the file position then. But the user didn't call fflush(), Python did, so we can't blame the user for relying on undefined behavior here. The problem can be repaired inside file_truncate() by seeking back to the original file position after the fflush() call -- but the original file position isn't always available now, so I'd also have to add another call to _portable_ftell() before the fflush() to find it. So that gets increasingly complicated. Much simpler would be to remove this block of code (which does fix the test program's problem on Windows, by simply getting rid of the undefined operation): /* Flush the file. */ Py_BEGIN_ALLOW_THREADS errno = 0; ret = fflush(f->f_fp); Py_END_ALLOW_THREADS if (ret != 0) goto onioerror; I don't understand why we're flushing the file. ftruncate() isn't a standard C function, so the standard sheds no light on why we might be doing that. AFAICT, POSIX/SUS doesn't give a reason to flush either: http://www.opengroup.org/onlinepubs/007904975/functions/ftruncate.html
On Sat, Sep 06, 2003 at 01:11:14PM -0400, Tim Peters wrote:
I don't understand why we're flushing the file. ftruncate() isn't a standard C function, so the standard sheds no light on why we might be doing that.
The fflush call as been there forever. The truncate method was added in 2.36 by Guido. I think the code was actually from Jim Roskind: http://groups.google.com/groups?selm=199412070213.SAA06932%40infoseek.com He says: Note that since the underlying ftruncate operates on a file descriptor (believe it or not), it was necessary to fflush() the stream before performing the truncate. I thought about doing a seek() as well, but could not find compelling reason to move the stream pointer. That still gives me no clue as to why the fflush() was deemed necessary. I found this posting: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=35E0DB62.1BDD2D30%40taraz.kz&rnum=1 but, AFACK, the reason the program is not working the way the poster expects is the missing fflush() call before the lseek() call (not the fflush() before the ftruncate()). Neil
http://www.python.org/sf/801631
gives a failing program on Windows, paraphrased:
f = file('test.dat', 'wb') f.write('1234567890') # 10 bytes f.close()
f = file('test.dat','rb+') f.read(5) print f.tell() # prints 5, as expected
f.truncate() # leaves the file at 10 bytes print f.tell() # prints 10
The problem is that fileobject.c's file_truncate() calls fflush() before truncating. The C standard says that the effect of calling fflush() is undefined if the most recent operation on a stream opened for update was an input operation. The stream is indeed opened for update here, and the most recent operation performed by the *user* was indeed a read. It so happens that MS's fflush() changes the file position then. But the user didn't call fflush(), Python did, so we can't blame the user for relying on undefined behavior here.
The problem can be repaired inside file_truncate() by seeking back to the original file position after the fflush() call -- but the original file position isn't always available now, so I'd also have to add another call to _portable_ftell() before the fflush() to find it.
So that gets increasingly complicated. Much simpler would be to remove this block of code (which does fix the test program's problem on Windows, by simply getting rid of the undefined operation):
/* Flush the file. */ Py_BEGIN_ALLOW_THREADS errno = 0; ret = fflush(f->f_fp); Py_END_ALLOW_THREADS if (ret != 0) goto onioerror;
I don't understand why we're flushing the file. ftruncate() isn't a standard C function, so the standard sheds no light on why we might be doing that. AFAICT, POSIX/SUS doesn't give a reason to flush either:
http://www.opengroup.org/onlinepubs/007904975/functions/ftruncate.html
ftruncate() is not a standard C function; it's a standard Unix system call. It works on a file descriptor (i.e. a small int), not on a stream (i.e. a FILE *). The fflush() call is necessary if the last call was a write, because in that case the stream's buffer may contain data that the OS file descriptor doesn't have yet. But ftruncate() is irrelevant, because on Windows, it is never called; there's a huge #ifdef MS_WINDOWS block containing Windows specific code, starting with the comment /* MS _chsize doesn't work if newsize doesn't fit in 32 bits, so don't even try using it. */ and the ftruncate() call is made in the #else part. It also looks like the MS_WINDOWS specific code block *does* attempt to record the current file position and seek back to it -- however it does this after fflush() has already messed with it. So perhaps moving the fflush() call into the #else part and doing something Windows-specific instead of calling fflush() to ensure the buffer is flushed inside the MS_WINDOWS part would be the right solution. I just realize that I have always worked under the assumption that fflush() after a read is a no-op; I just checked the 89 std and it says it is undefined. (I must have picked up that misunderstanding from some platform-specific man page.) This can be fixed by doing a ftell() followed by an fseek() call; this is required to flush the buffer if there was unwritten output data in the buffer, and is always allowed. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Neil Schemenauer]
The fflush call as been there forever. The truncate method was added in 2.36 by Guido. I think the code was actually from Jim Roskind:
http://groups.google.com/groups?selm=199412070213.SAA06932%40infoseek.com
He says:
Note that since the underlying ftruncate operates on a file descriptor (believe it or not), it was necessary to fflush() the stream before performing the truncate. I thought about doing a seek() as well, but could not find compelling reason to move the stream pointer.
That still gives me no clue as to why the fflush() was deemed necessary.
Ack, I glossed over the fileno() call in our file_truncate(). It's usually a Very Bad Idea to mix stream I/O and lower-level I/O operations without flushing your pants off, but I'm having a hard time thinking of a specific reason for doing so in the truncate case. Better safe than trying to out-think all possible implementations, though!
I found this posting:
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=35E 0DB62.1BDD2D30%40taraz.kz&rnum=1
but, AFACK, the reason the program is not working the way the poster expects is the missing fflush() call before the lseek() call (not the fflush() before the ftruncate()).
I think that's right. In the Python case, I verified in a debugger that the file position is 5 immediately before the fflush() call, and 10 immediately after it. It's surprising, but apparently OK by the C std. [Guido
ftruncate() is not a standard C function;
I suppose that clarifies my immediately preceding
ftruncate() isn't a standard C function,
<wink>?
it's a standard Unix system call.
Yes, and I gave a link to the current POSIX/SUS ftruncate() specification.
It works on a file descriptor (i.e. a small int), not on a stream (i.e. a FILE *).
Right, and I missed that, primarily because Windows doesn't have ftruncate() so I wasn't looking at that part of the code.
The fflush() call is necessary if the last call was a write, because in that case the stream's buffer may contain data that the OS file descriptor doesn't have yet.
I'm not really clear on why that should matter in the specific case of truncating a file, but will just live with it.
But ftruncate() is irrelevant, because on Windows, it is never called; there's a huge #ifdef MS_WINDOWS block containing Windows specific code ...
Right, I wrote that code. Windows has no way to say "here's a file, change the size to such-and-such"; the only way is to set the file pointer to the desired size, and then call the no-argument Win32 SetEndOfFile(); Python *used* to use the MS C _chsize() function, but that did insane things when passed a "large" size; the SetEndOfFile() code was introduced as part of fixing Python's Windows largefile support.
... It also looks like the MS_WINDOWS specific code block *does* attempt to record the current file position and seek back to it
Yes, because the file position must be changed on Windows in order to change the file size, but *Python's* docs promise that file.truncate() doesn't change the current position (which is natural behavior under POSIX ftruncate() but strained on Windows).
-- however it does this after fflush() has already messed with it.
Note that in the Windows test case, it's not simply that the current position wasn't preserved across the file.truncate() call, it's also that the file didn't change size. It's very easy to fix the former while leaving the latter broken.
So perhaps moving the fflush() call into the #else part and doing something Windows-specific instead of calling fflush() to ensure the buffer is flushed inside the MS_WINDOWS part would be the right solution.
I just realize that I have always worked under the assumption that fflush() after a read is a no-op; I just checked the 89 std and it says it is undefined. (I must have picked up that misunderstanding from some platform-specific man page.) This can be fixed by doing a ftell() followed by an fseek() call; this is required to flush the buffer if there was unwritten output data in the buffer, and is always allowed.
That's what I was hoping to avoid, but I don't care anymore: after staring it some more, I'm convinced that the current file_truncate() endures a ridiculous amount of complexity trying to gain a tiny bit of speed in what has to be a rare operation.
Ack, I glossed over the fileno() call in our file_truncate(). It's usually a Very Bad Idea to mix stream I/O and lower-level I/O operations without flushing your pants off, but I'm having a hard time thinking of a specific reason for doing so in the truncate case. Better safe than trying to out-think all possible implementations, though!
I think the fflush() is there for the following case: a file is opened for update, some data is written that overwrites some bytes in the middle but not yet flushed, and then the file is truncated to that position. Since ftruncate works on the file descriptor, you'd want the data flushed before truncating, otherwise things just get too complicated.
I suppose that clarifies my immediately preceding
ftruncate() isn't a standard C function,
<wink>?
I somehow misread what you wrote as "is a standard C function".
I just realize that I have always worked under the assumption that fflush() after a read is a no-op; I just checked the 89 std and it says it is undefined. (I must have picked up that misunderstanding from some platform-specific man page.) This can be fixed by doing a ftell() followed by an fseek() call; this is required to flush the buffer if there was unwritten output data in the buffer, and is always allowed.
That's what I was hoping to avoid, but I don't care anymore: after staring it some more, I'm convinced that the current file_truncate() endures a ridiculous amount of complexity trying to gain a tiny bit of speed in what has to be a rare operation.
Right. --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (3)
-
Guido van Rossum -
Neil Schemenauer -
Tim Peters