http://www.python.org/sf/595601 has been in limbo a long time. Looks like it popped up in real life, in Zope last week. Quickie: if you let run this long enough, it will die with a segfault (and under any version of Python): """ import threading class Worker(threading.Thread): def __init__(self, f): threading.Thread.__init__(self) self.f = f def run(self): for i in xrange(100): self.f.write('abc') def doit(): f = file('junk.txt', 'wb') w = Worker(f) w.start() f.close() w.join() if __name__ == '__main__': while True: print '.', doit() """ Who here cares?
Tim> http://www.python.org/sf/595601 Tim> has been in limbo a long time. Looks like it popped up in real Tim> life, in Zope last week. Tim> Quickie: if you let run this long enough, it will die with a Tim> segfault (and under any version of Python): ... Isn't this a case of "so don't do that"? I've never encountered this before, but then I don't close files in one thread while another holds a reference. Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1? Skip
On Mon, 2003-06-23 at 14:10, Skip Montanaro wrote:
Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1?
There could be lots of reasons why a file's refcount was greater than 1 without threads entering into the picture. -Barry
>> Would it be reasonable to raise an error or warning if a file's >> close() method was called when the reference count was greater than >> 1? Barry> There could be lots of reasons why a file's refcount was greater Barry> than 1 without threads entering into the picture. Yeah, but they probably all represent some sort of bug. How about a warning which is disabled by default but which the programmer can enable? Skip
[Skip Montanaro]
Yeah, but they probably all represent some sort of bug.
There's nothing fishy about having many references to a single file object.
import sys sys.getrefcount(sys.stdout) 5
How about a warning which is disabled by default but which the programmer can enable?
It's not addressing the right problem, and wouldn't stop the segfaults anyway.
[Skip Montanaro]
Isn't this a case of "so don't do that"?
I think so, but it's apparently debatable (since people at Zope Corp are debating it <wink>), and it doesn't excuse Python from segfaulting even if so.
I've never encountered this before, but then I don't close files in one thread while another holds a reference.
There's no problem with closing a shared file; the problem can arise only if one thread is trying to "do something" with a file at the same time another thread is trying to close that file.
Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1?
I don't think so. Shane Hathaway sketched an approach adding (in effect) an "in use" count to the FILE* contained by a Python file object. That count would be incremented and decremented, under protection of the GIL, around pieces of implementation code that passed on the FILE*. *Then* it would make sense to raise an exception if a close() call occurred while that count was above its initial value. This count is independent of how many references there may be to the Python file object.
Skip Montanaro wrote:
Tim> http://www.python.org/sf/595601
Tim> has been in limbo a long time. Looks like it popped up in real Tim> life, in Zope last week.
Tim> Quickie: if you let run this long enough, it will die with a Tim> segfault (and under any version of Python):
...
Isn't this a case of "so don't do that"? I've never encountered this before, but then I don't close files in one thread while another holds a reference.
Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1?
I think it would be very reasonable to raise an error if a file's close() method is called while another thread is executing a system call involving that file. The Python refcount doesn't provide that information, so Jeremy, Tim, Barry, and I have been discussing the idea of maintaining a special refcount on PyFileObjects. The new counter would be incremented before system calls involving f->f_fp and decremented afterward. The counter would provide to the close() method enough information to know it should raise an error instead of closing the file. What we're not sure of is whether we should be concerned about leaking file descriptors due to close() failing. We'd also like some confirmation that fixing this bug is worth a small burden of complexity. Shane P.S. I think this is my First Post on Python-dev. Howdy, all!
On Mon, 23 Jun 2003 14:43:43 -0400 Shane Hathaway <shane@zope.com> wrote:
What we're not sure of is whether we should be concerned about leaking file descriptors due to close() failing. We'd also like some confirmation that fixing this bug is worth a small burden of complexity.
I'm not yet comfortable with close() raising an exception. Maybe I should just get over it. Shane's suggestion of marking the object as failed and having the overlapping operation raise a different exception is interesting. It gives the programmer clear indication that something went wrong, but don't cause close() to fail. It may be too complicated, though: Say the overlapping read() fails; then you've got two errors you want to report. I think it's clear we should fix this, but it's not clear when. If the fix is complicated enough, I think it can wait for 2.4.
P.S. I think this is my First Post on Python-dev. Howdy, all!
You've been overdue here. Jeremy
Jeremy Hylton wrote:
On Mon, 23 Jun 2003 14:43:43 -0400 Shane Hathaway <shane@zope.com> wrote:
P.S. I think this is my First Post on Python-dev. Howdy, all!
You've been overdue here.
To be honest, I've been avoiding this list because I knew I'd like it too much, leading to less time spent with my family. Maybe I'll be OK if I only read this list during certain hours of the day. :-) http://hathaway.freezope.org Shane
Shane Hathaway <shane@zope.com>:
I think it would be very reasonable to raise an error if a file's close() method is called while another thread is executing a system call involving that file.
I don't think raising an exception is the right thing to do here. The closing thread hasn't really done anything wrong, it just happened to do the close at a moment that was inconvenient to another thread. I think the closing should simply be deferred until no operation on the file is in progress. Another thing -- is this problem confined to just closing? What if two threads perform some other operation on a file at the same time -- what stops them from interfering with each other? And whatever it is, why doesn't it stop closing from interfering with other operations? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[Greg Ewing]
I don't think raising an exception is the right thing to do here. The closing thread hasn't really done anything wrong, it just happened to do the close at a moment that was inconvenient to another thread.
At the level of C streams, it's engaging in undefined behavior. That's always wrong, except when Python explicitly decides to provide semantics beyond what C and POSIX specify. As far as POSIX is concerned, what happens when you close a stream while a read or write is in progress is undefined.
I think the closing should simply be deferred until no operation on the file is in progress.
That's a possible semantic Python could layer on top of the platform facilities, although it's not possible to do so in a bulletproof way (we simply don't control C-level FILE* objects, and C code can screw us whatever we try to do here; for example, it's flatly impossible for us to know whether a file operation is in progress on a FILE* -- we can only add cruft sufficient to tell whether the a file operation initiated by a Python file object is in progress on its contained FILE*, and because the Python C API has several functions that take (or return) FILE*s directly, that's not an empty qualification).
Another thing -- is this problem confined to just closing?
The specific segfault at issue is confined to closing, because that's the only Python file-object operation that resets the Python file object's stream to NULL.
What if two threads perform some other operation on a file at the same time -- what stops them from interfering with each other?
It depends on the precise operations you have in mind. If, for example, you're talking about two threads reading at the same time, then the platform stream library guarantees-- if it's threadsafe --that the reads will be serialized. OTOH, if you're talking about one thread reading and the other writing, that's engaging in undefined behavior, and there's no predicting the possible results. For that matter, you don't need threads at all -- e.g., the effects of writing followed by reading on a single file by a single thread, without an intervening flush or seek or reweind, are also undefined.
And whatever it is, why doesn't it stop closing from interfering with other operations?
Because Python resets a Python file object's stream to NULL when a Python file object is closed, it guarantees a segfault in some cases where the platform stream library may or may not have gone insane had the program been mucking with streams directly.
[Greg Ewing]
I don't think raising an exception is the right thing to do here. The closing thread hasn't really done anything wrong, it just happened to do the close at a moment that was inconvenient to another thread.
I disagree that the thread has done nothing wrong: if one thread is closing a file that another thread is still reading, the whole application is buggy, and if raising an exception in close() is the most expedient way to tell the application that it is buggy, that's fine with me. The application needs to be fixed anyway. [Tim]
At the level of C streams, it's engaging in undefined behavior. That's always wrong, except when Python explicitly decides to provide semantics beyond what C and POSIX specify. As far as POSIX is concerned, what happens when you close a stream while a read or write is in progress is undefined.
In such cases, Python ought to provide specified behavior beyond POSIX that at least guarantees that a Python program won't core dump or otherwise get into an undefined or unusable state as a result of engaging in such behavior. BTW, I try to have a practical attitude about behavior that standards don't define. I care about undefined behavior that is actually observed on some platform we (try to) support. For example, what happens when int arithmetic overflows is undefined by the C std, but we don't guard against this because in practice it is harmless enough. OTOH if closing a file that's being read by another thread can cause crashes on some platforms, Python should guard against this. We don't have the resources to guard against everything that standards call undefined; and there are lots of things that you want to do that can only be done by using undefined things: platforms often violate standards, and/or define things that the standard leaves undefined. As Tim knows, 100% standard-compliant life would be very dull. --Guido van Rossum (home page: http://www.python.org/~guido/)
I disagree that the thread has done nothing wrong: if one thread is closing a file that another thread is still reading, the whole application is buggy
It just seemed it would be a bit neater if the close succeeded and any subsequent read failed, since that's what would happen anyway if the close had been done a moment earlier. I guess it doesn't matter all that much. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Because Python resets a Python file object's stream to NULL when a Python file object is closed,
If that's the only problem, then surely it's just a matter of correct GIL usage? I.e. if a thread using a file object releases the GIL, it should re-acquire it and check that the FILE * in the file object is still valid before using it again. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
[Greg Ewing]
If that's the only problem, then surely it's just a matter of correct GIL usage? I.e. if a thread using a file object releases the GIL, it should re-acquire it and check that the FILE * in the file object is still valid before using it again.
Greg, read the patch comments -- they've been there a long time. There would be no problem if Python didn't release the GIL around potentially blocking I/O calls. But the instant you release the GIL, you have no guarantee that f->f_fp won't change to NULL from one nanosecond to the next. There isn't a simple solution to this.
participants (8)
-
Barry Warsaw -
Greg Ewing -
Guido van Rossum -
Jeremy Hylton -
Shane Hathaway -
Skip Montanaro -
Tim Peters -
Tim Peters