On Mon, 2003-06-23 at 14:10, Skip Montanaro wrote:

Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1?

There could be lots of reasons why a file's refcount was greater than 1 without threads entering into the picture. -Barry

[Skip Montanaro]

Isn't this a case of "so don't do that"?

I think so, but it's apparently debatable (since people at Zope Corp are debating it <wink>), and it doesn't excuse Python from segfaulting even if so.

I've never encountered this before, but then I don't close files in one thread while another holds a reference.

There's no problem with closing a shared file; the problem can arise only if one thread is trying to "do something" with a file at the same time another thread is trying to close that file.

Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1?

I don't think so. Shane Hathaway sketched an approach adding (in effect) an "in use" count to the FILE* contained by a Python file object. That count would be incremented and decremented, under protection of the GIL, around pieces of implementation code that passed on the FILE*. *Then* it would make sense to raise an exception if a close() call occurred while that count was above its initial value. This count is independent of how many references there may be to the Python file object.

Skip Montanaro wrote:

Tim> http://www.python.org/sf/595601

Tim> has been in limbo a long time. Looks like it popped up in real Tim> life, in Zope last week.

Tim> Quickie: if you let run this long enough, it will die with a Tim> segfault (and under any version of Python):

...

Isn't this a case of "so don't do that"? I've never encountered this before, but then I don't close files in one thread while another holds a reference.

Would it be reasonable to raise an error or warning if a file's close() method was called when the reference count was greater than 1?

I think it would be very reasonable to raise an error if a file's close() method is called while another thread is executing a system call involving that file. The Python refcount doesn't provide that information, so Jeremy, Tim, Barry, and I have been discussing the idea of maintaining a special refcount on PyFileObjects. The new counter would be incremented before system calls involving f->f_fp and decremented afterward. The counter would provide to the close() method enough information to know it should raise an error instead of closing the file. What we're not sure of is whether we should be concerned about leaking file descriptors due to close() failing. We'd also like some confirmation that fixing this bug is worth a small burden of complexity. Shane P.S. I think this is my First Post on Python-dev. Howdy, all!

Jeremy Hylton wrote:

On Mon, 23 Jun 2003 14:43:43 -0400 Shane Hathaway wrote:

...

P.S. I think this is my First Post on Python-dev. Howdy, all!

You've been overdue here.

To be honest, I've been avoiding this list because I knew I'd like it too much, leading to less time spent with my family. Maybe I'll be OK if I only read this list during certain hours of the day. :-) http://hathaway.freezope.org Shane

[Greg Ewing]

I don't think raising an exception is the right thing to do here. The closing thread hasn't really done anything wrong, it just happened to do the close at a moment that was inconvenient to another thread.

At the level of C streams, it's engaging in undefined behavior. That's always wrong, except when Python explicitly decides to provide semantics beyond what C and POSIX specify. As far as POSIX is concerned, what happens when you close a stream while a read or write is in progress is undefined.

I think the closing should simply be deferred until no operation on the file is in progress.

That's a possible semantic Python could layer on top of the platform facilities, although it's not possible to do so in a bulletproof way (we simply don't control C-level FILE* objects, and C code can screw us whatever we try to do here; for example, it's flatly impossible for us to know whether a file operation is in progress on a FILE* -- we can only add cruft sufficient to tell whether the a file operation initiated by a Python file object is in progress on its contained FILE*, and because the Python C API has several functions that take (or return) FILE*s directly, that's not an empty qualification).

Another thing -- is this problem confined to just closing?

The specific segfault at issue is confined to closing, because that's the only Python file-object operation that resets the Python file object's stream to NULL.

What if two threads perform some other operation on a file at the same time -- what stops them from interfering with each other?

It depends on the precise operations you have in mind. If, for example, you're talking about two threads reading at the same time, then the platform stream library guarantees-- if it's threadsafe --that the reads will be serialized. OTOH, if you're talking about one thread reading and the other writing, that's engaging in undefined behavior, and there's no predicting the possible results. For that matter, you don't need threads at all -- e.g., the effects of writing followed by reading on a single file by a single thread, without an intervening flush or seek or reweind, are also undefined.

And whatever it is, why doesn't it stop closing from interfering with other operations?

Because Python resets a Python file object's stream to NULL when a Python file object is closed, it guarantees a segfault in some cases where the platform stream library may or may not have gone insane had the program been mucking with streams directly.

Because Python resets a Python file object's stream to NULL when a Python file object is closed,

If that's the only problem, then surely it's just a matter of correct GIL usage? I.e. if a thread using a file object releases the GIL, it should re-acquire it and check that the FILE * in the file object is still valid before using it again. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[Greg Ewing]

If that's the only problem, then surely it's just a matter of correct GIL usage? I.e. if a thread using a file object releases the GIL, it should re-acquire it and check that the FILE * in the file object is still valid before using it again.

Greg, read the patch comments -- they've been there a long time. There would be no problem if Python didn't release the GIL around potentially blocking I/O calls. But the instant you release the GIL, you have no guarantee that f->f_fp won't change to NULL from one nanosecond to the next. There isn't a simple solution to this.