[Python-Dev] Destructors and Closing of File Objects

[Jeff Allen]

On 29/04/2013 15:42, Armin Rigo wrote:
> On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath <Nikolaus at rath.org> wrote:
>> It's indeed very informative, but it doesn't fully address the question
>> because of the _pyio module which certainly can't use any custom C code.
>> Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose
>> data in the write buffer when the interpreter exits without me calling
>> x.close(), but when using x = io.BufferedWriter(), the buffer is
>> guaranteed to get flushed?
> I actually described the behavior of CPython 2 while not realizing
> that CPython 3 silently dropped this guarantee.  (I also never
> realized that Jython/IronPython don't have the same guarantee; they
> could, if they implement 'atexit', like we did in PyPy.
On 29/04/2013 17:02, Antoine Pitrou wrote:
> It is dropped in the case of reference cycles, since there's no general
> way to decide in which order the tp_clear calls have to be done.
> Thus in the following layered situation: a TextIOWrapper on top of a
> BufferedWriter on top of a FileIO, if BufferedWriter.tp_clear is called
> first, it will flush and then close itself, closing the FileIO at the
> same time, and when TextIOWrapper.tp_clear will be called it will be
> too late to flush its own buffer.
>
> (I have to investigate a bit to confirm it is what happens)
>
> I will try to think of a scheme to make flushing more reliable, but
> nothing springs to my mind right now.
>
In Jython, objects are not "cleared" immediately they become unreachable 
and if the JVM does not collect them before it shuts down, no programmed 
finalization may be called. To get round this, files in need of closing 
are hooked to a list that is worked off as the JVM shuts down, the 
equivalent of atexit (I assume). It has the unfortunate effect that 
forgotten files may live even longer, making it even more desirable that 
the user remember to close them. (The io tests themselves are not good 
at this!) But at least the close comes eventually.

After discussion on jython-dev, I recently changed this mechanism 
(aiming at v2.7) so that every layer e.g. TextIOWrapper, BufferedWriter, 
FileIO is separately hooked to the list, and these are closed in reverse 
order of creation. Since close invokes flush when it matters, this will 
nearly always mean data is flushed down the stack before the path to 
disk gets severed, and always if you used open() to create the stack. I 
couldn't think of a perfect solution that didn't mean change to the API.

This idea, and some tidying up I did in the io tests, might be of use in 
CPython.

Jeff