[Python-Dev] Destructors and Closing of File Objects
Jeff Allen
"ja...py" at farowl.co.uk
Mon Apr 29 23:58:48 CEST 2013
On 29/04/2013 15:42, Armin Rigo wrote:
> On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath <Nikolaus at rath.org> wrote:
>> It's indeed very informative, but it doesn't fully address the question
>> because of the _pyio module which certainly can't use any custom C code.
>> Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose
>> data in the write buffer when the interpreter exits without me calling
>> x.close(), but when using x = io.BufferedWriter(), the buffer is
>> guaranteed to get flushed?
> I actually described the behavior of CPython 2 while not realizing
> that CPython 3 silently dropped this guarantee. (I also never
> realized that Jython/IronPython don't have the same guarantee; they
> could, if they implement 'atexit', like we did in PyPy.
On 29/04/2013 17:02, Antoine Pitrou wrote:
> It is dropped in the case of reference cycles, since there's no general
> way to decide in which order the tp_clear calls have to be done.
> Thus in the following layered situation: a TextIOWrapper on top of a
> BufferedWriter on top of a FileIO, if BufferedWriter.tp_clear is called
> first, it will flush and then close itself, closing the FileIO at the
> same time, and when TextIOWrapper.tp_clear will be called it will be
> too late to flush its own buffer.
>
> (I have to investigate a bit to confirm it is what happens)
>
> I will try to think of a scheme to make flushing more reliable, but
> nothing springs to my mind right now.
>
In Jython, objects are not "cleared" immediately they become unreachable
and if the JVM does not collect them before it shuts down, no programmed
finalization may be called. To get round this, files in need of closing
are hooked to a list that is worked off as the JVM shuts down, the
equivalent of atexit (I assume). It has the unfortunate effect that
forgotten files may live even longer, making it even more desirable that
the user remember to close them. (The io tests themselves are not good
at this!) But at least the close comes eventually.
After discussion on jython-dev, I recently changed this mechanism
(aiming at v2.7) so that every layer e.g. TextIOWrapper, BufferedWriter,
FileIO is separately hooked to the list, and these are closed in reverse
order of creation. Since close invokes flush when it matters, this will
nearly always mean data is flushed down the stack before the path to
disk gets severed, and always if you used open() to create the stack. I
couldn't think of a perfect solution that didn't mean change to the API.
This idea, and some tidying up I did in the io tests, might be of use in
CPython.
Jeff
More information about the Python-Dev
mailing list