[ Note: I already asked this on http://stackoverflow.com/questions/15917502 but didn't get any satisfactory answers] Hello, The description of tempfile.NamedTemporaryFile() says: ,---- | If delete is true (the default), the file is deleted as soon as it is | closed. `---- In some circumstances, this means that the file is not deleted after the program ends. For example, when running the following test under py.test, the temporary file remains: ,---- | from __future__ import division, print_function, absolute_import | import tempfile | import unittest2 as unittest | class cache_tests(unittest.TestCase): | def setUp(self): | self.dbfile = tempfile.NamedTemporaryFile() | def test_get(self): | self.assertEqual('foo', 'foo') `---- In some way this makes sense, because this program never explicitly closes the file object. The only other way for the object to get closed would presumably be in the __del__ destructor, but here the language references states that "It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits." So everything is consistent with the documentation so far. However, I'm confused about the implications of this. If it is not guaranteed that file objects are closed on interpreter exit, can it possibly happen that some data that was successfully written to a (buffered) file object is lost even though the program exits gracefully, because it was still in the file objects buffer and the file object never got closed? Somehow that seems very unlikely and un-pythonic to me, and the open() documentation doesn't contain any such warnings either. So I (tentatively) conclude that file objects are, after all, guaranteed to be closed. But how does this magic happen, and why can't NamedTemporaryFile() use the same magic to ensure that the file is deleted? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
[ Note: I already asked this on http://stackoverflow.com/questions/15917502 but didn't get any satisfactory answers]
Sorry, but that's not a reason to repost your question to this list. If you have to ask somewhere else, it would be python-list, aka, comp.lang.python.
Brian Curtin <brian@python.org> writes:
On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
[ Note: I already asked this on http://stackoverflow.com/questions/15917502 but didn't get any satisfactory answers]
Sorry, but that's not a reason to repost your question to this list. If you have to ask somewhere else, it would be python-list, aka, comp.lang.python.
I figured it belonged here because the question is really about the internal implementation of file objects, which to me didn't seem like a question about using Python. But I'll give it a few days and send another mail there if still haven't found the answer by then. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
You got your snswer 16 hours ago on S.O. On Monday, April 15, 2013, Nikolaus Rath wrote:
Brian Curtin <brian@python.org <javascript:;>> writes:
On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath <Nikolaus@rath.org<javascript:;>> wrote:
[ Note: I already asked this on http://stackoverflow.com/questions/15917502 but didn't get any satisfactory answers]
Sorry, but that's not a reason to repost your question to this list. If you have to ask somewhere else, it would be python-list, aka, comp.lang.python.
I figured it belonged here because the question is really about the internal implementation of file objects, which to me didn't seem like a question about using Python. But I'll give it a few days and send another mail there if still haven't found the answer by then.
Best,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
_______________________________________________ Python-Dev mailing list Python-Dev@python.org <javascript:;> http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
Guido van Rossum <guido@python.org> writes:
On Monday, April 15, 2013, Nikolaus Rath wrote:
Brian Curtin <brian@python.org <javascript:;>> writes:
On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath <Nikolaus@rath.org<javascript:;>> wrote:
[ Note: I already asked this on http://stackoverflow.com/questions/15917502 but didn't get any satisfactory answers]
Sorry, but that's not a reason to repost your question to this list. If you have to ask somewhere else, it would be python-list, aka, comp.lang.python.
I figured it belonged here because the question is really about the internal implementation of file objects, which to me didn't seem like a question about using Python. But I'll give it a few days and send another mail there if still haven't found the answer by then.
You got your answer 16 hours ago on S.O.
I guess you are referring to http://stackoverflow.com/a/15968516/293003 from Armin Ringo? ,---- | On Windows, NamedTemporaryFile uses a Windows-specific extension | (os.O_TEMPORARY) to ensure that the file is deleted when it is closed. | This probably also works if the process is killed in any way. However | there is no obvious equivalent on POSIX, most likely because on POSIX | you can simply delete files that are still in use; it only deletes the | name, and the file's content is only removed after it is closed (in any | way). But indeed assuming that we want the file name to persist until | the file is closed, like with NamedTemporaryFile, then we need "magic". | | We cannot use the same magic as for flushing buffered files. What occurs | there is that the C library handles it (in Python 2): the files are FILE | objects in C, and the C guarantees that they are flushed on normal | program exit (but not if the process is killed). In the case of Python | 3, there is custom C code to achieve the same effect. But it's specific | to this use case, not anything directly reusable. [...] `---- It's indeed very informative, but it doesn't fully address the question because of the _pyio module which certainly can't use any custom C code. Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose data in the write buffer when the interpreter exits without me calling x.close(), but when using x = io.BufferedWriter(), the buffer is guaranteed to get flushed? (Note: this isn't a complaint, just curiosity about the Python internals). Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
There are no guarantees in life. On the other hand: Don't worry, be happy. On Fri, Apr 26, 2013 at 7:39 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
Guido van Rossum <guido@python.org> writes:
On Monday, April 15, 2013, Nikolaus Rath wrote:
Brian Curtin <brian@python.org <javascript:;>> writes:
On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath <Nikolaus@rath.org<javascript:;>> wrote:
[ Note: I already asked this on http://stackoverflow.com/questions/15917502 but didn't get any satisfactory answers]
Sorry, but that's not a reason to repost your question to this list. If you have to ask somewhere else, it would be python-list, aka, comp.lang.python.
I figured it belonged here because the question is really about the internal implementation of file objects, which to me didn't seem like a question about using Python. But I'll give it a few days and send another mail there if still haven't found the answer by then.
You got your answer 16 hours ago on S.O.
I guess you are referring to http://stackoverflow.com/a/15968516/293003 from Armin Ringo?
,---- | On Windows, NamedTemporaryFile uses a Windows-specific extension | (os.O_TEMPORARY) to ensure that the file is deleted when it is closed. | This probably also works if the process is killed in any way. However | there is no obvious equivalent on POSIX, most likely because on POSIX | you can simply delete files that are still in use; it only deletes the | name, and the file's content is only removed after it is closed (in any | way). But indeed assuming that we want the file name to persist until | the file is closed, like with NamedTemporaryFile, then we need "magic". | | We cannot use the same magic as for flushing buffered files. What occurs | there is that the C library handles it (in Python 2): the files are FILE | objects in C, and the C guarantees that they are flushed on normal | program exit (but not if the process is killed). In the case of Python | 3, there is custom C code to achieve the same effect. But it's specific | to this use case, not anything directly reusable. [...] `----
It's indeed very informative, but it doesn't fully address the question because of the _pyio module which certainly can't use any custom C code. Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose data in the write buffer when the interpreter exits without me calling x.close(), but when using x = io.BufferedWriter(), the buffer is guaranteed to get flushed?
(Note: this isn't a complaint, just curiosity about the Python internals).
Best,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
Hi Nikolaus, On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
It's indeed very informative, but it doesn't fully address the question because of the _pyio module which certainly can't use any custom C code. Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose data in the write buffer when the interpreter exits without me calling x.close(), but when using x = io.BufferedWriter(), the buffer is guaranteed to get flushed?
I actually described the behavior of CPython 2 while not realizing that CPython 3 silently dropped this guarantee. (I also never realized that Jython/IronPython don't have the same guarantee; they could, if they implement 'atexit', like we did in PyPy. That's however more acceptable if the platform itself doesn't offer the guarantee.) Anyway, it's a guarantee that the C offers, so personally I find it reasonable to expect CPython files to offer it too; not offering it is kind of saying that there is a feature of C that is actually present at a higher level than the exact same feature in Python, which looks backward to me. Additionally, this might be introducing subtle bugs in programs when porting them to Python 3. However I realize that the two arguments presented above might not be accepted as relevant. (http://bugs.python.org/issue17852) A bientôt, Armin.
Le Mon, 29 Apr 2013 16:42:38 +0200, Armin Rigo <arigo@tunes.org> a écrit :
Hi Nikolaus,
On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
It's indeed very informative, but it doesn't fully address the question because of the _pyio module which certainly can't use any custom C code. Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose data in the write buffer when the interpreter exits without me calling x.close(), but when using x = io.BufferedWriter(), the buffer is guaranteed to get flushed?
I actually described the behavior of CPython 2 while not realizing that CPython 3 silently dropped this guarantee.
It is dropped in the case of reference cycles, since there's no general way to decide in which order the tp_clear calls have to be done. Thus in the following layered situation: a TextIOWrapper on top of a BufferedWriter on top of a FileIO, if BufferedWriter.tp_clear is called first, it will flush and then close itself, closing the FileIO at the same time, and when TextIOWrapper.tp_clear will be called it will be too late to flush its own buffer. (I have to investigate a bit to confirm it is what happens) I will try to think of a scheme to make flushing more reliable, but nothing springs to my mind right now. Note that the issue of how reference cycles involving globals are collected at interpreter shutdown is an orthogonal one, as pointed out in http://bugs.python.org/issue17852. Regards Antoine.
On 29/04/2013 15:42, Armin Rigo wrote:
It's indeed very informative, but it doesn't fully address the question because of the _pyio module which certainly can't use any custom C code. Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose data in the write buffer when the interpreter exits without me calling x.close(), but when using x = io.BufferedWriter(), the buffer is guaranteed to get flushed? I actually described the behavior of CPython 2 while not realizing
On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath <Nikolaus@rath.org> wrote: that CPython 3 silently dropped this guarantee. (I also never realized that Jython/IronPython don't have the same guarantee; they could, if they implement 'atexit', like we did in PyPy. On 29/04/2013 17:02, Antoine Pitrou wrote: It is dropped in the case of reference cycles, since there's no general way to decide in which order the tp_clear calls have to be done. Thus in the following layered situation: a TextIOWrapper on top of a BufferedWriter on top of a FileIO, if BufferedWriter.tp_clear is called first, it will flush and then close itself, closing the FileIO at the same time, and when TextIOWrapper.tp_clear will be called it will be too late to flush its own buffer.
(I have to investigate a bit to confirm it is what happens)
I will try to think of a scheme to make flushing more reliable, but nothing springs to my mind right now.
In Jython, objects are not "cleared" immediately they become unreachable and if the JVM does not collect them before it shuts down, no programmed finalization may be called. To get round this, files in need of closing are hooked to a list that is worked off as the JVM shuts down, the equivalent of atexit (I assume). It has the unfortunate effect that forgotten files may live even longer, making it even more desirable that the user remember to close them. (The io tests themselves are not good at this!) But at least the close comes eventually. After discussion on jython-dev, I recently changed this mechanism (aiming at v2.7) so that every layer e.g. TextIOWrapper, BufferedWriter, FileIO is separately hooked to the list, and these are closed in reverse order of creation. Since close invokes flush when it matters, this will nearly always mean data is flushed down the stack before the path to disk gets severed, and always if you used open() to create the stack. I couldn't think of a perfect solution that didn't mean change to the API. This idea, and some tidying up I did in the io tests, might be of use in CPython. Jeff
Hi Jeff, On Mon, Apr 29, 2013 at 11:58 PM, Jeff Allen <"ja...py"@farowl.co.uk> wrote:
In Jython, (...)
Thanks Jeff for pointing this out. Jython thus uses a custom mechanism similar to PyPy's, which is also similar to atexit's. It should not be too hard to implement it in CPython 3 as well, if this ends up classified as a bug. This is what my bug report was about (sorry if I failed to be clear enough about it). Nikolaus: the bug report contains a failing test, is that what you're looking for? A bientôt, Armin.
Armin Rigo <arigo@tunes.org> writes:
Hi Jeff,
On Mon, Apr 29, 2013 at 11:58 PM, Jeff Allen <"ja...py"@farowl.co.uk> wrote:
In Jython, (...)
Thanks Jeff for pointing this out. Jython thus uses a custom mechanism similar to PyPy's, which is also similar to atexit's. It should not be too hard to implement it in CPython 3 as well, if this ends up classified as a bug. This is what my bug report was about (sorry if I failed to be clear enough about it).
Personally, I think it should just be mentioned in the documentation for the buffered writers. Otherwise it's hard to justify what deserves such a special mechanism and what doesn't (what about e.g. tempfile.NamedTemporaryFile).
Nikolaus: the bug report contains a failing test, is that what you're looking for?
That's what I was trying to write as well, yes. Now I know how to do it :-) Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
Armin Rigo <arigo@tunes.org> writes:
Hi Nikolaus,
On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
It's indeed very informative, but it doesn't fully address the question because of the _pyio module which certainly can't use any custom C code. Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose data in the write buffer when the interpreter exits without me calling x.close(), but when using x = io.BufferedWriter(), the buffer is guaranteed to get flushed?
I actually described the behavior of CPython 2 while not realizing that CPython 3 silently dropped this guarantee. (I also never realized that Jython/IronPython don't have the same guarantee; they could, if they implement 'atexit', like we did in PyPy. That's however more acceptable if the platform itself doesn't offer the guarantee.)
Anyway, it's a guarantee that the C offers, so personally I find it reasonable to expect CPython files to offer it too; not offering it is kind of saying that there is a feature of C that is actually present at a higher level than the exact same feature in Python, which looks backward to me.
Additionally, this might be introducing subtle bugs in programs when porting them to Python 3.
However I realize that the two arguments presented above might not be accepted as relevant. (http://bugs.python.org/issue17852)
Thanks for the research! I tried writing a test for this myself, but failed to get the cyclic reference in place properly for it to happen. Seems I'm not the only one who was unaware and/or surprised by he change in behavior. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
participants (6)
-
Antoine Pitrou -
Armin Rigo -
Brian Curtin -
Guido van Rossum -
Jeff Allen -
Nikolaus Rath