NamedTemporaryFile and context managers
In issue14243 [1] there are two issues being tracked: - the difference in opening shared files between posix and Windows - the behavior of closing the underlying file in the middle of NamedTemporaryFile's context management I'd like to address and get feedback on the context management issue. ```python from tempfile import NamedTemporaryFile with NamedTemporaryFile() as fp: fp.write(b'some data') fp = open(fp.name()) data = fp.read() assert data == 'some_data' ``` Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that cannot open a temp file for reading while it is still open for writing). This would look like: ```python from tempfile import NamedTemporaryFile with NamedTemporaryFile() as fp: fp.write(b'some data') fp.close() # Windows workaround fp.open() data = fp.read() assert data == 'some_data' ``` The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2]. To handle this scenario, my proposal is two-fold: 1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close 2) add `.open()` to NamedTemporaryFile A possible side effect of (1) is that temp files may accumulate if the interpreter crashes, but given the file-management abilities in today's software that seems like a minor annoyance at most. The backwards compatibility issue of (1) is that the file is no longer deleted after a manual `close()` -- but why one would call close() and then stay inside the CM, outside of testing, I cannot fathom. [3] So, opinions on modifying NamedTemporaryFile to not delete on close() if inside a CM, and add open() ? -- ~Ethan~ [1] https://bugs.python.org/issue14243 [2] plus, the `.open()` doesn't yet exist [3] feel free to educate me :-)
On 4/8/21 1:43 PM, Antoine Pitrou wrote:
The main hurdle is that on Windows we let the OS manage the lifetime of the file, which means that it is deleted as soon as it is closed. We would need to remove that branch and treat all NamedTemporaryFiles the same. We could add reopen(), but since close() is already there... although I do like the name of `reopen`. -- ~Ethan~
Well this works: from tempfile import NamedTemporaryFile import os with NamedTemporaryFile(delete=False) as fp: fp.write(b'some data') fp.close() with open(fp.name, 'rb') as fp2: data = fp2.read() os.remove(fp.name) assert data == b'some data' Of course it is theoretically possible that another app will do something to the temporary file between the "fp.close()" and the "open" on the next line, or between closing fp2 and the "os.remove". Is this a concern? Rob Cliffe On 08/04/2021 22:42, Ethan Furman wrote:
On 4/8/2021 4:43 PM, Antoine Pitrou wrote:
I think capturing the intent is important, rather than just putting .close() followed by .open(). Maybe a name that embodies "reopen for reading without deleting" would make the intent clear? Maybe .reopen() already captures that. Anyway, +1 for a method that combines the close/open. Eric
I generally use a slightly different pattern for this: with NamedTemporaryFile() as fp: fp.write(b'some data’) fp.flush() fp.seek(0) data = fp.read() That is, reuse the same “fp” and just reset the stream. An advantage of this approach is that you don’t need a named temporary file for this (and could even use a spooled one). That said, I at times use this pattern with a named temporary file with a quick self-test for the file contents before handing of the file name to an external proces.
I’ve never had the need for such an API, but must say that I barely use Windows and hence have not run into the “cannot open file for reading what it is still open for writing” issue.
Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 08.04.2021 23:31, Ethan Furman wrote:
What's the problem with `NamedTemporaryFile(mode='w+b')`? (it's actually the default!) You can both read and write without reopening. If you don't want to seek() between reading and writing, you can dup() the descriptor and wrap it with another file object: `fp2=open(os.dup(fp.file.fileno()))`
-- Regards, Ivan
08.04.21 23:31, Ethan Furman пише:
These issues are usually solved by using TemporaryDirectory: with TemporaryDirectory() as td: filename = os.path.join(td, 'tempfile') with open(filename, 'wb') as fp: fp.write(b'some data') with open(filename, 'rb') as fp: data = fp.read() What if make NamedTemporaryFile a wrapper around TemporaryDirectory and always create a new temporary directory for file? @contextmanager def NamedTemporaryFile(): with TemporaryDirectory() as td: filename = os.path.join(td, 'tempfile') with open(filename, 'wb') as fp: yield fp
On 4/8/21 1:43 PM, Antoine Pitrou wrote:
The main hurdle is that on Windows we let the OS manage the lifetime of the file, which means that it is deleted as soon as it is closed. We would need to remove that branch and treat all NamedTemporaryFiles the same. We could add reopen(), but since close() is already there... although I do like the name of `reopen`. -- ~Ethan~
Well this works: from tempfile import NamedTemporaryFile import os with NamedTemporaryFile(delete=False) as fp: fp.write(b'some data') fp.close() with open(fp.name, 'rb') as fp2: data = fp2.read() os.remove(fp.name) assert data == b'some data' Of course it is theoretically possible that another app will do something to the temporary file between the "fp.close()" and the "open" on the next line, or between closing fp2 and the "os.remove". Is this a concern? Rob Cliffe On 08/04/2021 22:42, Ethan Furman wrote:
On 4/8/2021 4:43 PM, Antoine Pitrou wrote:
I think capturing the intent is important, rather than just putting .close() followed by .open(). Maybe a name that embodies "reopen for reading without deleting" would make the intent clear? Maybe .reopen() already captures that. Anyway, +1 for a method that combines the close/open. Eric
I generally use a slightly different pattern for this: with NamedTemporaryFile() as fp: fp.write(b'some data’) fp.flush() fp.seek(0) data = fp.read() That is, reuse the same “fp” and just reset the stream. An advantage of this approach is that you don’t need a named temporary file for this (and could even use a spooled one). That said, I at times use this pattern with a named temporary file with a quick self-test for the file contents before handing of the file name to an external proces.
I’ve never had the need for such an API, but must say that I barely use Windows and hence have not run into the “cannot open file for reading what it is still open for writing” issue.
Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 08.04.2021 23:31, Ethan Furman wrote:
What's the problem with `NamedTemporaryFile(mode='w+b')`? (it's actually the default!) You can both read and write without reopening. If you don't want to seek() between reading and writing, you can dup() the descriptor and wrap it with another file object: `fp2=open(os.dup(fp.file.fileno()))`
-- Regards, Ivan
08.04.21 23:31, Ethan Furman пише:
These issues are usually solved by using TemporaryDirectory: with TemporaryDirectory() as td: filename = os.path.join(td, 'tempfile') with open(filename, 'wb') as fp: fp.write(b'some data') with open(filename, 'rb') as fp: data = fp.read() What if make NamedTemporaryFile a wrapper around TemporaryDirectory and always create a new temporary directory for file? @contextmanager def NamedTemporaryFile(): with TemporaryDirectory() as td: filename = os.path.join(td, 'tempfile') with open(filename, 'wb') as fp: yield fp
participants (7)
-
Antoine Pitrou -
Eric V. Smith -
Ethan Furman -
Ivan Pozdeev -
Rob Cliffe -
Ronald Oussoren -
Serhiy Storchaka